One of my projects has a long-running task that constantly needs information from the database. I needed a mechanism to assure that the task will automatically reconnect to the database if and when that connection was broken.
I came up with this scheme using a trick with rescue blocks (code abbreviated for clarity) in this gist.
Here's a line-by-line explanation:
Line 4: This is where your application's database access logic would be.
Line 5: Catch a database access exception here
Here is where it gets interesting:
Line 7: Open a new block and retry the connection.
Line 10: This retry will retry the reconnect method and will loop as long as the database connection is still down.
Line 11: The else clause will execute if _no_ exception happened in line 10, and will retry the original database call in line 4.
In my case and example, I am not counting retries because I don't care that I've failed - I must continue to retry. You may want to use "retry if retries < 3" as a break mechanism.
I have also removed some mailer code that notifies me when the reconnect fails so I can (manually) see what happened to the connection. The moment the connection is re-established, life goes on as normal within the infinite while loop.