Assign to lead to find an owner for the bug.

+Richard who might know the upstart magic to do that. I can't find a waiting time setting for respawn.

There isn't a way to do this with "respawn limit", but we could have a special exit code for the process which indicates we should sleep before exiting the upstart job.

Assigning to deputy (Don) who can sync with Richard to look at what needs to be done.

My recommendation would be to skip trying to do this in the upstart
job, and instead change monitor_db.py to do some sort of check for
"database down" at startup, and then include a long wait (with retries)
before giving up.

Also, this is the sort of event that should generate some sort of e-mail
alert.

I have no idea how these processes are launched or managed.

Would changes in startup in monitor_db be sufficient?

> Would changes in startup in monitor_db be sufficient?

Should be.  Part of monitor_db initialization involves a lot
of DB queries to recover/reconstruct state.  So, the specific
change would be to start that recovery off with a dummy query
to confirm that the database is up.  If it fails, retry for
some extended period of time (or possibly even retry forever).

If we retry for long enough, eventually we can terminate without
triggering the upstart respawn limit.

The sanity check sounds like a good approach. We need to do this in both monitor_db.py and host_scheduler.py. Let's add a util function to be shared in both places.

Hey Charlene, can you take a look?

Do we have any idea of how common and how impactful this is?  For triaging, as usual.