I don't know why the scheduler was not running. Looks like it was stuck for ~3 hours trying to get google credentials:
https://cs.corp.google.com/chromeos_public/chromite/third_party/googleapiclient/http.py?q=%22Attempting+refresh+to+obtain+initial+access_token%22&sq=package:chromeos&l=1408


08/16 12:21:08.381 INFO |        monitor_db:0150| os.environ: {'USERNAME': 'chromeos-test', 'SUDO_COMMAND': '/usr/local/autotest/scheduler/monitor_db.py /usr/local/autotest/results --production', 'TERM': 'linux', 'SHELL': '/bin/bash', 'TZ': 'America/Los_Angeles', 'DJANGO_SETTINGS_MODULE': 'autotest_lib.frontend.settings', 'SUDO_UID': '0', 'SUDO_GID': '0', 'LOGNAME': 'chromeos-test', 'USER': 'chromeos-test', 'NO_GCE_CHECK': 'False', 'MAIL': '/var/mail/chromeos-test', 'PATH': '/usr/sbin:/usr/bin:/sbin:/bin', 'SUDO_USER': 'root', 'HOME': '/usr/local/google/home/chromeos-test'}
08/16 12:21:08.381 WARNI| metadata_reporter:0142| Elasticsearch db deprecated, no metadata will be reported.
08/16 12:21:08.381 INFO | metadata_reporter:0150| Metadata reporting thread is started.
08/16 12:21:08.392 INFO |    connectionpool:0207| Starting new HTTP connection (1): metadata.google.internal
08/16 12:21:08.700 INFO |        monitor_db:0200| 12:21:08 08/16/17> dispatcher starting
08/16 12:21:08.703 INFO |        monitor_db:0201| My PID is 20359
08/16 12:21:08.740 NOTIC|      cros_logging:0037| ts_mon was set up.
08/16 12:21:13.396 INFO |            client:0570| Attempting refresh to obtain initial access_token
08/16 12:21:13.509 INFO |            client:0872| Refreshing access_token
08/16 13:21:13.542 INFO |            client:0606| Refreshing due to a 401 (attempt 1/2)
08/16 13:21:13.637 INFO |            client:0872| Refreshing access_token
08/16 14:21:13.581 INFO |            client:0606| Refreshing due to a 401 (attempt 1/2)
08/16 14:21:13.678 INFO |            client:0872| Refreshing access_token
08/16 14:59:18.595 INFO |    httplib2_utils:0276| RetriableHttp: attempt 1 receiving status 504, will retry
08/16 15:21:13.618 INFO |            client:0606| Refreshing due to a 401 (attempt 1/2)
08/16 15:21:13.714 INFO |            client:0872| Refreshing access_token

the host-scheduler on this server was also down after the push_to_prod