See the RESOURCE_EXHAUSTED in master scheduler log every other minutes, is this affecting the scheduler tick rate.
04/18 14:53:56.742 WARNI| monitors:0161| HttpsMonitor.send received status 429: {
"error": {
"code": 429,
"message": "Resource has been exhausted (e.g. check quota).",
"status": "RESOURCE_EXHAUSTED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.DebugInfo",
"detail": "[ORIGINAL ERROR] generic::resource_exhausted: APPLICATION_ERROR;streamz/StreamzCollection.Write;Point value for metric '/chrome/infra/chromeos/autotest/drone/active_process_percentage' not of expected type 'INT64': column { name: \"drone_hostname\" string_value: \"chromeos-server11.hot.corp.google.com\" } double_value: 0.13833333333333334 start_timestamp { seconds: 1492537073 } end_timestamp { seconds: 1492552435 };AppErrorCode=9;StartTimeMs=1492552436631;tcp;Deadline(sec)=15.0;ResFormat=UNCOMPRESSED;ServerTimeSec=0.03117084503173828;LogBytes=256;FailFast;EffSecLevel=none;ReqFormat=UNCOMPRESSED;ReqID=f11fddb44942002;GlobalID=a91dd1189f7c4949;Server=10.87.117.193:4801"
}
]
}
}
04/18 14:54:14.137 INFO | thread_lib:0071| (Worker.cros-autotest-shard1.cbf.corp.google.com) finished.
Comment 1 by akes...@chromium.org
, Apr 18 2017