Not only this failure is affected: none of the messages have updated for ~6 hours.

(I reverted to using build.chromium.org for the meantime)

SheriffOMatic being stuck is problematic for sheriffing.

Sorry for not responding earlier. I was at Camp Chrome yesterday and today. 

Looks like the Chromium tree is two days stale. Looking at the request logs there are several recent errors that look similar for different builders. ie: 

   0: {
    logMessage:  "Error fetching build https://build.chromium.org/p/chromium.memory/Linux CFI/8052: failed to decode data in memcache (data probably corrupt: 0 bytes). key chromium.memory/Linux CFI/8052 err unexpected end of JSON input"     
    severity:  "ERROR"     
    time:  "2018-05-24T22:30:24.169925Z"     
   }
   1: {
    logMessage:  "Error fetching build https://build.chromium.org/p/chromium.memory/Linux ChromiumOS MSan Tests/7263: failed to decode data in memcache (data probably corrupt: 0 bytes). key chromium.memory/Linux ChromiumOS MSan Tests/7263 err unexpected end of JSON input"     
    severity:  "ERROR"     
    time:  "2018-05-24T22:30:24.169937Z"     
   }
   2: {
    logMessage:  "Error fetching build https://build.chromium.org/p/chromium.memory/Linux ChromiumOS MSan Tests/7262: failed to decode data in memcache (data probably corrupt: 0 bytes). key chromium.memory/Linux ChromiumOS MSan Tests/7262 err unexpected end of JSON input"     
    severity:  "ERROR"     
    time:  "2018-05-24T22:30:24.169984Z"     
   } 

After digging a bit more, a few other error messages. 

{
    logMessage:  "couldn't get test expectations for http/tests/devtools/audits2/audits2-limited-run.js: errors fetching Expectation files: [error reading: Call error 11: Deadline exceeded (timeout)]"     
    severity:  "ERROR"     
    time:  "2018-05-24T22:36:39.885509Z"     
   } 

0: {
    logMessage:  "Process terminated because the request deadline was exceeded. (Error code 123)"     
    severity:  "ERROR"     
    time:  "2018-05-24T22:36:47.274477Z"     
   }

Tried temporarily rolling back the version and re-running the cron, but this didn't seem to help. So it's probably not related to any recent code changes. 

Looking through the code a bit, I think both the "Error fetching build" and "couldn't get test expectations" errors are non-blocking, so I think "Process terminated because the request deadline was exceeded" is probably what's causing the cron to fail. 

Looks like the data is up to date again. 

I'm not super comfortable that we've solved the root problem here, though. For now, I filed a followup bug to add alerting so that we can catch/respond to this problem faster/better in the future:  crbug.com/846511 

Of note, since the last release we've been sending a lot more requests to test-results.appspot.com.com, which is a little surprising: https://screenshot.googleplex.com/icKbWMMutg7

I did land a change that queries for more test-results data, but it should be cached. https://chromium-review.googlesource.com/c/infra/infra/+/1066819 is the CL, and it's trying to add extra test result data (perf artifacts) for each test failure in an alert.

I can think of two things: -verify that it's caching responses from test-reulsts, and also filter these checks to just the chromium.perf tree so chromium is unaffected.

Upon closer inspection it appears that test-results responses are *not* cached by the analyzer. That's probably why we're making so many more requests now, and also might be why this cron task is timing out recently.