(maybe this is http://g/chromeos-infra-discuss/TnmWSSS64Ek)

Internal bug: http://b/80138348

There have been multiple canary failures for various boards since 5/19
or 5/20.  Many/most of the failures are attributable to known bugs that
are already dealt with, or in progress.

There _have_ been ongoing internal alerts regarding "ArchiverExportRateLow".

shuqianz@ - please investigate the following:
  * What's causing the alerts, how to fix them, and whether they're
    contributing to the problem here.
  * Whether there have been any Infra related canary failures
    for the runs after the weekend outage was fixed:
        https://uberchromegw.corp.google.com/i/chromeos/builders/master-release/builds/4241
        https://uberchromegw.corp.google.com/i/chromeos/builders/master-release/builds/4242
        https://uberchromegw.corp.google.com/i/chromeos/builders/master-release/builds/4243

Copy from an email thread:

From the linked graph, it looks like the afe_jobs table isn't getting exproted.
Looking at logs under /var/log/ci_results_archiver on the sentinel server, afe_jobs table is exporting 0 entries each time:

INFO 2018-05-22 09:20:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 09:40:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 10:00:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 10:20:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 10:40:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 11:00:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 11:20:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 11:40:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 12:00:03 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 12:20:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 12:40:02 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.
INFO 2018-05-22 13:00:03 afe_jobs [archive_builder.py:173] Success: Exported 0 entries, updated 0 entries.

This suggests that the DB connection to AFE isn't returning correct data to ci_results_archiver.

File a separate bug for the ci_result_archiver service, which I guess is the cause of this issue.

The ci_results_archiver service was broken since 5/20 due to the lab outage we had this Monday. The slave has been recovered, and the ci_results_archiver service is back to normal. Stainless should get test results slowly. 

Issue 845619 was fixed, and now Stainless shows test results for 10705.0.0 (built on May 22 11:11 PST).

https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-05-21&last_date=2018-05-23&suite=%5Ebvt%5C-cq%24&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false