New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 837691 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

ArchiverExportRateLow id:2732430442186678406

Project Member Reported by gu...@chromium.org, Apr 27 2018

Issue description

We repeatedly receive this alert, although jkop@ seems had submit some fixes.

 
Labels: Chase-Pending
Owner: jkop@chromium.org
likely same root cause as Issue 837687

Comment 3 by jkop@chromium.org, Apr 27 2018

Status: Assigned (was: Untriaged)

Comment 4 by jkop@chromium.org, Apr 27 2018

Status: Started (was: Assigned)
First error:

INFO 2018-04-26 16:21:06 tko_jobs [client.py:795] Refreshing access_token
Traceback (most recent call last):
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/ci_results_archiver/main.py", line 215, in <module>
    main()
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/ci_results_archiver/main.py", line 209, in main
    _BuilderMain(table_type=options.table_type, configs=configs)
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/ci_results_archiver/main.py", line 114, in _BuilderMain
    builder.Run()
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/venv/ci_results_archiver/archive_builder.py", line 154, in Run
    exported_table_suffixes = self._exporter.ExportNewEntries(entries)
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/venv/ci_results_archiver/bigquery_exporter.py", line 47, in ExportNewEntries
    entries, self._table_spec.partition_timestamp_column)
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/venv/ci_results_archiver/bigquery_exporter.py", line 91, in _PartitionByDate
    date = entry[timestamp_column].date()
AttributeError: 'NoneType' object has no attribute 'date'
ERROR 2018-04-26 16:21:07 scheduler [base.py:131] Job "tko_jobs (trigger: interval[0:20:00], next run at: 2018-04-26 23:40:00 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/google/home/chromeos-test/.cache/cros_venv/venv-2.7.6-9b73359f13f2cf7b6ed8f86d2c7d6ff7/local/lib/python2.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/ci_results_archiver/main.py", line 100, in _RunBuilder
    subprocess32.check_call(builder_argv, timeout=timeout.total_seconds())
  File "/usr/local/google/home/chromeos-test/.cache/cros_venv/venv-2.7.6-9b73359f13f2cf7b6ed8f86d2c7d6ff7/local/lib/python2.7/site-packages/subprocess32.py", line 602, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/local/google/home/chromeos-test/.cache/cros_venv/venv-2.7.6-9b73359f13f2cf7b6ed8f86d2c7d6ff7/bin/python', '/usr/local/google/home/chromeos-test/chromiumos/infra/ci_results_archiver/ci_results_archiver/main.py', '--config=/etc/ci_results_archiver/config.yaml', '--one-shot', '--table-type=tko_jobs']' returned non-zero exit status 1
INFO 2018-04-26 16:40:00 scheduler [base.py:123] Running job "afe_jobs (trigger: interval[0:20:00], next run at: 2018-04-27 00:00:00 UTC)" (scheduled at 2018-04-26 23:40:00+00:00)
INFO 2018-04-26 16:40:00 scheduler [base.py:123] Running job "tko_jobs (trigger: interval[0:20:00], next run at: 2018-04-27 00:00:00 UTC)" (scheduled at 2018-04-26 23:40:00+00:00)
INFO 2018-04-26 16:40:00 scheduler [base.py:123] Running job "cidb_builds (trigger: interval[0:20:00], next run at: 2018-04-27 00:00:00 UTC)" (scheduled at 2018-04-26 23:40:00+00:00)
NOTICE 2018-04-26 16:40:01 cidb_builds [cros_logging.py:38] ts_mon was set up.
NOTICE 2018-04-26 16:40:01 tko_jobs [cros_logging.py:38] ts_mon was set up.
NOTICE 2018-04-26 16:40:01 afe_jobs [cros_logging.py:38] ts_mon was set up.

Comment 5 by jkop@chromium.org, Apr 28 2018

Unrelated to Issue 837687, one revert attempted but no change.

Comment 6 by jkop@chromium.org, Apr 28 2018

Seems like almost certainly a result of an autotest change, started within 10 minutes of a push to prod.
Status: Fixed (was: Started)

Comment 8 by jkop@chromium.org, May 2 2018

Status: Verified (was: Fixed)
Was caused by crrev.com/c/1018529 and fixed by the revert in crrev.com/c/1033511

Comment 9 by jkop@chromium.org, May 2 2018

Correction: It was not caused by those at all. There has been no push to prod since that revert landed Friday evening, and it resolved itself at about 16:15 Pacific Monday. http://shortn/_ZatC1nphZB

Regardless, it's definitely fixed.

Comment 10 by jkop@chromium.org, May 2 2018

Status: Started (was: Verified)
It's back as of last night.

Comment 11 by jkop@chromium.org, May 3 2018

Status: Fixed (was: Started)
Appears fixed again, this time more explicably. crrev.com/c/1033510

Sign in to add a comment