Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: Verified
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment
ts_mon outage for builders
Project Member Reported by dgarr...@chromium.org, Jan 13 2017 Back to list
I've seen this warning on a variety of builders.....

For example, at the end of:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-release/builds/2770/steps/ManifestVersionedSync/logs/stdio


18:31:58: WARNING: Failed to configure ts_mon, monitoring is disabled: 'module' object has no attribute 'RetriableHttp'
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/ts_mon_config.py", line 96, in SetupTsMonGlobalState
    config.process_argparse_options(parser.parse_args(args=args))
  File "/b/cbuild/internal_master/chromite/third_party/infra_libs/ts_mon/config.py", line 238, in process_argparse_options
    credentials)
  File "/b/cbuild/internal_master/chromite/third_party/infra_libs/ts_mon/common/monitors.py", line 96, in __init__
    http = httplib2_utils.RetriableHttp(
AttributeError: 'module' object has no attribute 'RetriableHttp'

 
Owner: ayatane@chromium.org
Is this reproducing?
Yes, on every builder I check.
Probably, since httplib2_utils indeed doesn't have a RetriableHttp.  Six months ago our ts_mon was updated with upstream without updating httplib2_utils with upstream, and now something is trying to actually use the new ts_mon code (in HttpsMonitor).

The easiest solution would be to update our httplib2_utils with upstream Chrome infra though I'm not certain that it won't just push the error elsewhere.
Labels: -Pri-2 Pri-0
Summary: ts_mon outage for builders (was: ts_mon broken?)
Viceroy indicates that we have no monarch data from builders, starting late in the evening on Jan 6.

https://viceroy.corp.google.com/chromeos/long_term

This means we're flying blind. Upping to P0.
RE 4 where is our httplib2_utils coming from? Is that a chromite third-party package?
Answering my own question, yes, we have httplib2 in chromite's third_party. It claims to be version 0.8. Not sure what version ts_mon is expecting.

Why did this just start happening on Jan 6? I don't see any relevant CLs during that time.
Cc: sergeybe...@chromium.org aga...@chromium.org
+sergeyberezin +agable I'm seeing various versions of httplib2 in the chrome_infra checkout. Not sure which one is the blessed one that is expected to work with ts_mon. Thoughts?
Actually, I'm a bit misled. httplib2_utils does not ship with httplib2, it looks like a separate helper library authored by chrome infra. My guess is we updated our ts_mon pin without updating that. However, should be safe to update it, as we don't have any other references to it in chromite...
Labels: -current-issue
Status: Verified
Sign in to add a comment