ts_mon outage for builders |
|||||
Issue descriptionI've seen this warning on a variety of builders..... For example, at the end of: https://uberchromegw.corp.google.com/i/chromeos/builders/master-release/builds/2770/steps/ManifestVersionedSync/logs/stdio 18:31:58: WARNING: Failed to configure ts_mon, monitoring is disabled: 'module' object has no attribute 'RetriableHttp' Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/lib/ts_mon_config.py", line 96, in SetupTsMonGlobalState config.process_argparse_options(parser.parse_args(args=args)) File "/b/cbuild/internal_master/chromite/third_party/infra_libs/ts_mon/config.py", line 238, in process_argparse_options credentials) File "/b/cbuild/internal_master/chromite/third_party/infra_libs/ts_mon/common/monitors.py", line 96, in __init__ http = httplib2_utils.RetriableHttp( AttributeError: 'module' object has no attribute 'RetriableHttp'
,
Jan 13 2017
Yes, on every builder I check.
,
Jan 13 2017
Probably, since httplib2_utils indeed doesn't have a RetriableHttp. Six months ago our ts_mon was updated with upstream without updating httplib2_utils with upstream, and now something is trying to actually use the new ts_mon code (in HttpsMonitor). The easiest solution would be to update our httplib2_utils with upstream Chrome infra though I'm not certain that it won't just push the error elsewhere.
,
Jan 13 2017
Viceroy indicates that we have no monarch data from builders, starting late in the evening on Jan 6. https://viceroy.corp.google.com/chromeos/long_term This means we're flying blind. Upping to P0.
,
Jan 13 2017
,
Jan 13 2017
RE 4 where is our httplib2_utils coming from? Is that a chromite third-party package?
,
Jan 13 2017
Answering my own question, yes, we have httplib2 in chromite's third_party. It claims to be version 0.8. Not sure what version ts_mon is expecting. Why did this just start happening on Jan 6? I don't see any relevant CLs during that time.
,
Jan 13 2017
+sergeyberezin +agable I'm seeing various versions of httplib2 in the chrome_infra checkout. Not sure which one is the blessed one that is expected to work with ts_mon. Thoughts?
,
Jan 13 2017
Actually, I'm a bit misled. httplib2_utils does not ship with httplib2, it looks like a separate helper library authored by chrome infra. My guess is we updated our ts_mon pin without updating that. However, should be safe to update it, as we don't have any other references to it in chromite...
,
Jan 13 2017
Please review https://chromium-review.googlesource.com/#/c/428333/
,
Jan 17 2017
,
Jan 23 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by akes...@chromium.org
, Jan 13 2017