New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Verified
Closed: Jan 2017
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug

Sign in to add a comment

ts_mon outage for builders

Project Member Reported by, Jan 13 2017 Back to list

Issue description

I've seen this warning on a variety of builders.....

For example, at the end of:

18:31:58: WARNING: Failed to configure ts_mon, monitoring is disabled: 'module' object has no attribute 'RetriableHttp'
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/", line 96, in SetupTsMonGlobalState
  File "/b/cbuild/internal_master/chromite/third_party/infra_libs/ts_mon/", line 238, in process_argparse_options
  File "/b/cbuild/internal_master/chromite/third_party/infra_libs/ts_mon/common/", line 96, in __init__
    http = httplib2_utils.RetriableHttp(
AttributeError: 'module' object has no attribute 'RetriableHttp'

Is this reproducing?
Yes, on every builder I check.
Probably, since httplib2_utils indeed doesn't have a RetriableHttp.  Six months ago our ts_mon was updated with upstream without updating httplib2_utils with upstream, and now something is trying to actually use the new ts_mon code (in HttpsMonitor).

The easiest solution would be to update our httplib2_utils with upstream Chrome infra though I'm not certain that it won't just push the error elsewhere.
Labels: -Pri-2 Pri-0
Summary: ts_mon outage for builders (was: ts_mon broken?)
Viceroy indicates that we have no monarch data from builders, starting late in the evening on Jan 6.

This means we're flying blind. Upping to P0.
RE 4 where is our httplib2_utils coming from? Is that a chromite third-party package?
Answering my own question, yes, we have httplib2 in chromite's third_party. It claims to be version 0.8. Not sure what version ts_mon is expecting.

Why did this just start happening on Jan 6? I don't see any relevant CLs during that time.
+sergeyberezin +agable I'm seeing various versions of httplib2 in the chrome_infra checkout. Not sure which one is the blessed one that is expected to work with ts_mon. Thoughts?
Actually, I'm a bit misled. httplib2_utils does not ship with httplib2, it looks like a separate helper library authored by chrome infra. My guess is we updated our ts_mon pin without updating that. However, should be safe to update it, as we don't have any other references to it in chromite...
Labels: -current-issue
Status: Verified

Sign in to add a comment