Relevant part from the log:

2016-06-30 21:02:47,280 - tools_build - DEBUG - Received git log entry [snip]
2016-06-30 21:02:47,281 - tools_build - INFO - Processing commit 8c7a5b159b444e0193f9b2d8a094d352c226e7d1 : bugs {u'chromium': [620572]}
2016-06-30 21:02:47,282 - tools_build - DEBUG - The following revision refers to this bug: [snip]
2016-06-30 21:02:47,540 - tools_build - DEBUG - Attempting to save issue:  620572 
[W2016-07-01T04:02:48.461970+00:00 31693 139794231305984 googleapiclient.http:90] Retry #1 for request: POST https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues/620572/comments?alt=json&sendEmail=true

It's not clear why it retries that POST, but I guess the first one must've actually succeeded.
Also why is the log format different?

The log format is different because the last one is from google apiclient library instead of bugdroid.

apiclient auto retries when previous try failed, but from the log the previous attempt actually returns 200:

try #1:
21:02:47.548
POST
200
1.06 KB
414 ms
python-issue-tracker-manager/2.0 google-api-python-client/1.4.2 (gzip)
/_ah/spi/MonorailApi.issues_comments_insert

try #2:
21:02:48.484
POST
200
1.06 KB
356 ms
python-issue-tracker-manager/2.0 google-api-python-client/1.4.2 (gzip)
/_ah/spi/MonorailApi.issues_comments_insert


The retry logic is here: https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http-pysrc.html#_retry_request

I cannot really tell why it would retry on a successful attempt. Any idea?

I'm still seeing this, it happened today on  issue 620610  - comments #12 and #13 are identical, posted by bugdroid in the same minute.

Ben, Julie: bugdroid has "Owner: TBD" in go/chrome-infra-services - can you conscript a new owner and reassign this bug?

Mr. Gable, you are up.

And also, please add yourself in go/chrome-infra-services.

agable to the rescue :)  Thanks Julie!

Still happening:
https://bugs.chromium.org/p/chromium/issues/detail?id=584079 - comment #336 and #337.

Please find someone to own bugdroid, and fix this.

Aaron is the appropriate owner.

This is a separate problem from the double commits, right?

Yes it's a separate problem, this is 2 comments for the same git commit.

Ping.  Are you just hoping I'll get so annoyed by this that I fix it myself?

https://bugs.chromium.org/p/chromium/issues/detail?id=587527#c44 - comments #44 and #45.

Currently I'm prioritizing getting PolyGerrit ready for Chromium over this.

I hadn't realized that this bug had been marked Pri-1. I disagree with that assessment -- duplicate comments are an annoyance at worst; they are not a full failure or loss of information. If I had a good guess of why this was happening and thought I could just fix it in a single morning, I would have done so already. Unfortunately, my gut says it will take more investigation than that, so I haven't set aside the time. I'm also not sure why this got unassigned from me on Sept 1st, causing it to drop off my radar for 2.5 months.

If you believe that this should be trumping my other priorities, please re-raise the priority of this bug and I can see about shuffling my tasks.

My concern is that for duplicate comments to happen so regularly, Bugdroid might be making some fundamentally broken assumptions about transactions or state persistence.  If there's some deep brokenness there, maybe it's manifesting in other ways as well that are less noticeable.  Would anyone actually notice if Bugdroid was *not* posting comments when it should?

I agree that this is lower priority than the other stuff you're doing - I was surprised jparent gave it to you.  But services still need owners, so back to jparent to find someone with time to look at this.

I'm not sure why this was marked a Pri-1, I disagree with that prioritization.  I gave it to Aaron because he is the owner of Bugdroid at the moment.

 Issue 671939  has been merged into this issue.

https://chrome-internal-review.googlesource.com/323284

In the process of investigating a different bug I think I got a lead on this one.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/5636360ea8a1d54dfdacf34983c6e47d95aefc25

commit 5636360ea8a1d54dfdacf34983c6e47d95aefc25
Author: Aaron Gable <agable@chromium.org>
Date: Thu Feb 02 17:51:05 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/5636360ea8a1d54dfdacf34983c6e47d95aefc25

commit 5636360ea8a1d54dfdacf34983c6e47d95aefc25
Author: Aaron Gable <agable@chromium.org>
Date: Thu Feb 02 17:51:05 2017

I love how bugdroid updated this bug twice.

But thanks for fixing it :D

It doesn't look like this is fixed, I saw it again today:
https://bugs.chromium.org/p/chromium/issues/detail?id=643689#c13 and #c14 are the same.

I've got no ideas for what would be causing it now; going to leave this open but not investigate as a high priority.

If you ctrl+f for 19ccf90ae5de8deb5ed44c8500b5c7fe3c8fc70d in https://luci-logdog.appspot.com/v/?s=infra-internal%2Fbb%2Finternal.infra.cron%2Fbugdroid%2F39512%2F%2B%2Frecipes%2Fsteps%2Fbugdroid%2F0%2Fstdout, you can see that it does the right thing. It polls it, prints the commit message, posts to the bug, and then moves on. When it polls the same commit later, it appropriately ignores it.

The second comment was posted 3 seconds after the first, at :52. There is a stack trace and a retry log in bugdroid at that point:


2017-02-09 15:54:50,136 - chromium_src - ERROR - Uncaught Exception in <infra.services.bugdroid.bugdroid.BugdroidGitPollerHandler object at 0x7fcd82f5ca90>
Traceback (most recent call last):
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/gitiles_poller.py", line 180, in _ProcessGitLogEntry
    handler.ProcessLogEntry(log_entry)
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/bugdroid.py", line 129, in ProcessLogEntry
    issue = itm.getIssue(bug)
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/IssueTrackerManager.py", line 332, in getIssue
    projectId=self.project_name, issueId=issue_id).execute(num_retries=5)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/http.py", line 761, in execute
    return self.postproc(resp, content)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/model.py", line 207, in response
    return self.deserialize(content)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/model.py", line 262, in deserialize
    content = content.decode('utf-8')
  File "/b/build/slave/bugdroid/build/infra/ENV/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8c in position 706: invalid start byte
[E2017-02-09T23:54:50.136197+00:00 24923 140519040608000 infra.services.bugdroid.gitiles_poller:186] Uncaught Exception in <infra.services.bugdroid.bugdroid.BugdroidGitPollerHandler object at 0x7fcd82f5ca90>
Traceback (most recent call last):
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/gitiles_poller.py", line 180, in _ProcessGitLogEntry
    handler.ProcessLogEntry(log_entry)
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/bugdroid.py", line 129, in ProcessLogEntry
    issue = itm.getIssue(bug)
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/IssueTrackerManager.py", line 332, in getIssue
    projectId=self.project_name, issueId=issue_id).execute(num_retries=5)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/http.py", line 761, in execute
    return self.postproc(resp, content)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/model.py", line 207, in response
    return self.deserialize(content)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/model.py", line 262, in deserialize
    content = content.decode('utf-8')
  File "/b/build/slave/bugdroid/build/infra/ENV/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8c in position 706: invalid start byte
2017-02-09 15:54:50,137 - chromium_src - INFO - Handler is not fatal. Continuing.
[W2017-02-09T23:54:51.855655+00:00 24923 140519007037184 googleapiclient.http:90] Retry #1 for request: POST https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues/643689/comments?alt=json&sendEmail=true


It's not clear if the stack trace caused the retry or if they're coincidental (especially since the retry itself comes from googleapiclient.py, while the stack trace comes from gitiles_poller.py; I'd expect the latter to be catching exceptions thrown by the former, but not the other way around, so I'm not sure how an exception in gitiles_poller would cause a retry in googleapiclient).

Either way, it's not clear why googleapiclient would retry even though the first request was successful.

Issue 703548 has been merged into this issue.

Issue 699987 has been merged into this issue.

Marked isssue 699987 as duplicate, which besides examples for double or more comments also has an example of a wrongly associated CL.

Issue 620189 has been merged into this issue.

 Issue 647109  has been merged into this issue.

Ping!  On Issue 727337 I got 3 duplicate comments for the same commit (#10, #11 and #12)

I wish I had cycles for this; with the Gerrit launch coming up, I don't. Luckily, bugdroid is code-review-system-agnostic when it decides to horrificially misbehave, so this problem isn't related to the gerrit migration.

 Issue 731936  has been merged into this issue.

Here's one recent example:

https://bugs.chromium.org/p/chromium/issues/detail?id=680153#c25 - three comments for the same commit.

Logs: https://uberchromegw.corp.google.com/i/internal.infra.cron/builders/bugdroid/builds/59769/steps/bugdroid/logs/stdio

Just the log lines related to this issue:

[I2017-06-26T22:30:14.686217+00:00 24538 140716181272320 infra.services.bugdroid.bugdroid:78] (gerrit_chrome_internal) Processing commit b9a51dde6ce7a455474f3bc68fa4215cac5d4e1c : bugs {u'chromium': [680153]}
[D2017-06-26T22:30:14.686498+00:00 24538 140716181272320 infra.services.bugdroid.bugdroid:81] (gerrit_chrome_internal) The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-reef-uni-private/+/b9a51dde6ce7a455474f3bc68fa4215cac5d4e1c

commit b9a51dde6ce7a455474f3bc68fa4215cac5d4e1c
Author: C Shapiro <shapiroc@google.com>
Date: Mon Jun 26 22:28:37 2017

[I2017-06-26T22:30:14.689248+00:00 24538 140716181272320 googleapiclient.discovery:817] (googleapiclient.discovery) URL being requested: GET https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues/680153?alt=json
[I2017-06-26T22:30:14.689500+00:00 24538 140716181272320 oauth2client.client:539] (oauth2client.client) Attempting refresh to obtain initial access_token
[I2017-06-26T22:30:14.689799+00:00 24538 140716181272320 oauth2client.client:797] (oauth2client.client) Refreshing access_token
[D2017-06-26T22:30:14.995698+00:00 24538 140716181272320 infra.services.bugdroid.bugdroid:94] (gerrit_chrome_internal) Attempting to save issue:  680153 
[I2017-06-26T22:30:15.001222+00:00 24538 140716181272320 googleapiclient.discovery:817] (googleapiclient.discovery) URL being requested: POST https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues/680153/comments?alt=json&sendEmail=true
[W2017-06-26T22:30:17.427452+00:00 24538 140716181272320 googleapiclient.http:90] (root) Retry #1 for request: POST https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues/680153/comments?alt=json&sendEmail=true
[W2017-06-26T22:30:18.812614+00:00 24538 140716181272320 googleapiclient.http:90] (root) Retry #2 for request: POST https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues/680153/comments?alt=json&sendEmail=true

During this 4 second window two other HTTP requests for other issues also fail and are retried.  One POST to issue 735653 fails with some unknown error and is retried by googleapiclient.http, then it fails again a second time with the usual UnicodeDecodeError.  Neither of these cause duplicate comments on that issue.

Also during this 4 second window the oauth access token is refreshed.

We see 5 '901' responses during this Bugdroid run: http://shortn/_TosCdVLoDM.  That corresponds with the 5 instances of 'Retry #... for request' we see in the logs.

901 is a fake response code used by our monitoring for STATUS_ERROR, which is one of socket.error, socket.herror, socket.gaierror.

googleapiclient.http._retry_request only retries ssl.SSLError exceptions, which are a subclass of socket.error.  So that must be what these mysterious errors are.  If only it logged the exception text as well...

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fbe9699f74530c345566070690753a20abcf7f58

commit fbe9699f74530c345566070690753a20abcf7f58
Author: David Sansome <dsansome@chromium.org>
Date: Tue Jun 27 15:40:09 2017

Add some logging to investigate UnicodeDecodeError decoding monorail responses.

Bug:  625036 
Change-Id: Id127a9643e5cc44ff5fde6f55f5e0b5d3d234be5
Reviewed-on: https://chromium-review.googlesource.com/549875
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/fbe9699f74530c345566070690753a20abcf7f58/infra/services/bugdroid/monorail_client.py
[modify] https://crrev.com/fbe9699f74530c345566070690753a20abcf7f58/infra/services/bugdroid/test/monorail_client_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/464fe5f82a554e921f8083f6a19b5211c0284a75

commit 464fe5f82a554e921f8083f6a19b5211c0284a75
Author: David Sansome <dsansome@chromium.org>
Date: Thu Jun 29 01:23:28 2017

Log ssl.SSLErrors raised by http.request

Bug:  625036 
Change-Id: If87d4942e29a8d3c90b96fd127254a57c1ba7c2f
Reviewed-on: https://chromium-review.googlesource.com/549677
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Dave Sansome <dsansome@chromium.org>

[modify] https://crrev.com/464fe5f82a554e921f8083f6a19b5211c0284a75/infra/services/bugdroid/monorail_client.py

It looks like there's some subtle corruption in the bytes coming off the network.  That's causing SSL errors, and the unicode decode errors:

SSLError: [Errno 1] _ssl.c:1429: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac

And:

[D2017-06-28T23:59:58.756481+00:00 23659 139731912349440 infra.services.bugdroid.monorail_client:135] (root) Error decoding UTF-8 HTTP response.  Response headers:
{'status': '200'}
Response body:
'H\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xd7x\xa2\xd9\xf1\x9d\x13\xd9[\xc6o\x99\xc0Z\xd0\xd6\x9a'
[E2017-06-28T23:59:58.756831+00:00 23659 139731912349440 infra.services.bugdroid.gerrit_poller:157] (gerrit_chrome_internal) Uncaught Exception in <infra.services.bugdroid.bugdroid.BugdroidGitPollerHandler object at 0x7f16489abc10>
Traceback (most recent call last):
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/gerrit_poller.py", line 154, in _ProcessGitLogEntry
    handler.ProcessLogEntry(log_entry)
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/bugdroid.py", line 97, in ProcessLogEntry
    project, issue, log_parser.should_send_email(log_entry.msg))
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/monorail_client.py", line 139, in update_issue
    req.execute(num_retries=5)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/http.py", line 761, in execute
    return self.postproc(resp, content)
  File "/b/build/slave/bugdroid/build/infra/infra/services/bugdroid/monorail_client.py", line 131, in request_postproc
    return original_postproc(resp, content)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/model.py", line 207, in response
    return self.deserialize(content)
  File "/b/build/slave/bugdroid/build/infra/ENV/local/lib/python2.7/site-packages/googleapiclient/model.py", line 262, in deserialize
    content = content.decode('utf-8')
  File "/b/build/slave/bugdroid/build/infra/ENV/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd7 in position 799: invalid continuation byte

This could either be a libssl bug, a network driver bug, a kernel bug, or something else wrong with this VM.

I'm going to respawn slave23-c7 and see if it helps.

Respawning the slave didn't fix it.

Next idea: maybe this is a thread-safety bug.  We're sharing the same httplib2.Http object and the same google-api-python-client between multiple GitilesPoller and GerritPoller threads.  If all threads are sharing the same HTTP connection then two requests made at the same time will stomp over each other's data.

https://developers.google.com/api-client-library/python/guide/thread_safety

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/3762c6dcee74a4300c1917b93b3f897ac24ada62

commit 3762c6dcee74a4300c1917b93b3f897ac24ada62
Author: David Sansome <dsansome@chromium.org>
Date: Mon Jul 03 01:12:25 2017

Make a new httplib2.Http for each request.

Avoids sharing the same object (and connection pool, and socket) between multiple
threads.
https://developers.google.com/api-client-library/python/guide/thread_safety

Bug:  625036 
Change-Id: I76123b62b59ce095ef4483c33b2cb3e2ae1045ce
Reviewed-on: https://chromium-review.googlesource.com/557740
Commit-Queue: Dave Sansome <dsansome@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/3762c6dcee74a4300c1917b93b3f897ac24ada62/infra/services/bugdroid/monorail_client.py

There have been no more client HTTP exceptions since the thread-safety fix went in 3 days ago: http://shortn/_PaCkxU6wYw, so I'm declaring this fixed \o/

\o/ \o/ \o/

You win my favorite-person-of-the-day award :)