Note: This is just about the codereview.chromium.org review description being updated after the patch lands with the commit position. Luckily, the git commits themselves are still good. (Phew. That woulda been hard to fix.)

This is commit position bot that likely stopped.

2016-10-20 09:57:20,786 - root - ERROR - Caught error while processing u'ee48f463535c73de660d4bda1185373e0fde11b7'
Traceback (most recent call last):
  File "commit_position_bot.py", line 195, in ProcessLogEntry
    session.close_issue(issue)
  File "/usr/local/google/home/chrome-cron/src/infra/depot_tools/rietveld.py", line 73, in close_issue
    self.post("/%d/close" % issue, [('xsrf_token', self.xsrf_token())])
  File "/usr/local/google/home/chrome-cron/src/infra/depot_tools/rietveld.py", line 400, in post
    ctype, body = upload.EncodeMultipartFormData(data, [])
  File "/usr/local/google/home/chrome-cron/src/infra/depot_tools/third_party/upload.py", line 839, in EncodeMultipartFormData
    body = CRLF.join(lines)
TypeError: sequence item 3: expected string, NoneType found

I don't understand what's wrong with Rietveld wrapper. maybe xsrf token is empty or smth like that.

So, I killed all 3 daemons for chormium, webrtc and v8. Let's see what happens.

So, the reason for failure is that apparently _xrsf_token in rietveld class is None even after xsrf_token() call IF the rietveld domain is chromiumcodereview.appspot.com .

https://chromereviews.googleplex.com/524547013

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/050ef8d32d5a1c14ec15a1e8cdfb92b5f33d8283

commit 050ef8d32d5a1c14ec15a1e8cdfb92b5f33d8283
Author: tandrii <tandrii@google.com>
Date: Fri Oct 21 12:41:04 2016

FTR, the error was:

LoginRequiredError: You are not logged in. Please login first by running:
  depot-tools-auth login chromiumcodereview.appspot.com

So my CL should resolve the problem.

Except that now we are over GoB quota:

HTTPError: HTTP Error 429: Too Many Requests
2016-10-21 12:45:20,413 - chrome.bugdroid.gitiles_poller - ERROR - Over quota. Failed refs request: https://chromium.googlesource.com/chromium/src.git/+refs?format=TEXT (HTTP Error 429: Too Many Requests).

Well, I don't have more time for this issue. The quota will be reset ~soonish (~6 hours) and the bot will start working. That's it. To whoever is next trooper -> please check the logs by 12pm MTV time if latest chromium/src master commit's CL doesn't get updated footer.

The bot has caught up with chromium, and is updating V8 and WebRTC. So, i think this is fixed, even if there will be some delays due to out of quota errors.

This is happening again, e.g.:
https://codereview.chromium.org/2569583002/,
https://codereview.chromium.org/2549103003/

Reopening.

Argh. I'm told to postpone Chromium gnumbd killing till January, so let me repair the stupid bot again.

Last processed commit is https://chromium.googlesource.com/chromium/src/+/1efb9a8a10ce04325dd43fd660ea2f748db2e0a2 according to the bot, and it's indeed processed.

SOmething weird is happening, possibly caused by me on d7492769f838600b19aa8190eb67c0a18458488d:

2016-12-12 13:44:34,604 - root - ERROR - Caught error while processing u'd7492769f838600b19aa8190eb67c0a18458488d'
Traceback (most recent call last):
  File "commit_position_bot.py", line 200, in ProcessLogEntry
    current_hash = self._get_current_hash(desc_lines)
  File "commit_position_bot.py", line 81, in _get_current_hash
    offset = -2 if desc_lines[-1].startswith('Cr-Commit-Position') else -1
IndexError: list index out of range

Wow, the d74927 has a CL with **empty** description: https://codereview.chromium.org/2558083004

it became empty when CQ decided to close it the issue upon committing.

I changed description of CL manually. And the bot processed it. However, there is another CL with empty description: 910dbb06389a97ef582641437c3115290d5b1468 
 https://codereview.chromium.org/2550093005

So, it was definitely caused by my CQ change. Investigating deeper.

So, what happened was a combination of: 
* my work on git-numberer in which CQ started adding more lines to description other than Review-Url
* extra codepath in CQ to figure out if CL was committed when git push fails for whatever reason. In cases when CL actually was committed, CQ would just update description of CL and close it without actually loading description from giot commit object. Hence, empty description.

CQ CL with a fix is landing & will deployed soon https://chrome-internal-review.googlesource.com/c/311495/.

Meanwhile, I'm manually updating all CLs with empty descriptions.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/5e8aeee07a5b7d1cc57eb6a9a8582dca113363d6

commit 5e8aeee07a5b7d1cc57eb6a9a8582dca113363d6
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Dec 12 14:30:50 2016

The cleanup of CQ glitch is handled in  issue 673334  for all the projects that don't have the commit position bot, even though it's very unlikely they were affected. For chromium, I've manually fixed all Cls since Friday and commit position bot has just processed latest HEAD (f2ae32c8f18e80cb12a37b359c11fe54eba38f94)