That buildslave is completely bonkers.
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/buildslaves/build296-m2

All builds on it have been failing because 'repo' can not be updated.

A clobber build has gotten far enough in the sync slave: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/15220

I'll declare victory for this particular buildslave.

Next time when we see this repo version issue again, please do not clobber and let me try some possible fixes on the problemed builder. 

I searched for other recent victims:

(warning: This is a very bad query (even explain ... gets stuck for minutes). Don't productionalize)


cidb_debug:
mysql> select f2.bot_hostname, f1.id from buildTable f1 inner join (select max(id) id, bot_hostname, summary from buildTable group by bot_hostname) as f2 on f1.bot_hostname = f2.bot_hostname and f1.id = f2.id and f1.summary like '%SrcCheckOutException%' and start_time > '2017-01-10' limit 10;
+-------------------------------+-------+
| bot_hostname                  | id    |
+-------------------------------+-------+
| build296-m2.golo.chromium.org | 21098 |
+-------------------------------+-------+
1 row in set (0.20 sec)


cidb-prod-replica:
mysql> select f2.bot_hostname, f1.id from buildTable f1 inner join (select max(id) id, bot_hostname, summary from buildTable where start_time > '2017-01-10' group by bot_hostname) as f2 on f1.bot_hostname = f2.bot_hostname and f1.id = f2.id and f1.summary like '%SrcCheckOutException%' and start_time > '2017-01-10' limit 10;
Empty set (8.54 sec)


There aren't any other slaves at the moment.

what happened is:

"repo selfupdate" raised the warning "warning: Skipped upgrade to unverified version" without updating the repo. 

"repo init" went through without throwing any exception.

Then "repo sync" failed because of no --cache-dir option, which doesn't exist in old repo version.


One possible solution is to catch the output of "repo selfupdate". If the output contains the warning of old version and failure in update repo version, we can remove ".repo" folder like what we do when "repo selfupdate" really throws an exception: http://shortn/_ryeIVm9iid

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e7a2a91bf249c3f0c3921cb2405ef9d72247c9e6

commit e7a2a91bf249c3f0c3921cb2405ef9d72247c9e6
Author: Ningning Xia <nxia@chromium.org>
Date: Sat Jan 21 00:18:58 2017

Remove 'repo' dir when 'repo selfupdate' raises version warnings

Capture 'repo selfupdate' output, when 'repo selfupdate' complains about
repo version warnings and skips updating itself, remove the .repo/repo
folder. 'repo init' will clone a new copy of repo then.

BUG= chromium:683373 
TEST=unit_tests

Change-Id: I1905f71537ccbc6f6e1c61b61d190c021242e252
Reviewed-on: https://chromium-review.googlesource.com/430997
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/e7a2a91bf249c3f0c3921cb2405ef9d72247c9e6/cbuildbot/repository.py
[modify] https://crrev.com/e7a2a91bf249c3f0c3921cb2405ef9d72247c9e6/cbuildbot/repository_unittest.py

The pre-flight builder on R56 is exhibiting this.

https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-pre-flight-branch%20release-R56-9000.B/builds/196/steps/MasterSlaveLKGMSync/logs/stdio

Do we need to put this CL on 56 to capture more data?

If we want to get Chrome uprevved on 56 should we clobber it?

I'll take a look. please do not clobber first.

The fix just got merged into ToT, but not in 56.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c9762208ccaba775a9a91fa87d538c5e75174866

commit c9762208ccaba775a9a91fa87d538c5e75174866
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Jan 25 18:32:55 2017

Add monarch metrics for repo selfupdate/init failures

BUG= chromium:683373 
TEST=unit_tests

Change-Id: Ib9a0a35340aba9759248eba114f9a919bf4e03f6
Reviewed-on: https://chromium-review.googlesource.com/432837
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/c9762208ccaba775a9a91fa87d538c5e75174866/cbuildbot/repository.py
[modify] https://crrev.com/c9762208ccaba775a9a91fa87d538c5e75174866/lib/constants.py