I've been seeing this on the eve-firmware-9584.B branch builder as well.  It usually works after a retry, but yesterday it seems completely stuck when I need it of course..

I tried to launch it as a tryjob with:
cros tryjob --production --branch firmware-eve-9584.B eve-firmware

But I get a different failure in ManifestVersionedSync:
https://ci.chromium.org/p/chromeos/builds/b8947086633925436560

07:56:38: INFO: Recording status inflight for [u'eve-firmware']
07:56:41: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-manifest-versions/builder-status/9584.151.0/eve-firmware
07:56:42: ERROR: <class 'chromite.cbuildbot.manifest_version.GenerateBuildSpecException'>: Builder already inflight
Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 777, in PerformStage
    fail_if_exists=fail_if_exists)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 797, in SetInFlight
    dashboard_url=dashboard_url)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 769, in _UploadStatus
    raise GenerateBuildSpecException('Builder already inflight')
GenerateBuildSpecException: Builder already inflight

Based on #4, the lazy fix is to migrate all 6 build slaves on this waterfall to GCE.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/ee4bb6903a022d7c76508efe502705d89b43856a

commit ee4bb6903a022d7c76508efe502705d89b43856a
Author: Don Garrett <dgarrett@google.com>
Date: Tue May 08 22:59:43 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/3c56915ec2193f7be4e4bc3941f369482a648338

commit 3c56915ec2193f7be4e4bc3941f369482a648338
Author: Don Garrett <dgarrett@google.com>
Date: Tue May 08 23:18:23 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/938d831110a1439c23cd7cafcfcea3b5521f6a1c

commit 938d831110a1439c23cd7cafcfcea3b5521f6a1c
Author: Don Garrett <dgarrett@google.com>
Date: Tue May 08 23:23:47 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/7faf481945c5e19b99fd14133443c27f41cb5bee

commit 7faf481945c5e19b99fd14133443c27f41cb5bee
Author: Don Garrett <dgarrett@google.com>
Date: Wed May 09 00:08:17 2018

The builders have been swapped out, I've started two new builds to see if the solve the issue.


https://uberchromegw.corp.google.com/i/chromeos.branch/builders/eve%20firmware%20firmware-eve-campfire-9584.131.B/builds/16
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/eve%20firmware%20firmware-eve-9584.B/builds/176

 Issue 841259  has been merged into this issue.

hungte@'s build from the duped bug ran before the builders were swapped out. Hopefully a re-build will succeed.

Trying to see if things are working today, I hit issues with the waterfall itself.

The waterfall is back, and waterfall builds hit this error.

08:56:36: INFO: No manifest-versions checkout exists at /b/swarming/w/ir/cache/cbuild/repository/manifest-versions-internal
08:56:36: INFO: Cloning fresh manifest-versions checkout.
08:56:36: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/manifest-versions /b/swarming/w/ir/cache/cbuild/repository/manifest-versions-internal in /b/swarming/w/ir/cache/cbuild/repository/manifest-versions-internal
08:59:13: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9042.192.0/snappy-firmware
08:59:15: WARNING: GS_ERROR: Your "Oauth 2.0 User Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config").
Failure: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed.
 


This looks like an updated version of gsutil is needed. That's a big scary, because the gsutil command line has changed over time and this ripples out into a huge merge effort.

https://cros-goldeneye.corp.google.com/chromeos/console/firmwareQualEditDetails?firmwareQualId=461&disco=AAAAB830upw&ts=5af3386a&usp=comment_email_document&bypassOAuth=true is blocked on this bug.

I've just gotten told that this should be an issue that belongs to the Build team in the new organization of Infra teams, and that I should hand it off to that team.

Passing to vapier@ based on that reasoning.

this seems like a CI issue ... but either way, let's queue this up for the CI oncall to take a look

> This looks like an updated version of gsutil is needed.

Can you clarify why you think this? The referenced command works fine in my local chroot...

With the same version of gsutil? Notice that the branch builds are using the pinned version of 4.19.

They are also running it using builder authentication botos, which can get a bit weird. I originally though this was hacked around using GCE because GCE instances authenticate via a service account instead of credentials embedded in the .boto file mentioned on the command line.

I don't really understand the error, but suspect it's related to an expired certificate embedded in gsutil. I was hoping that cert was related to explicit .boto authentication, but it seems that it's not.

However, I AM confused about why the same builds seemed to work when run as tryjobs..... sometimes.

My hunch is that some infrastructure configuration (e.g. service account) has been ripped out from under the firmware branch.

If we think it is a problem with old GSUtil should we try to update the branch's GSUtil to a newer version? 

(is this just a matter of bumping src/third_party/portage-stable/net-misc/gsutil/ and hoping for the best?)

Bumping the version might work, but gsutil's command line interface changed a little somewhere along there, and we had to update our libraries to match.

Update: it looks like this is some bad interaction between the .boto credential and the old gsutil 4.19 version these branches are on. dgarrett@ is going to try upreving gsutil.

Here is the most recent uprev CL. 

https://crrev.com/c/373073

I've created this merge CL: https://crrev.com/c/1053121 and am running a tryjob against it for snappy here:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8946970338168394016

I suggest we merge the CL after we see if the tryjob gets past the problem.

Oops. I misspelled the config name in the first tryjob. Re-launched here:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8946968660212293488

That tryjob failed with what looks like a real compile error, but did upload limited build artifacts.

I've cherry-picked in the hopeful fix to the reef firmware branch (used by snappy), and requested a build on the snappy-firmware builder.

My hope is that the manifest-versions problem will be fixed, but I expect the build to fail in build-packages.

Is there a short term solution, for example to trigger a build manually?
We have an incoming build and now there's no way to get new official images from factory branch...

Merging https://crrev.com/c/1053121 is what's required. I just did the manual triggers to test the fix.

Thanks. Do we need to update that to all branches (i.e., cherry-pick to all branches with failure?)

Well I cherry-picked those into veyron_7505 and it now builds and passed MasterSlaveLKGMSync  . So sounds like all branches may need to be upgraded in future...

 Issue 842600  has been merged into this issue.

We have confirmation now that the cherry-pick fixes the issue. Anyone wishing to build a firmware branch will have to cherry-pick https://chromium-review.googlesource.com/c/chromiumos/chromite/+/373073 back to their firmware branch in order for that branch to continue to interoperate with Cloud Storage. This cannot be automated.

Note.... for very old branches, that cherry-pick will be more complicated, because of changes to the GS commandline arguments over time.

I ran into this with firmware-cyan-7287.57.B which did not cherry pick cleanly.

I had to:
cd chromite
git checkout -b firmware-cyan-7287.57.B cros/firmware-cyan-7287.57.B 
git cherry-pick -sx 9d4dbe0d7ad069f80b7bf076e27ce89159b78f36 bc71179a4ac963ebf56807a835a7fd507c56af39 b62e341ff76debde2542f1a5b39d8642ddf2c422 021e187bfa1cd689e8cd25332e0359f21974a9e5 9ab368dc7dd84d3234b0238b13bb03999b5fe359 8e212d96220123161d16305de849c385afeb76ae
git push cros HEAD:refs/for/firmware-cyan-7287.57.B

 Issue 842287  has been merged into this issue.

I assume the cherry-pick needs to be applied to factory branches as well?
Thanks.


 

If they're very old, yes.

I cherry-picked 021e187bfa1cd689e8cd25332e0359f21974a9e5 9ab368dc7dd84d3234b0238b13bb03999b5fe359 8e212d96220123161d16305de849c385afeb76a
to factory-oak-8182.B.

But when I ran unit test:
$ ./cbuildbot/run_tests

I saw:
15:30:29: ERROR: The following 4 tests failed:
  cli/cros/cros_payload_unittest
  cli/flash_unittest
  lib/paygen/paygen_payload_lib_unittest
  lib/portage_util_unittest

CL https://chromium-review.googlesource.com/c/chromiumos/chromite/+/373073 landed on 8734.0.0 so I assume all branches branched off afterward should not have this problem. But there's one still have it, though?

https://uberchromegw.corp.google.com/i/chromeos.branch/builders/reef%20firmware%20firmware-cr50-release-9308.25.B/builds/42

This shouldn't be marked fixed until firmware branches actually build. All firmware branches should be working and maintained as we need the ability to apply security fixes quickly.

Not a p0 though

re#47,

Also "reef factory factory-reef-8811.B" has this issue but should already have CL:373073 .

I ran a script to parse the current versions from the branches on the waterfall, here they are (I reserve the right to be wrong):

 firmware-cr50-9308.B is version 4.19
factory-smaug-7265.B is version 4.13
factory-glados-7828.B is version 4.15
firmware-poppy-10431.B is version 4.27
firmware-cr50-release-9308.25.B is version 4.19
firmware-gandof-6301.155.B is version 4.7
firmware-edgar-7287.167.B is version 4.13
factory-strago-7458.B is version 4.13
factory-veyron-7505.B is version 4.21
firmware-strago-7287.B is version 4.13
firmware-terra-7287.154.B is version 4.13
factory-scarlet-10211.B is version 4.27
firmware-smaug-7900.B is version 4.12
firmware-samus-6300.B is version 4.7
fatal: Path 'lib/gs.py' exists on disk, but not in 'cros/firmware-snow-2695.B'.
factory-reef-8811.B is version 4.19
firmware-storm-6315.B is version 4.7
factory-arkham-7077.113.B is version 4.11
firmware-gru-8785.B is version 4.19
factory-samus-6658.B is version 4.7
factory-soraka-10323.39.B is version 4.27
firmware-oak-8438.B is version 4.19
firmware-cr50-9308.24.B is version 4.19
firmware-reks-7287.133.B is version 4.13
firmware-cyan-7287.57.B is version 4.21
factory-gru-9017.B is version 4.19
factory-oak-8182.B is version 4.21
factory-glados-7657.B is version 4.15
factory-eve-9667.B is version 4.19
firmware-gale-8281.B is version 4.18
firmware-guado-6301.108.B is version 4.7
firmware-paine-6301.58.B is version 4.7
firmware-auron-6301.B is version 4.7
factory-coral-10122.B is version 4.27
firmware-fizz-10139.B is version 4.27
firmware-rikku-6301.110.B is version 4.7
firmware-tidus-6301.109.B is version 4.7
firmware-celes-7287.92.B is version 4.21
firmware-servo-9040.B is version 4.19
firmware-cr50-mp-release-9308.87.B is version 4.19
firmware-coral-10068.B is version 4.27
firmware-fizz-10139.94.B is version 4.27
factory-veyron-6591.B is version 4.7
factory-gale-8743.19.B is version 4.19
factory-poppy-10504.B is version 4.27
firmware-eve-campfire-9584.131.B is version 4.27
firmware-smaug-7132.B is version 4.12
firmware-veyron-6588.B is version 4.7
firmware-glados-7820.B is version 4.15
factory-fizz-10167.B is version 4.27
firmware-eve-9584.B is version 4.19
fatal: Path 'lib/gs.py' exists on disk, but not in 'cros/firmware-link-2695.B'.
firmware-link-2695.B is version 4.19
firmware-yuna-6301.59.B is version 4.7
fatal: Path 'lib/gs.py' exists on disk, but not in 'cros/firmware-parrot-2685.B'.
firmware-parrot-2685.B is version 4.7
firmware-reef-9042.B is version 4.20
fatal: Path 'lib/gs.py' exists on disk, but not in 'cros/firmware-uboot_v2-1299.B'.
firmware-uboot_v2-1299.B is version 4.20
firmware-ultima-7287.131.B is version 4.13
factory-whirlwind-6812.41.B is version 4.8
firmware-scarlet-10388.B is version 4.27
firmware-buddy-6301.202.B is version 4.7
factory-gru-8652.B is version 4.19
factory-auron-6772.B is version 4.8
firmware-lulu-6301.136.B is version 4.7


I'll take a stab at updating some of these.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ecc4a4cc5cfba7d670a3079dd4311f8912acfba8

commit ecc4a4cc5cfba7d670a3079dd4311f8912acfba8
Author: Ningning Xia <nxia@chromium.org>
Date: Tue May 15 20:03:49 2018

Log gs errors in gs _RetryFilter

Temporary log for debuging chromium:642986. Will be reverted after the issue is
identified.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: Ia5716057cf30415203f6153a9d1d78966de1a5e0
Reviewed-on: https://chromium-review.googlesource.com/379616
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Trybot-Ready: Ningning Xia <nxia@chromium.org>
(cherry picked from commit 2241deca14274e9a7c53c86f4fb33b31e5b205c3)
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1059722
Reviewed-by: Sean Paul <seanpaul@google.com>
Commit-Queue: Sean Paul <seanpaul@google.com>
Tested-by: Sean Paul <seanpaul@google.com>

[modify] https://crrev.com/ecc4a4cc5cfba7d670a3079dd4311f8912acfba8/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/924964824b6b0cd9233a3a1891b61e5e9b5837c6

commit 924964824b6b0cd9233a3a1891b61e5e9b5837c6
Author: Ningning Xia <nxia@chromium.org>
Date: Tue May 15 20:03:50 2018

Add logs in gs.Copy()

We saw gs related errors in BuildStages and most of them were related to
gs.Copy(). Revert the log change in the _RetryFilter and add warning
in gs.Copy(), to reduce spam logs in _RetryFilter and turn on logs in
gs.Copy() for investigation.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I66b09629075abfa6d4a338b840aee79a6932c9be
Reviewed-on: https://chromium-review.googlesource.com/396883
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
(cherry picked from commit 03998c2458a6abae87366d7c631c379b97c6f976)
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1059723
Reviewed-by: Sean Paul <seanpaul@google.com>
Commit-Queue: Sean Paul <seanpaul@google.com>
Tested-by: Sean Paul <seanpaul@google.com>

[modify] https://crrev.com/924964824b6b0cd9233a3a1891b61e5e9b5837c6/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c270083765ded25227941aa9c1bf3398942ee204

commit c270083765ded25227941aa9c1bf3398942ee204
Author: Ningning Xia <nxia@chromium.org>
Date: Tue May 15 20:07:18 2018

Log gs errors in gs _RetryFilter

Temporary log for debuging chromium:642986. Will be reverted after the issue is
identified.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: Ia5716057cf30415203f6153a9d1d78966de1a5e0
Reviewed-on: https://chromium-review.googlesource.com/379616
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Trybot-Ready: Ningning Xia <nxia@chromium.org>
(cherry picked from commit 2241deca14274e9a7c53c86f4fb33b31e5b205c3)
Signed-off-by: Sean Paul <seanpaul@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1059463

[modify] https://crrev.com/c270083765ded25227941aa9c1bf3398942ee204/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4c0e836b08ccfc4cd98f106e8e70d7b65e5b7bfc

commit 4c0e836b08ccfc4cd98f106e8e70d7b65e5b7bfc
Author: Ningning Xia <nxia@chromium.org>
Date: Tue May 15 20:07:19 2018

Add logs in gs.Copy()

We saw gs related errors in BuildStages and most of them were related to
gs.Copy(). Revert the log change in the _RetryFilter and add warning
in gs.Copy(), to reduce spam logs in _RetryFilter and turn on logs in
gs.Copy() for investigation.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I66b09629075abfa6d4a338b840aee79a6932c9be
Reviewed-on: https://chromium-review.googlesource.com/396883
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
(cherry picked from commit 03998c2458a6abae87366d7c631c379b97c6f976)
Signed-off-by: Sean Paul <seanpaul@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1059464

[modify] https://crrev.com/4c0e836b08ccfc4cd98f106e8e70d7b65e5b7bfc/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8861af5c50ffeb61d105aa968fedff1ed55ba2b6

commit 8861af5c50ffeb61d105aa968fedff1ed55ba2b6
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Tue May 15 20:07:20 2018

Do not rely on ext in case of auto_compress.

It used to use -z option, which filters files by extension.
Though, what we need here is -Z, which always runs gzip compressing,
regardless of file extensions.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I0083fa56aaee21c44e769d8b6dd18f5ffb6f3a8f
Reviewed-on: https://chromium-review.googlesource.com/489221
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit efa9dac8187495d263cd652060e1c7cdfd678cec)
Signed-off-by: Sean Paul <seanpaul@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1059465

[modify] https://crrev.com/8861af5c50ffeb61d105aa968fedff1ed55ba2b6/lib/gs_unittest.py
[modify] https://crrev.com/8861af5c50ffeb61d105aa968fedff1ed55ba2b6/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/351d836f4a07f6d450f16cbf72374f082bdd14b8

commit 351d836f4a07f6d450f16cbf72374f082bdd14b8
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Tue May 15 20:07:21 2018

Do not rely on ext in case of auto_compress.

It used to use -z option, which filters files by extension.
Though, what we need here is -Z, which always runs gzip compressing,
regardless of file extensions.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I0083fa56aaee21c44e769d8b6dd18f5ffb6f3a8f
Reviewed-on: https://chromium-review.googlesource.com/489221
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit efa9dac8187495d263cd652060e1c7cdfd678cec)
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1059724
Reviewed-by: Sean Paul <seanpaul@google.com>
Commit-Queue: Sean Paul <seanpaul@google.com>
Tested-by: Sean Paul <seanpaul@google.com>

[modify] https://crrev.com/351d836f4a07f6d450f16cbf72374f082bdd14b8/lib/gs_unittest.py
[modify] https://crrev.com/351d836f4a07f6d450f16cbf72374f082bdd14b8/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f99f52747e2c7c600e404a8ac3d71c1ad0e74c8e

commit f99f52747e2c7c600e404a8ac3d71c1ad0e74c8e
Author: Mike Frysinger <vapier@chromium.org>
Date: Tue May 15 20:07:22 2018

gs: LS: fix local usage in --debug mode

Make sure we don't rely on implicit settings for output capture.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I6dc8772fa4ee46d99e0fa7a478c9ebaca4457075
Reviewed-on: https://chromium-review.googlesource.com/656104
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
(cherry picked from commit 7901ea1a0de59bc5f51dd5536654531244e5e202)
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1059725
Reviewed-by: Sean Paul <seanpaul@google.com>
Commit-Queue: Sean Paul <seanpaul@google.com>
Tested-by: Sean Paul <seanpaul@google.com>

[modify] https://crrev.com/f99f52747e2c7c600e404a8ac3d71c1ad0e74c8e/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/807b49ee0e31832c1b307da237c44081c32e58f2

commit 807b49ee0e31832c1b307da237c44081c32e58f2
Author: Mike Frysinger <vapier@chromium.org>
Date: Tue May 15 20:07:23 2018

gs: LS: fix local usage in --debug mode

Make sure we don't rely on implicit settings for output capture.

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I6dc8772fa4ee46d99e0fa7a478c9ebaca4457075
Reviewed-on: https://chromium-review.googlesource.com/656104
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
(cherry picked from commit 7901ea1a0de59bc5f51dd5536654531244e5e202)
Signed-off-by: Sean Paul <seanpaul@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1059466

[modify] https://crrev.com/807b49ee0e31832c1b307da237c44081c32e58f2/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/d2cd1adb5e28f8e7c1eb3ccdca894d93bb6bb3a6

commit d2cd1adb5e28f8e7c1eb3ccdca894d93bb6bb3a6
Author: Xixuan Wu <xixuan@chromium.org>
Date: Tue May 15 20:07:23 2018

gsutil: upgrade from 4.19 to 4.27

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I4bdb25524ae8b388491894756f9445a03b9d94cb
Reviewed-on: https://chromium-review.googlesource.com/653370
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit 9e2f6cc0b698872b0fe7bf7863de844ded5fce63)
Signed-off-by: Sean Paul <seanpaul@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1060253

[modify] https://crrev.com/d2cd1adb5e28f8e7c1eb3ccdca894d93bb6bb3a6/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/89dfa871f506e90498fce830db678672703f1612

commit 89dfa871f506e90498fce830db678672703f1612
Author: Xixuan Wu <xixuan@chromium.org>
Date: Tue May 15 20:07:24 2018

gsutil: upgrade from 4.19 to 4.27

BUG= chromium:839949 
TEST=./lib/gs_unittest.py --network && ./lib/paygen/gslib_unittest.py --network && ./lib/paygen/gslock_unittest.py --network && ./lib/paygen/gspaths_unittest.py --network

Change-Id: I4bdb25524ae8b388491894756f9445a03b9d94cb
Reviewed-on: https://chromium-review.googlesource.com/653370
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit 9e2f6cc0b698872b0fe7bf7863de844ded5fce63)
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1059726
Reviewed-by: Sean Paul <seanpaul@google.com>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>
Commit-Queue: Sean Paul <seanpaul@google.com>
Tested-by: Sean Paul <seanpaul@google.com>

[modify] https://crrev.com/89dfa871f506e90498fce830db678672703f1612/lib/gs.py

From the branch waterfall, it looks like some builders got stuck at ManifestVersionedSync stage for quite a long time and therefore blocking other builders -- we have seen case of pending for >44hours.

Is it possible to suspend the builders that we knew to have the GS problem and resume them once the problem been addressed?

Thanks.

After these builders are migrated to the new infrastructure, it will be a button press.

Today it's hard.

This sounds drastic and naive -- maybe we can take those builders off branch mapping (i.e. off the branch waterfall) and put them back again later?

Thanks.

That's what I meant by hard. It's possible. gmeinke@ is the current CI oncall, and so can help with that.

Uploaded https://crrev.com/i/627186 as example to temporarily disable branch builders (list from #51), and to be restored once the gsutil been updated.

I am bumping the priority as some builders cannot get to run, for example,

https://uberchromegw.corp.google.com/i/chromeos.branch/builders/fizz%20pre-flight%20firmware-fizz-10139.B

and this is impacting the schedule.

Thanks.

FWIW, I've hit a snag trying to update gandof (although the problem is not specific to that board, but rather old chromite instances). I can't manage to kick off a trybot to verify my cherry-picks. I've tried using local branch, Change-Id, and Gerrit Id, but all fail in different ways.

The unit tests pass, should we consider merging without trybot verification since the builders are already broken?

Hum....

Given the age of the branch, try running this in your checkout....

cbuildbot --buildroot <working_dir>  --debug <build_config>

Or

cbuildbot --buildroot <working_dir> --buildbot --debug <build_config>

Both are capable of screwing up your current checkout though.

You could also just run:

cbuildbot/run_tests

and/or

cbuildbot/lib/gs_unittest --network

(adjust directory as needed)

We are working on identifying the root cause and figuring out a workable solution

Multiple boards are affected here .. please so let us know if any work around available 

We have a root cause for the branch waterfall being affected. Currently, the branch waterfall has exhausted all machine resources because the gsutil failure on these branches occasionally takes 20 hours instead of 20 minutes. We are looking into a few options:

* Make them fail faster so that folks can revive these branches with the upgraded gsutil when time allows; this would flush the pending requests and restore service to those branches that are working fine. Pending investigation. Maybe in cbuildbot_launcher?
* Add more machines; this is harder than it sounds
* Hack the .boto files on these machines to make them succeed using machine credentials instead of service account credentials (we're not sure if this will work).
* Disable all failing branches in the scheduler so that they don't schedule until they are repair

IMHO, if we need to go into individual branch to make change to workaround the problem (that seems to be implied by option 1?) we may as well uprev gsutil instead? Grant it that this change to shorten the timeout may be easier to apply to all "bad" branches, though.

No, cbuildbot_launcher is ToT always.

That timeout is being imposed by the waterfall itself. Any change will apply to every build it does.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/d37a787d605de54fe54b0ed1405a79c6294aa27f

commit d37a787d605de54fe54b0ed1405a79c6294aa27f
Author: Don Garrett <dgarrett@google.com>
Date: Thu May 17 18:44:42 2018

reduced the timeout from 16h to 2h and restarting the chromeos.branch waterfall

waterfall has been restarted

monitoring the builds closely to make sure our hack works, we should know shortly. the fizz builds are on two build slaves and working.

first attempt failed to limit the maximum time, we have a second attempt that is currently testing and will need to do another chromeos.branch waterfall restart

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/46e4eb04da8a5203a9792c76349f20d458bd3524

commit 46e4eb04da8a5203a9792c76349f20d458bd3524
Author: Gregory Meinke <gmeinke@google.com>
Date: Thu May 17 23:24:32 2018

I think that 2h maybe too aggressive and good branches that don't have the gsutil error are failing during build or test stages at the 2h interval. Maybe we should bump it up to 3h or even 4h.

Decreasing to P1. We are getting builds completed now, but still an issue.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/e18de1fb038bab08d00b344bffa664c5bb7d86a2

commit e18de1fb038bab08d00b344bffa664c5bb7d86a2
Author: Gregory Meinke <gmeinke@google.com>
Date: Fri May 18 15:31:53 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/5f87d7f182d1da853eb99305c97a468b2586803e

commit 5f87d7f182d1da853eb99305c97a468b2586803e
Author: Gregory Meinke <gmeinke@google.com>
Date: Fri May 18 19:37:23 2018

OTOH, just wondering if we can update the gsutil on some branches that have more builders associate with them first (e.g. firmware-strago-7287.B, firmware-glados-7820.B, firmware-veyron-6588.B)? That will help reduce the numbers of GS access error faster, IMHO.
Thanks. 

Yes, go for it.

temporarily assigning to Lann who is testing a fix in cbuildbot_launcher.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ebae73d32af06e32ae73788c815f8958303c2b08

commit ebae73d32af06e32ae73788c815f8958303c2b08
Author: Lann Martin <lannm@chromium.org>
Date: Wed May 23 15:17:07 2018

Fix certificate errors for old gsutils/boto versions

This bundles a recent version of cacerts.txt that is injected into the
cbuildbot environment for boto by cbuildbot_launch.

BUG= chromium:839949 
TEST=none

Change-Id: Id3b341db1cf0873f974d01da4ce3db9dc63ce8cc
Reviewed-on: https://chromium-review.googlesource.com/1068111
Reviewed-by: Lann Martin <lannm@chromium.org>
Tested-by: Lann Martin <lannm@chromium.org>

[add] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/third_party/boto/README.chromite
[add] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/third_party/boto/boto/cacerts/cacerts.txt
[add] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/lib/boto_compat_unittest.py
[add] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/third_party/boto/LICENSE
[add] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/lib/boto_compat.py
[modify] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/scripts/cbuildbot_launch.py
[add] https://crrev.com/ebae73d32af06e32ae73788c815f8958303c2b08/lib/boto_compat_unittest

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/17c0de834972eb8dafbce519651c19d7094a1c22

commit 17c0de834972eb8dafbce519651c19d7094a1c22
Author: chromite-chromium-autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed May 23 16:26:34 2018

Roll src/third_party/chromite/ 0e55e6fd1..ebae73d32 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/0e55e6fd174f..ebae73d32af0

$ git log 0e55e6fd1..ebae73d32 --date=short --no-merges --format='%ad %ae %s'
2018-05-21 lannm Fix certificate errors for old gsutils/boto versions

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:839949 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I33a36b59acdbace2b098e80aad3c82e3bb4a82b7
Reviewed-on: https://chromium-review.googlesource.com/1070313
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#561114}
[modify] https://crrev.com/17c0de834972eb8dafbce519651c19d7094a1c22/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8c4c680713d82d8a5c9ac7777319fa9fced9cff6

commit 8c4c680713d82d8a5c9ac7777319fa9fced9cff6
Author: Lann Martin <lannm@chromium.org>
Date: Wed May 23 17:28:22 2018

Use different conditional logic for FixBotoCerts.

The environment-based logic didn't work. We'll try using the explicit
script args to apply the fix only for old factory and firmware branches.

BUG= chromium:839949 
TEST=cbuildbot_launch eve-firmware --branch=firmware-eve-9584.B \
     --buildroot=~/tmp/cbuild

Change-Id: Iff0c1b0b6b94983a81352b1e82ab23d2e6c75e3e
Reviewed-on: https://chromium-review.googlesource.com/1070336
Reviewed-by: Lann Martin <lannm@chromium.org>
Tested-by: Lann Martin <lannm@chromium.org>

[modify] https://crrev.com/8c4c680713d82d8a5c9ac7777319fa9fced9cff6/lib/boto_compat.py
[modify] https://crrev.com/8c4c680713d82d8a5c9ac7777319fa9fced9cff6/scripts/cbuildbot_launch.py

 Issue 841258  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/52a752e403b6a99e43856e08a78cd49bc7eac489

commit 52a752e403b6a99e43856e08a78cd49bc7eac489
Author: chromite-chromium-autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed May 23 18:45:11 2018

Roll src/third_party/chromite/ ebae73d32..8c4c68071 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/ebae73d32af0..8c4c680713d8

$ git log ebae73d32..8c4c68071 --date=short --no-merges --format='%ad %ae %s'
2018-05-23 lannm Use different conditional logic for FixBotoCerts.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:839949 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I1002f1400440f99179d9488749d9b5bf9f54ad76
Reviewed-on: https://chromium-review.googlesource.com/1067683
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#561179}
[modify] https://crrev.com/52a752e403b6a99e43856e08a78cd49bc7eac489/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/eb5ce7b182c36a143e44ce13baf678e7af4dd7d9

commit eb5ce7b182c36a143e44ce13baf678e7af4dd7d9
Author: Lann Martin <lannm@chromium.org>
Date: Wed May 23 19:40:53 2018

Account for hard-coded BOTO_CONFIG in FixBotoCerts.

FixBotoCerts works by setting BOTO_PATH to a list of boto config files,
which are all applied. The problem is that if BOTO_CONFIG is set, it is
used instead of BOTO_PATH. Deal with this by moving the BOTO_CONFIG
value into BOTO_PATH if present.

BUG= chromium:839949 
TEST=cbuildbot_launch eve-firmware --branch=firmware-eve-9584.B \
     --buildroot=~/tmp/cbuild

Change-Id: I7997e04976a926a0f17df2c91dfbf7498c3acabd
Reviewed-on: https://chromium-review.googlesource.com/1070478
Reviewed-by: Mike Nichols <mikenichols@chromium.org>
Tested-by: Lann Martin <lannm@chromium.org>

[modify] https://crrev.com/eb5ce7b182c36a143e44ce13baf678e7af4dd7d9/lib/boto_compat.py
[modify] https://crrev.com/eb5ce7b182c36a143e44ce13baf678e7af4dd7d9/lib/boto_compat_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/02f8ae585af8de33cfb1c5dcb1ee722eb22c56ff

commit 02f8ae585af8de33cfb1c5dcb1ee722eb22c56ff
Author: Lann Martin <lannm@chromium.org>
Date: Wed May 23 20:37:46 2018

Deal with hard-coded BOTO_CONFIG default in FixBotoCerts.

Previously we tried to use BOTO_PATH to change the boto config, but the
gs module will always set a BOTO_CONFIG file if one isn't passed. Deal
with this by merging boto config files ourself and pointing BOTO_CONFIG
at this new merged config.

BUG= chromium:839949 
TEST=cbuildbot_launch eve-firmware --branch=firmware-eve-9584.B \
     --buildroot=~/tmp/cbuild

Change-Id: Ie92e86f5c2b9ae9d9c62837d35f6c5141d366c48
Reviewed-on: https://chromium-review.googlesource.com/1070569
Reviewed-by: Lann Martin <lannm@chromium.org>
Tested-by: Lann Martin <lannm@chromium.org>

[modify] https://crrev.com/02f8ae585af8de33cfb1c5dcb1ee722eb22c56ff/lib/boto_compat.py
[modify] https://crrev.com/02f8ae585af8de33cfb1c5dcb1ee722eb22c56ff/lib/boto_compat_unittest.py

This appears to be fixed now, at least for eve-firmware.

Note that there may be issues with re-running failed builds due to old 'builder-status' files in GS. If needed we can probably bulk delete these files.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/271ca3738e0dd09f784da2b3ef12448515cfb0be

commit 271ca3738e0dd09f784da2b3ef12448515cfb0be
Author: chromite-chromium-autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed May 23 21:39:17 2018

Roll src/third_party/chromite/ 12a27e545..eb5ce7b18 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/12a27e5450a7..eb5ce7b182c3

$ git log 12a27e545..eb5ce7b18 --date=short --no-merges --format='%ad %ae %s'
2018-05-23 lannm Account for hard-coded BOTO_CONFIG in FixBotoCerts.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:839949 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ic55f0d2d3667c0d1388953d40e5ced8204141d4d
Reviewed-on: https://chromium-review.googlesource.com/1070542
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#561261}
[modify] https://crrev.com/271ca3738e0dd09f784da2b3ef12448515cfb0be/DEPS

To clarify: the above fix is currently limited to factory and firmware branches <= 9667. I made this constraint because the fix is a pretty invasive hack and I wanted to protect newer branches from accidental/future breakage. All of this should go away as the CI team unbranches CI code.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d182a9ee9ddc38434c594dc0c30ca26eb7269bbc

commit d182a9ee9ddc38434c594dc0c30ca26eb7269bbc
Author: chromite-chromium-autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed May 23 22:56:33 2018

Roll src/third_party/chromite/ eb5ce7b18..02f8ae585 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/eb5ce7b182c3..02f8ae585af8

$ git log eb5ce7b18..02f8ae585 --date=short --no-merges --format='%ad %ae %s'
2018-05-23 lannm Deal with hard-coded BOTO_CONFIG default in FixBotoCerts.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:839949 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I8acf9eb459085252096d0808f2638dc1e4f6efe5
Reviewed-on: https://chromium-review.googlesource.com/1070708
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#561295}
[modify] https://crrev.com/d182a9ee9ddc38434c594dc0c30ca26eb7269bbc/DEPS

It seems the GS access problem still remains but at the later stage?

For example, 

https://uberchromegw.corp.google.com/i/chromeos.branch/builders/strago%20firmware%20firmware-strago-7287.B/builds/426

seems to be failing at InitSDK stage,

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos.branch%2Fstrago_firmware_firmware-strago-7287.B%2F426%2F%2B%2Frecipes%2Fsteps%2FInitSDK%2F0%2Fstdout

Thanks.

Please continue to cherry-pick the gsutil upgrade into old branches for a 100% certain fix. The hack is a hack; it was a nightmare to engineer it and it can't be guaranteed to work everywhere.

RE: #101

Somehow the build has gone through (but took more time than before), though.

The original builds should be fixed. As mentioned in #102 this will likely continue to be an issue in various places, e.g.: crbug.com/847676