Chrome OS CQ builders failing because of missing chromeos-chrome prebuilts |
|||||||||||
Issue descriptionChrome OS CQ runs after https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/14701 are failing with the following error: @@@STEP_FAILURE@@@ 03:11:57: ERROR: Cannot find prebuilts for chromeos-base/chromeos-chrome on chell 03:11:57: INFO: Translating result Cannot find prebuilts for chromeos-base/chromeos-chrome on chell to fail. 03:11:57: INFO: Running cidb query on pid 24823, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f4b41629f90> 03:11:57: INFO: Running cidb query on pid 24823, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f4b4160a5d0>
,
May 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/9a66c4dd0d61dad05b3505c4be8f914f54fb7f20 commit 9a66c4dd0d61dad05b3505c4be8f914f54fb7f20 Author: Hidehiko Abe <hidehiko@chromium.org> Date: Wed May 24 11:02:09 2017 Revert "Enable goma on bots not using AFDO." This reverts commit 99841d812748bf1a6a7bc2097651fab576203d27. Reason for revert: paladin bots started to fail, because it fails to find the cache. BUG= chromium:725860 Original change's description: > Enable goma on bots not using AFDO. > > Now, the found errors for non-AFDO bots should be fixed. > Also, goma log uploading step is fixed so that logs should be > uploaded even in case of timeout. > > BUG=chromium:279618 > TEST=Ran bots. cbuildbot with chell-chrome-pfq manually. > > Change-Id: I467fb5cee98998f2e8393e9744ac50545fd5597a > Reviewed-on: https://chromium-review.googlesource.com/509411 > Commit-Ready: Hidehiko Abe <hidehiko@chromium.org> > Tested-by: Hidehiko Abe <hidehiko@chromium.org> > Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> > Reviewed-by: Xixuan Wu <xixuan@chromium.org> > BUG=chromium:279618 Change-Id: I53fd7b8ae673bad6c642be226b3af84bab5f140b Reviewed-on: https://chromium-review.googlesource.com/513664 Reviewed-by: Ryo Hashimoto <hashimoto@chromium.org> Tested-by: Hidehiko Abe <hidehiko@chromium.org> [modify] https://crrev.com/9a66c4dd0d61dad05b3505c4be8f914f54fb7f20/cbuildbot/stages/build_stages.py
,
May 24 2017
Reverted. So after next ChromePFQ success, palandin should be fixed. Let's see next cycle. akeshet@, BTW, how could we have tested this case...?
,
May 24 2017
mmm, I didn't understand this yet... could you explain?
,
May 24 2017
Re: #4. prebuilt was not found because of USE flag mismatch...
,
May 24 2017
We should pin chrome back to a working version. (we can unpin immediately after to allow the pfq to keep trying to uprev). But keeping the entire tree blocked while waiting for the pfq to uprev is too optimistic about the pfq's chance of success.
,
May 24 2017
Tree closed.
,
May 24 2017
,
May 24 2017
I think this is effecting the PCQ as well
,
May 24 2017
,
May 24 2017
Based on the timing, I believe breakage was introduced by https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4384 which is chrome 60.0.3109.0 . Therefore I suggest pinning back to the previous version 60.0.3107.0
,
May 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/8a2545ca6a106ba9cdc7b6db16420594e9f51454 commit 8a2545ca6a106ba9cdc7b6db16420594e9f51454 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed May 24 15:37:39 2017 Chrome: Pin to version 60.0.3107.0_rc-r1 DO NOT REVERT THIS CL. In general, reverting chrome (un)pin CLs does not do what you expect. Instead, use `cros pinchrome` to generate new CLs. BUG= 725860 TEST=None CQ-DEPEND=*I31847de19d4a168513965b19933afd4eb8b355db Change-Id: I9dd7e1105f4bd1bb9d207ec882e83d0a77c00c05 Reviewed-on: https://chromium-review.googlesource.com/513565 Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: David Riley <davidriley@chromium.org> [rename] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos-base/chromeos-chrome/chromeos-chrome-60.0.3107.0_rc-r1.ebuild [add] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/profiles/default/linux/package.mask/chromepin [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos-base/chromeos-chrome/Manifest [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/amd64-generic-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/daisy-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/arm-generic-LATEST_RELEASE_CHROME_BINHOST.conf [rename] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos-base/chromium-source/chromium-source-60.0.3107.0_rc-r1.ebuild [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/x86-generic-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/veyron_jerry-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/host/amd64-LATEST_RELEASE_CHROME_BINHOST.conf
,
May 24 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/overlays/chromeos-partner-overlay/+/0420be4ec197a2981e6234729dafa50b14719d5e commit 0420be4ec197a2981e6234729dafa50b14719d5e Author: Aviv Keshet <akeshet@chromium.org> Date: Wed May 24 15:37:45 2017
,
May 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/8a2545ca6a106ba9cdc7b6db16420594e9f51454 commit 8a2545ca6a106ba9cdc7b6db16420594e9f51454 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed May 24 15:37:39 2017 Chrome: Pin to version 60.0.3107.0_rc-r1 DO NOT REVERT THIS CL. In general, reverting chrome (un)pin CLs does not do what you expect. Instead, use `cros pinchrome` to generate new CLs. BUG= 725860 TEST=None CQ-DEPEND=*I31847de19d4a168513965b19933afd4eb8b355db Change-Id: I9dd7e1105f4bd1bb9d207ec882e83d0a77c00c05 Reviewed-on: https://chromium-review.googlesource.com/513565 Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: David Riley <davidriley@chromium.org> [rename] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos-base/chromeos-chrome/chromeos-chrome-60.0.3107.0_rc-r1.ebuild [add] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/profiles/default/linux/package.mask/chromepin [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos-base/chromeos-chrome/Manifest [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/amd64-generic-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/daisy-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/arm-generic-LATEST_RELEASE_CHROME_BINHOST.conf [rename] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos-base/chromium-source/chromium-source-60.0.3107.0_rc-r1.ebuild [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/x86-generic-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/target/veyron_jerry-LATEST_RELEASE_CHROME_BINHOST.conf [modify] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/chromeos/binhost/host/amd64-LATEST_RELEASE_CHROME_BINHOST.conf
,
May 24 2017
Chrome pinned. Now, unpinning.
,
May 24 2017
,
May 24 2017
heh oh yeah, there's only 1 CL needed for unpinning
,
May 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/58183306a645dfea60ed71c6afdad56580ec2da7 commit 58183306a645dfea60ed71c6afdad56580ec2da7 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed May 24 15:50:21 2017 Chrome: Unpin chrome DO NOT REVERT THIS CL. In general, reverting chrome (un)pin CLs does not do what you expect. Instead, use `cros pinchrome` to generate new CLs. BUG= 725860 TEST=None Change-Id: I7781ff206de0163984aaeec48837c4b7100cef88 Reviewed-on: https://chromium-review.googlesource.com/513438 Reviewed-by: David Riley <davidriley@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> [delete] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/profiles/default/linux/package.mask/chromepin
,
May 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/58183306a645dfea60ed71c6afdad56580ec2da7 commit 58183306a645dfea60ed71c6afdad56580ec2da7 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed May 24 15:50:21 2017 Chrome: Unpin chrome DO NOT REVERT THIS CL. In general, reverting chrome (un)pin CLs does not do what you expect. Instead, use `cros pinchrome` to generate new CLs. BUG= 725860 TEST=None Change-Id: I7781ff206de0163984aaeec48837c4b7100cef88 Reviewed-on: https://chromium-review.googlesource.com/513438 Reviewed-by: David Riley <davidriley@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> [delete] https://crrev.com/8a2545ca6a106ba9cdc7b6db16420594e9f51454/profiles/default/linux/package.mask/chromepin
,
May 24 2017
Let's turn this into discussion of how this slipped through. Normally the BinhostTest stage on pfq master is supposed to protect against this, but ensuring that the useflags set from the next version of chrome will match those needed by the paladins. Do we understand why #2 broke this? Did the builders fail to upload prebuilts? Did useflags changes in unpredicted ways that BinhostTest didn't know about?
,
May 24 2017
,
May 24 2017
Re #20; Great to know BinhostTest does test it. With enabling goma; - prebuilt is uploaded. The package has USE=goma, because chrome-pfq bots can use goma. - However, USE=goma is set in build_stages only if goma can be used (= Chrome is actually built and no AFDO (WIP bug)) https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/stages/build_stages.py?l=456 - On paladin, USE=goma is not set, because it fails to find prebuilt package, due to USE mismatch. - BinhostTest could not catch it, because USE flag is extracted from json dump. So, chrome-pfq's do not have it. So, how can we enable goma then? 1) If we set USE=goma for all bots regardless of whether chrome build is run, try/CQ won't pass, because prebuilt won't be found due to USE mismatch between master's requirement and prebuilt. So need chump? 2) If using another env var is another option, but it could be inconsistent with other use flags controlling toolchain, like clang. 3) Tweak portage system somehow to filter "goma" from USE when prebuilt look-up. Or, any other ideas? WDYT, akeshet@?
,
May 24 2017
,
May 24 2017
Yes, sounds like we need to do a combination of 3 or something like it, and blacklist goma from being inspected by binhost_test. +davidjames and +dgarrett have a better understanding than I of #3 and what is involved in that prebuilt lookup
,
May 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/138778ae42b621be12c8bbb9afe6ec496f4b3120 commit 138778ae42b621be12c8bbb9afe6ec496f4b3120 Author: Hidehiko Abe <hidehiko@chromium.org> Date: Mon May 29 05:39:13 2017 Look at USE_GOMA env var. Using USE variable to control goma triggered prebuilt-not-found error. This CL introduces USE_GOMA variable, also to control goma. BUG=chromium:279618, chromium:725860 TEST=Ran locally with USE_GOMA=true. Ran try. Change-Id: Id0f85f82efd99b062a05a3a6e4adc84bd9e32a59 Reviewed-on: https://chromium-review.googlesource.com/516644 Commit-Ready: Hidehiko Abe <hidehiko@chromium.org> Tested-by: Hidehiko Abe <hidehiko@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/138778ae42b621be12c8bbb9afe6ec496f4b3120/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/2ddf24348c2959b39034c169d824149cfee9dd95 commit 2ddf24348c2959b39034c169d824149cfee9dd95 Author: Hidehiko Abe <hidehiko@chromium.org> Date: Thu Jun 01 04:14:40 2017 Use USE_GOMA instead of USE. Using USE had a problem that prebuilt is not found from paladin. With this CL, bots start to use USE_GOMA instead. CQ-DEPEND=CL:516644 BUG=chromium:279618, chromium:725860 TEST=Ran cbuildbot. Change-Id: Iaa29f5301baf6cae4a34a5a4e962ae385adb70ba Reviewed-on: https://chromium-review.googlesource.com/516944 Commit-Ready: Hidehiko Abe <hidehiko@chromium.org> Tested-by: Hidehiko Abe <hidehiko@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/2ddf24348c2959b39034c169d824149cfee9dd95/cbuildbot/stages/build_stages.py
,
Jun 1 2017
Now the fix changes are all in, so closing.
,
Aug 1 2017
,
Jan 22 2018
|
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by hashimoto@chromium.org
, May 24 2017Status: Assigned (was: Available)