TestSimpleChromeWorkflow is failing in the PFQ |
||||||||||
Issue descriptionTestSimpleChromeWorkflow started failing early today in all builders. The failures seem to be specific to running Simple Chrome with a local chroot; Simple Chrome from a chrome checkout using prebuilts seems to be working OK. Output from: https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/10532/steps/TestSimpleChromeWorkflow/logs/stdio 11:39:53: INFO: Running cidb query on pid 34677, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f8ddece9090> Preconditions for the stage successfully met. Beginning to execute stage... 11:39:53: INFO: Running cidb query on pid 34677, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f8ddecf3cd0> 11:39:53: INFO: RunCommand: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache chrome-sdk --board amd64-generic --cwd /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1 --nostart-goma --gomadir /b/c/goma_client --chrome-src /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src -- true in /b/c/cbuild/repository 11:39:54: DEBUG: Configured cache_dir to '/tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache' 11:39:54: DEBUG: RunCommand: curl --fail https://storage.googleapis.com/chromeos-mirror/gentoo/distfiles/gsutil_4.19.tar.gz -o /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/staging/tarball-cacheYQHDwn/gsutil_4.19.tar.gz 11:39:54: DEBUG: RunCommand: tar -xpf /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/staging/tarball-cacheYQHDwn/gsutil_4.19.tar.gz in /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/staging/tarball-cacheYQHDwn/extract 11:39:54: DEBUG: Attempting to compile local crcmod for gsutil 11:39:54: DEBUG: RunCommand: python ./setup.py build --build-base /tmp/cbuildbot-tmpmj5Egl/chromite.gsutil.crcmodFvFpGe --build-platlib /tmp/cbuildbot-tmpmj5Egl/chromite.gsutil.crcmodFvFpGe in /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/gsutil_4.19.tar.gz/gsutil/third_party/crcmod 11:39:54: INFO: Could not read /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1/metadata.json, falling back to /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1/partial-metadata.json 11:39:54: ERROR: cros chrome-sdk failed before completing. cros: Unhandled exception: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/bin/cros", line 168, in <module> DoMain() File "/b/c/cbuild/repository/chromite/bin/cros", line 164, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 910, in ScriptWrapperMain ret = target(argv[1:]) File "/b/c/cbuild/repository/chromite/scripts/cros.py", line 65, in main code = _RunSubCommand(subcommand) File "/b/c/cbuild/repository/chromite/scripts/cros.py", line 50, in _RunSubCommand return subcommand.Run() File "/b/c/cbuild/repository/chromite/cli/cros/cros_chrome_sdk.py", line 1019, in Run toolchain_url=self.options.toolchain_url) as ctx: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/b/c/cbuild/repository/chromite/cli/cros/cros_chrome_sdk.py", line 379, in Prepare self.toolchain_path, toolchain_url % {'target': target_tc}) KeyError: u'version' @@@STEP_FAILURE@@@ 11:39:54: ERROR: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 693, in Run self.PerformStage() File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 296, in PerformStage self._BuildChrome(sdk_cmd, goma) File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 243, in _BuildChrome sdk_cmd.Run(['true']) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 2858, in Run cros_build_lib.RunCommand(cros_cmd, cwd=self.cwd, **run_args) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 653, in RunCommand raise RunCommandError(msg, cmd_result) RunCommandError: return code: 1; command: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache chrome-sdk --board amd64-generic --cwd /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1 --nostart-goma --gomadir /b/c/goma_client --chrome-src /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src -- true cmd=['cros', '--log-level', 'debug', '--cache-dir', '/tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache', 'chrome-sdk', '--board', u'amd64-generic', '--cwd', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--sdk-path', u'/b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1', '--nostart-goma', '--gomadir', '/b/c/goma_client', '--chrome-src', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--', 'true'], cwd=/b/c/cbuild/repository
,
Sep 1 2017
For best results, CrOS infra bugs need to be filed against component Infra>Client>ChromeOS. Better still, use go/cros-infra-bug.
,
Sep 1 2017
I wasn't 100% certain this was an infra issue and I know that debugging Simple Chrome is something of a lost art. Anyway, I discovered the problem: In chromeos-image-archive for 9900: https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/amd64-generic-chromium-pfq/R62-9900.0.0-rc1/?authuser=0 The metadata.json files have only %(target) in the string: "toolchain-url": "2017/08/%(target)s-2017.08.30.175308.tar.xz" However, in 9901: https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/amd64-generic-chromium-pfq/R62-9901.0.0-rc1/?authuser=0 The matadata.json files also have %(version): "toolchain-url": "2017/08/%%(target)s-%(version)s.tar.xz" cros_chrome_sdk.py is only providing 'target' in the dictionary used to expand the string: https://cs.chromium.org/chromium/src/third_party/chromite/cli/cros/cros_chrome_sdk.py?q=cros_chrome_sdk.py&sq=package:chromium&dr&l=379
,
Sep 1 2017
sounds like https://chromium-review.googlesource.com/615224 we probably want to revert that now as i think Hidehiko has gone home for the weekend
,
Sep 1 2017
we can manually fix the toolchain config file in the meantime and chump that through and then kill the current sdk bot run so the PFQs should recover
,
Sep 1 2017
I am OK with babysitting the PFQ over the long weekend, I have an unrelated change I am trying to get to work on it. If someone can revert the CL and/or fix the toolchain config at some point today (before 8 pm PST) that's when the next scheduled PFQ run is and that will be sufficient. Thanks!
,
Sep 1 2017
https://chromium-review.googlesource.com/648269 should recover TestSimpleChromeWorkflow i just killed the current chromiumos-sdk bot so it'll delay pushing another bad setting :).
,
Sep 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/46a21764952735d11028bea30fad55982d426f40 commit 46a21764952735d11028bea30fad55982d426f40 Author: Mike Frysinger <vapier@chromium.org> Date: Fri Sep 01 23:04:01 2017 sdk_version.conf: manually recover from bad push BUG= chromium:761496 TEST=None Change-Id: Iefbf2b31c9368e8ad703b14ecb4c1fca1e5fc930 Reviewed-on: https://chromium-review.googlesource.com/648269 Reviewed-by: Steven Bennetts <stevenjb@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/46a21764952735d11028bea30fad55982d426f40/chromeos/binhost/host/sdk_version.conf
,
Sep 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/de798ca14f39c39e64665dcfe0c9143ac234cdbb commit de798ca14f39c39e64665dcfe0c9143ac234cdbb Author: Mike Frysinger <vapier@chromium.org> Date: Sat Sep 02 17:29:01 2017 sdk_version.conf: manually recover from bad push BUG= chromium:761496 TEST=None Change-Id: Ic471ccc7cd413c4814404897965f8487e816e696 Reviewed-on: https://chromium-review.googlesource.com/648280 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/de798ca14f39c39e64665dcfe0c9143ac234cdbb/chromeos/binhost/host/sdk_version.conf
,
Sep 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ea8b11f3c9b253c4db6564fba2e1c5e4d4a2a6d6 commit ea8b11f3c9b253c4db6564fba2e1c5e4d4a2a6d6 Author: Mike Frysinger <vapier@chromium.org> Date: Sun Sep 03 23:22:28 2017 sdk_version.conf: manually recover from bad push BUG= chromium:761496 TEST=None Change-Id: I70d03a4f6e169e05aec3a5958c12628e61da8637 Reviewed-on: https://chromium-review.googlesource.com/648846 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/ea8b11f3c9b253c4db6564fba2e1c5e4d4a2a6d6/chromeos/binhost/host/sdk_version.conf
,
Sep 4 2017
Sorry for breakage. Fix is sent out to review. https://chromium-review.googlesource.com/c/chromiumos/chromite/+/647390
,
Sep 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/704272a72674fe140308060727ef56c4bd70aec0 commit 704272a72674fe140308060727ef56c4bd70aec0 Author: Mike Frysinger <vapier@chromium.org> Date: Mon Sep 04 13:50:38 2017 sdk_version.conf: fix typo in previous commit BUG= chromium:761496 TEST=None Change-Id: I13b83028f2582621abf97b4031cca82d392e93f7 Reviewed-on: https://chromium-review.googlesource.com/648792 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/704272a72674fe140308060727ef56c4bd70aec0/chromeos/binhost/host/sdk_version.conf
,
Sep 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/130767eef3b320b0e9a352a1fc749188d0824e01 commit 130767eef3b320b0e9a352a1fc749188d0824e01 Author: Hidehiko Abe <hidehiko@chromium.org> Date: Mon Sep 04 18:43:27 2017 Fix TC_PATH in SDK uprev commit. TC_PATH here needs to be filled by "version". BUG= chromium:761496 TEST=Ran unittest locally. Ran on bots. Change-Id: I055601d0c68df56bdcc7f28620ac221027eb8e85 Reviewed-on: https://chromium-review.googlesource.com/647390 Commit-Ready: Hidehiko Abe <hidehiko@chromium.org> Tested-by: Hidehiko Abe <hidehiko@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/130767eef3b320b0e9a352a1fc749188d0824e01/cbuildbot/stages/sdk_stages.py [modify] https://crrev.com/130767eef3b320b0e9a352a1fc749188d0824e01/cbuildbot/stages/sdk_stages_unittest.py
,
Sep 5 2017
Looks like the PFQ succeeded for a while but is now failing again: https://bugs.chromium.org/p/chromium/issues/detail?id=761907 I just discovered this issue (I'm the gardener this week).
,
Sep 5 2017
the latest SDK bot run fixed this again
,
Sep 5 2017
,
Sep 5 2017
,
Sep 5 2017
go/simplechrome is failing in a chrome ToT checkout, see issue 762169
,
Sep 5 2017
This was failing on a bunch of R62 branch builders, do we need to pick into the 62 release branch?
,
Sep 5 2017
i've cherry picked it to R62 now too
,
Sep 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/55ed632081d8a73c8a0f37f3d863cb94e70792d3 commit 55ed632081d8a73c8a0f37f3d863cb94e70792d3 Author: Mike Frysinger <vapier@chromium.org> Date: Tue Sep 05 22:00:10 2017 sdk_version.conf: manually recover from bad push BUG= chromium:761496 TEST=None Change-Id: Iefbf2b31c9368e8ad703b14ecb4c1fca1e5fc930 (cherry picked from commit 46a21764952735d11028bea30fad55982d426f40) Reviewed-on: https://chromium-review.googlesource.com/651646 Reviewed-by: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/55ed632081d8a73c8a0f37f3d863cb94e70792d3/chromeos/binhost/host/sdk_version.conf
,
Sep 5 2017
,
Sep 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/ef2b001d9caec50ccf8738bd3c287cd610d4c50f commit ef2b001d9caec50ccf8738bd3c287cd610d4c50f Author: Hidehiko Abe <hidehiko@chromium.org> Date: Wed Sep 06 00:10:32 2017 Fix TC_PATH in SDK uprev commit. TC_PATH here needs to be filled by "version". BUG= chromium:761496 TEST=Ran unittest locally. Ran on bots. Change-Id: I055601d0c68df56bdcc7f28620ac221027eb8e85 (cherry picked from commit 130767eef3b320b0e9a352a1fc749188d0824e01) Reviewed-on: https://chromium-review.googlesource.com/651686 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> Commit-Queue: Bernie Thompson <bhthompson@chromium.org> Tested-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/ef2b001d9caec50ccf8738bd3c287cd610d4c50f/cbuildbot/stages/sdk_stages.py [modify] https://crrev.com/ef2b001d9caec50ccf8738bd3c287cd610d4c50f/cbuildbot/stages/sdk_stages_unittest.py
,
Sep 6 2017
Issue 762424 has been merged into this issue.
,
Sep 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/19b6932663ad4a39622231cf8a25b53fe215a4c9 commit 19b6932663ad4a39622231cf8a25b53fe215a4c9 Author: Charlie Andrews <charliea@chromium.org> Date: Thu Sep 14 22:42:48 2017 Migrate the PowerTracingAgent away from using BrowserThread::FILE This is part of a broader migration (see http://bit.ly/2f8ookS), after which BrowserThread::FILE will be no more. Bug: 761496 , 667892 Change-Id: I7a6542091a86e6e84b0f83dd799567232473a699 Reviewed-on: https://chromium-review.googlesource.com/665386 Commit-Queue: Charlie Andrews <charliea@chromium.org> Reviewed-by: Francois Doray <fdoray@chromium.org> Reviewed-by: oysteine <oysteine@chromium.org> Cr-Commit-Position: refs/heads/master@{#502054} [modify] https://crrev.com/19b6932663ad4a39622231cf8a25b53fe215a4c9/content/browser/tracing/power_tracing_agent.cc [modify] https://crrev.com/19b6932663ad4a39622231cf8a25b53fe215a4c9/content/browser/tracing/power_tracing_agent.h
,
Jan 22 2018
,
Jan 23 2018
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by steve...@chromium.org
, Sep 1 2017