New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 761496 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug

Blocking:
issue 761907
issue 762169



Sign in to add a comment

TestSimpleChromeWorkflow is failing in the PFQ

Project Member Reported by steve...@chromium.org, Sep 1 2017

Issue description

TestSimpleChromeWorkflow started failing early today in all builders.

The failures seem to be specific to running Simple Chrome with a local chroot; Simple Chrome from a chrome checkout using prebuilts seems to be working OK.

Output from: https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/10532/steps/TestSimpleChromeWorkflow/logs/stdio

11:39:53: INFO: Running cidb query on pid 34677, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f8ddece9090>
Preconditions for the stage successfully met. Beginning to execute stage...
11:39:53: INFO: Running cidb query on pid 34677, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f8ddecf3cd0>
11:39:53: INFO: RunCommand: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache chrome-sdk --board amd64-generic --cwd /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1 --nostart-goma --gomadir /b/c/goma_client --chrome-src /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src -- true in /b/c/cbuild/repository
11:39:54: DEBUG: Configured cache_dir to '/tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache'
11:39:54: DEBUG: RunCommand: curl --fail https://storage.googleapis.com/chromeos-mirror/gentoo/distfiles/gsutil_4.19.tar.gz -o /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/staging/tarball-cacheYQHDwn/gsutil_4.19.tar.gz
11:39:54: DEBUG: RunCommand: tar -xpf /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/staging/tarball-cacheYQHDwn/gsutil_4.19.tar.gz in /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/staging/tarball-cacheYQHDwn/extract
11:39:54: DEBUG: Attempting to compile local crcmod for gsutil
11:39:54: DEBUG: RunCommand: python ./setup.py build --build-base /tmp/cbuildbot-tmpmj5Egl/chromite.gsutil.crcmodFvFpGe --build-platlib /tmp/cbuildbot-tmpmj5Egl/chromite.gsutil.crcmodFvFpGe in /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache/common/gsutil_4.19.tar.gz/gsutil/third_party/crcmod
11:39:54: INFO: Could not read /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1/metadata.json, falling back to /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1/partial-metadata.json
11:39:54: ERROR: cros chrome-sdk failed before completing.
cros: Unhandled exception:
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/bin/cros", line 168, in <module>
    DoMain()
  File "/b/c/cbuild/repository/chromite/bin/cros", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 910, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/c/cbuild/repository/chromite/scripts/cros.py", line 65, in main
    code = _RunSubCommand(subcommand)
  File "/b/c/cbuild/repository/chromite/scripts/cros.py", line 50, in _RunSubCommand
    return subcommand.Run()
  File "/b/c/cbuild/repository/chromite/cli/cros/cros_chrome_sdk.py", line 1019, in Run
    toolchain_url=self.options.toolchain_url) as ctx:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/c/cbuild/repository/chromite/cli/cros/cros_chrome_sdk.py", line 379, in Prepare
    self.toolchain_path, toolchain_url % {'target': target_tc})
KeyError: u'version'

@@@STEP_FAILURE@@@
11:39:54: ERROR: Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 693, in Run
    self.PerformStage()
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 296, in PerformStage
    self._BuildChrome(sdk_cmd, goma)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 243, in _BuildChrome
    sdk_cmd.Run(['true'])
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 2858, in Run
    cros_build_lib.RunCommand(cros_cmd, cwd=self.cwd, **run_args)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 653, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache chrome-sdk --board amd64-generic --cwd /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1 --nostart-goma --gomadir /b/c/goma_client --chrome-src /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src -- true
cmd=['cros', '--log-level', 'debug', '--cache-dir', '/tmp/cbuildbot-tmpmj5Egl/chrome-sdk-cacheJqH8Wk/cache', 'chrome-sdk', '--board', u'amd64-generic', '--cwd', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--sdk-path', u'/b/c/cbuild/repository/buildbot_archive/amd64-generic-chromium-pfq/R63-9902.0.0-rc1', '--nostart-goma', '--gomadir', '/b/c/goma_client', '--chrome-src', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--', 'true'], cwd=/b/c/cbuild/repository


 
Cc: vapier@chromium.org
+vapier@

Components: Infra>Client>ChromeOS
Owner: ayatane@chromium.org
Status: Assigned (was: Untriaged)
For best results, CrOS infra bugs need to be filed against
component Infra>Client>ChromeOS.  Better still, use go/cros-infra-bug.

I wasn't 100% certain this was an infra issue and I know that debugging Simple Chrome is something of a lost art.

Anyway, I discovered the problem:

In chromeos-image-archive for 9900:
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/amd64-generic-chromium-pfq/R62-9900.0.0-rc1/?authuser=0

The metadata.json files have only %(target) in the string:
"toolchain-url": "2017/08/%(target)s-2017.08.30.175308.tar.xz"

However, in 9901:
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/amd64-generic-chromium-pfq/R62-9901.0.0-rc1/?authuser=0

The matadata.json files also have %(version):
"toolchain-url": "2017/08/%%(target)s-%(version)s.tar.xz"

cros_chrome_sdk.py is only providing 'target' in the dictionary used to expand the string:
https://cs.chromium.org/chromium/src/third_party/chromite/cli/cros/cros_chrome_sdk.py?q=cros_chrome_sdk.py&sq=package:chromium&dr&l=379

Cc: hidehiko@chromium.org
sounds like https://chromium-review.googlesource.com/615224

we probably want to revert that now as i think Hidehiko has gone home for the weekend
we can manually fix the toolchain config file in the meantime and chump that through and then kill the current sdk bot run so the PFQs should recover
I am OK with babysitting the PFQ over the long weekend, I have an unrelated change I am trying to get to work on it.

If someone can revert the CL and/or fix the toolchain config at some point today (before 8 pm PST) that's when the next scheduled PFQ run is and that will be sufficient.

Thanks!

https://chromium-review.googlesource.com/648269 should recover TestSimpleChromeWorkflow

i just killed the current chromiumos-sdk bot so it'll delay pushing another bad setting :).
Project Member

Comment 8 by bugdroid1@chromium.org, Sep 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/46a21764952735d11028bea30fad55982d426f40

commit 46a21764952735d11028bea30fad55982d426f40
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Sep 01 23:04:01 2017

sdk_version.conf: manually recover from bad push

BUG= chromium:761496 
TEST=None

Change-Id: Iefbf2b31c9368e8ad703b14ecb4c1fca1e5fc930
Reviewed-on: https://chromium-review.googlesource.com/648269
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/46a21764952735d11028bea30fad55982d426f40/chromeos/binhost/host/sdk_version.conf

Project Member

Comment 9 by bugdroid1@chromium.org, Sep 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/de798ca14f39c39e64665dcfe0c9143ac234cdbb

commit de798ca14f39c39e64665dcfe0c9143ac234cdbb
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Sep 02 17:29:01 2017

sdk_version.conf: manually recover from bad push

BUG= chromium:761496 
TEST=None

Change-Id: Ic471ccc7cd413c4814404897965f8487e816e696
Reviewed-on: https://chromium-review.googlesource.com/648280
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/de798ca14f39c39e64665dcfe0c9143ac234cdbb/chromeos/binhost/host/sdk_version.conf

Project Member

Comment 10 by bugdroid1@chromium.org, Sep 3 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ea8b11f3c9b253c4db6564fba2e1c5e4d4a2a6d6

commit ea8b11f3c9b253c4db6564fba2e1c5e4d4a2a6d6
Author: Mike Frysinger <vapier@chromium.org>
Date: Sun Sep 03 23:22:28 2017

sdk_version.conf: manually recover from bad push

BUG= chromium:761496 
TEST=None

Change-Id: I70d03a4f6e169e05aec3a5958c12628e61da8637
Reviewed-on: https://chromium-review.googlesource.com/648846
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/ea8b11f3c9b253c4db6564fba2e1c5e4d4a2a6d6/chromeos/binhost/host/sdk_version.conf

Sorry for breakage.
Fix is sent out to review.
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/647390

Project Member

Comment 12 by bugdroid1@chromium.org, Sep 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/704272a72674fe140308060727ef56c4bd70aec0

commit 704272a72674fe140308060727ef56c4bd70aec0
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Sep 04 13:50:38 2017

sdk_version.conf: fix typo in previous commit

BUG= chromium:761496 
TEST=None

Change-Id: I13b83028f2582621abf97b4031cca82d392e93f7
Reviewed-on: https://chromium-review.googlesource.com/648792
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/704272a72674fe140308060727ef56c4bd70aec0/chromeos/binhost/host/sdk_version.conf

Project Member

Comment 13 by bugdroid1@chromium.org, Sep 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/130767eef3b320b0e9a352a1fc749188d0824e01

commit 130767eef3b320b0e9a352a1fc749188d0824e01
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Mon Sep 04 18:43:27 2017

Fix TC_PATH in SDK uprev commit.

TC_PATH here needs to be filled by "version".

BUG= chromium:761496 
TEST=Ran unittest locally. Ran on bots.

Change-Id: I055601d0c68df56bdcc7f28620ac221027eb8e85
Reviewed-on: https://chromium-review.googlesource.com/647390
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/130767eef3b320b0e9a352a1fc749188d0824e01/cbuildbot/stages/sdk_stages.py
[modify] https://crrev.com/130767eef3b320b0e9a352a1fc749188d0824e01/cbuildbot/stages/sdk_stages_unittest.py

Blocking: 761907
Cc: achuith@chromium.org
Looks like the PFQ succeeded for a while but is now failing again:
https://bugs.chromium.org/p/chromium/issues/detail?id=761907

I just discovered this issue (I'm the gardener this week). 
Status: Fixed (was: Assigned)
the latest SDK bot run fixed this again
Blocking: -761907
Blocking: 761907
 Issue 761907  has been merged into this issue.
go/simplechrome is failing in a chrome ToT checkout, see  issue 762169 

This was failing on a bunch of R62 branch builders, do we need to pick into the 62 release branch?
i've cherry picked it to R62 now too
Project Member

Comment 21 by bugdroid1@chromium.org, Sep 5 2017

Labels: merge-merged-release-R62-9901.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/55ed632081d8a73c8a0f37f3d863cb94e70792d3

commit 55ed632081d8a73c8a0f37f3d863cb94e70792d3
Author: Mike Frysinger <vapier@chromium.org>
Date: Tue Sep 05 22:00:10 2017

sdk_version.conf: manually recover from bad push

BUG= chromium:761496 
TEST=None

Change-Id: Iefbf2b31c9368e8ad703b14ecb4c1fca1e5fc930
(cherry picked from commit 46a21764952735d11028bea30fad55982d426f40)
Reviewed-on: https://chromium-review.googlesource.com/651646
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/55ed632081d8a73c8a0f37f3d863cb94e70792d3/chromeos/binhost/host/sdk_version.conf

Blocking: 762169
Project Member

Comment 23 by bugdroid1@chromium.org, Sep 6 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ef2b001d9caec50ccf8738bd3c287cd610d4c50f

commit ef2b001d9caec50ccf8738bd3c287cd610d4c50f
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Wed Sep 06 00:10:32 2017

Fix TC_PATH in SDK uprev commit.

TC_PATH here needs to be filled by "version".

BUG= chromium:761496 
TEST=Ran unittest locally. Ran on bots.

Change-Id: I055601d0c68df56bdcc7f28620ac221027eb8e85
(cherry picked from commit 130767eef3b320b0e9a352a1fc749188d0824e01)
Reviewed-on: https://chromium-review.googlesource.com/651686
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/ef2b001d9caec50ccf8738bd3c287cd610d4c50f/cbuildbot/stages/sdk_stages.py
[modify] https://crrev.com/ef2b001d9caec50ccf8738bd3c287cd610d4c50f/cbuildbot/stages/sdk_stages_unittest.py

Issue 762424 has been merged into this issue.
Project Member

Comment 25 by bugdroid1@chromium.org, Sep 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19b6932663ad4a39622231cf8a25b53fe215a4c9

commit 19b6932663ad4a39622231cf8a25b53fe215a4c9
Author: Charlie Andrews <charliea@chromium.org>
Date: Thu Sep 14 22:42:48 2017

Migrate the PowerTracingAgent away from using BrowserThread::FILE

This is part of a broader migration (see http://bit.ly/2f8ookS), after
which BrowserThread::FILE will be no more.

Bug:  761496 ,  667892 
Change-Id: I7a6542091a86e6e84b0f83dd799567232473a699
Reviewed-on: https://chromium-review.googlesource.com/665386
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Reviewed-by: Francois Doray <fdoray@chromium.org>
Reviewed-by: oysteine <oysteine@chromium.org>
Cr-Commit-Position: refs/heads/master@{#502054}
[modify] https://crrev.com/19b6932663ad4a39622231cf8a25b53fe215a4c9/content/browser/tracing/power_tracing_agent.cc
[modify] https://crrev.com/19b6932663ad4a39622231cf8a25b53fe215a4c9/content/browser/tracing/power_tracing_agent.h

Comment 26 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Comment 27 by dchan@chromium.org, Jan 23 2018

Status: Fixed (was: Archived)

Sign in to add a comment