TestSimpleChromeWorkflow failed on multiple pfq builders |
|||||||||
Issue descriptionTestSimpleChromeWorkflow seems to failed on all the pfq builders for arm devices: bob,daisy, daisy_skate, hana, kevin-arcnext, nyan_big, peach_pit, scarlet, veyron_jerry, veyron_minnie, veyron_rialto Builder link: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8945654212669482992 Log: https://luci-logdog.appspot.com/v/?s=chromeos/bb/chromeos/daisy-chromium-pfq/12102/+/recipes/steps/TestSimpleChromeWorkflow/0/stdout 04:33:44: INFO: RunCommand: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache chrome-sdk --board arm-generic --cwd /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/c/cbuild/repository/buildbot_archive/arm-generic-chromium-pfq/R68-10714.0.0-rc7 --nogn-gen --nostart-goma --gomadir /b/c/goma_cache/client --chrome-src /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src -- /b/c/cbuild/repository/chromite/bin/deploy_chrome --build-dir /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release --staging-only --staging-dir /tmp/cbuildbot-tmpSadmar/chrome-sdk-stageDy4nPf in /b/c/cbuild/repository 04:33:45: DEBUG: Configured cache_dir to '/tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache' 04:33:46: INFO: Staging flags taken from USE in environment: highdpi deploy_chrome: Unhandled exception: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/bin/deploy_chrome", line 169, in <module> DoMain() File "/b/c/cbuild/repository/chromite/bin/deploy_chrome", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 911, in ScriptWrapperMain ret = target(argv[1:]) File "/b/c/cbuild/repository/chromite/scripts/deploy_chrome.py", line 679, in main deploy.Perform() File "/b/c/cbuild/repository/chromite/scripts/deploy_chrome.py", line 351, in Perform self._PrepareStagingDir() File "/b/c/cbuild/repository/chromite/scripts/deploy_chrome.py", line 325, in _PrepareStagingDir self.copy_paths, self.chrome_dir) File "/b/c/cbuild/repository/chromite/scripts/deploy_chrome.py", line 637, in _PrepareStagingDir strip_flags=strip_flags, copy_paths=copy_paths) File "/b/c/cbuild/repository/chromite/lib/chrome_util.py", line 438, in StageChromeFromBuildDir copied_paths += copier.Copy(build_dir, staging_dir, p, sloppy=sloppy) File "/b/c/cbuild/repository/chromite/lib/chrome_util.py", line 224, in Copy self._CopyFile(p, dest, path) File "/b/c/cbuild/repository/chromite/lib/chrome_util.py", line 153, in _CopyFile [self.strip_bin] + strip_flags + ['-o', dest, src]) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 688, in DebugRunCommand return RunCommand(*args, **kwargs) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 665, in RunCommand raise RunCommandError(msg, cmd_result) chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache/chrome-sdk/tarballs/arm-generic+__b__c__cbuild__repository__buildbot_archive__arm-generic-chromium-pfq__R68-10714.0.0-rc7+target_toolchain/bin/armv7a-cros-linux-gnueabi-strip --strip-unneeded -o /tmp/cbuildbot-tmpSadmar/chrome-sdk-stageDy4nPf/libmojo_core_arc64.so /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release/libmojo_core_arc64.so /tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache/chrome-sdk/tarballs/arm-generic+__b__c__cbuild__repository__buildbot_archive__arm-generic-chromium-pfq__R68-10714.0.0-rc7+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.27.0/strip.elf:/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release/libmojo_core_arc64.so: File format not recognized cmd=['/tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache/chrome-sdk/tarballs/arm-generic+__b__c__cbuild__repository__buildbot_archive__arm-generic-chromium-pfq__R68-10714.0.0-rc7+target_toolchain/bin/armv7a-cros-linux-gnueabi-strip', '--strip-unneeded', '-o', '/tmp/cbuildbot-tmpSadmar/chrome-sdk-stageDy4nPf/libmojo_core_arc64.so', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release/libmojo_core_arc64.so'] [1;31m04:33:48: ERROR: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 704, in Run self.PerformStage() File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 326, in PerformStage self._TestDeploy(sdk_cmd) File "/b/c/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 298, in _TestDeploy '--staging-only', '--staging-dir', tempdir]) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 2796, in Run return cros_build_lib.RunCommand(cros_cmd, cwd=self.cwd, **run_args) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 665, in RunCommand raise RunCommandError(msg, cmd_result) RunCommandError: return code: 1; command: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache chrome-sdk --board arm-generic --cwd /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/c/cbuild/repository/buildbot_archive/arm-generic-chromium-pfq/R68-10714.0.0-rc7 --nogn-gen --nostart-goma --gomadir /b/c/goma_cache/client --chrome-src /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src -- /b/c/cbuild/repository/chromite/bin/deploy_chrome --build-dir /b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release --staging-only --staging-dir /tmp/cbuildbot-tmpSadmar/chrome-sdk-stageDy4nPf cmd=['cros', '--log-level', 'debug', '--cache-dir', '/tmp/cbuildbot-tmpSadmar/chrome-sdk-cachedRNF5I/cache', 'chrome-sdk', '--board', u'arm-generic', '--cwd', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--sdk-path', u'/b/c/cbuild/repository/buildbot_archive/arm-generic-chromium-pfq/R68-10714.0.0-rc7', '--nogn-gen', '--nostart-goma', '--gomadir', '/b/c/goma_cache/client', '--chrome-src', '/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--', '/b/c/cbuild/repository/chromite/bin/deploy_chrome', '--build-dir', u'/b/c/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release', '--staging-only', '--staging-dir', '/tmp/cbuildbot-tmpSadmar/chrome-sdk-stageDy4nPf'], cwd=/b/c/cbuild/repository [0m +sheriffs +steven, any ideas about this?
,
May 24 2018
,
May 24 2018
Looks like this auto test platform_ToolchainOptions also needs to be updated with the new binary. 05/24 10:01:04.409 ERROR|platform_Toolchain:0321| Test LOAD Writable and Exec 2 failures FAILED: /opt/google/chrome/libmojo_core_arc32.so /opt/google/chrome/libmojo_core_arc64.so This failed on informational builders: veyron_minnie, tricky, peach_pit, caroline https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8945644207497157952
,
May 24 2018
I believe +bpastene@ was looking at this
,
May 24 2018
I was fixing something a little different. Looks like the binutils in chrome-sdk doesn't know how to strip the libmojo_core_arc* libs, but only on arm?
,
May 24 2018
If still investigating/a fix will not come soon, can we revert first since this is affecting quite a few pfq builders and pfq informational builders.
,
May 25 2018
,
May 25 2018
Thanks for the revert. I think we can probably just strip these binaries at build time using Android toolchain, and then tell chromite not to strip them. Sound ok?
,
May 25 2018
+yunlian from tool chain for inputs.
,
May 25 2018
,
May 28 2018
cros_vm_sanity_test has been failing a lot today on https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel
,
May 29 2018
Re #11: Thanks for the report. That seems unrelated here given that the libmojo change has been reverted, but something certainly regressed. I'll look into it. (Note that the test is experimental, so even if it fails, it won't turn the build red or block a CQ run.)
,
May 31 2018
Please remove the test ASAP if it does not block anything ? especially no one care if the test failed. This is causing every build to be red in chromeOS and require manual triage in order to figure which board is actually failed.
,
May 31 2018
,
May 31 2018
> This is causing every build to be red in chromeOS Can you please link to one of the builds? The original breaking change has been reverted (https://chromium-review.googlesource.com/1072927), so whatever failure your seeing is likely something else. And to be clear, my comment in #12 was referring to a test on a chromium bot, not the PFQ or anything in ChromeOS-land.
,
May 31 2018
here's the GE link https://cros-goldeneye.corp.google.com/chromeos/console/listBuild?milestone=68#%2F sponge link https://sponge.corp.google.com/invocations?searchFor=user:chromeos-test+build:buddy-release/R68-10718.8.0+suite:bvt-cq+platform_ToolchainOptions screenshot of Goldeneye https://screenshot.googleplex.com/7QX5OpYzj2E.png
,
May 31 2018
Looks like the error is indeed relevant here: AssertionError: File dir-ROOT-A/opt/google/chrome/libmojo_core_arc64.so contains unsatisfied symbols from https://luci-milo.appspot.com/buildbot/chromeos_release/daisy-release%20release-R68-10718.B/6 Is it just 68 that's failing? It looks like the revert was already cherry picked to 68 (https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1075987) so I'm not sure why it would still be failing. Someone more familiar with cros-land branch/merging or the PFQ will have to take a look.
,
May 31 2018
yes, looks like the problem only appear in M68. The CL you listed supposed to landed in 10718.6.0, we have build 2 more times since but test still failed. +bhthompson any idea ?
,
May 31 2018
bernie said https://bugs.chromium.org/p/chromium/issues/detail?id=848082 should fix this.
,
Jun 1 2018
Sounds like it's fixed now. Let's close this out. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by xiaoyinh@chromium.org
, May 24 2018