New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 906572 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Nov 21
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

BinhostTest failure: master-chromium-pfq test failed

Project Member Reported by zhuohao@google.com, Nov 19

Issue description

Link to build or pfq page.
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8929455626711957056/+/steps/BinhostTest/0/stdout

build # for that buildbot.
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929455626711957056

Snippet of log that contains the failure.
21:42:04: ERROR: When we take out config-requested useflags ('chrome_internal',) for public/partner builds, grunt-android-pi-pre-flight-branch cannot find Chrome prebuilts (probably due to USE flag mismatch)
Build settings: _CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_tests', 'buildcheck', 'cfi', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'grunt_march', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'thinlto', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-march=amdfam10', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))
Closest matching configs:
	Boards: set([_BoardKey(board='amd64-generic', useflags=())])
		USE changes: ['-grunt_march']
		Build settings: _CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_tests', 'buildcheck', 'cfi', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'grunt_march', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'thinlto', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-march=amdfam10', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))
	Boards: set([_BoardKey(board='betty-arcnext', useflags=('chrome_internal',)), _BoardKey(board='betty', useflags=('chrome_internal',))])
		USE changes: ['-grunt_march']
		Build settings: _CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_tests', 'buildcheck', 'cfi', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'grunt_march', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'thinlto', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-march=amdfam10', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))
	Boards: set([_BoardKey(board='caroline', useflags=('chrome_internal',)), _BoardKey(board='tricky', useflags=('chrome_internal',)), _BoardKey(board='reef', useflags=('chrome_internal',)), _BoardKey(board='eve', useflags=('chrome_internal',)), _BoardKey(board='caroline-arcnext', useflags=('chrome_internal',)), _BoardKey(board='cyan', useflags=('chrome_internal',)), _BoardKey(board='peppy', useflags=('chrome_internal',)), _BoardKey(board='coral', useflags=('chrome_internal',)), _BoardKey(board='nocturne', useflags=('chrome_internal',)), _BoardKey(board='eve-arcnext', useflags=('chrome_internal',)), _BoardKey(board='terra', useflags=('chrome_internal',))])
		USE changes: ['-grunt_march']
		Build settings: _CompatId(arch='amd64', useflags=('accessibility', 'authpolicy', 'autotest', 'build_tests', 'buildcheck', 'cfi', 'chrome_debug', 'chrome_remoting', 'clang', 'cups', 'debug_fission', 'evdev_gestures', 'fonts', 'gold', 'grunt_march', 'highdpi', 'libcxx', 'nacl', 'opengles', 'ozone_platform_default_gbm', 'ozone_platform_gbm', 'runhooks', 'smbprovider', 'thinlto', 'v4l2_codec', 'vaapi', 'xkbcommon'), cflags=('-O2', '-pipe', '-march=amdfam10', '-g', '-fno-exceptions', '-fno-unwind-tables', '-fno-asynchronous-unwind-tables'))

...

21:42:04: ERROR: When we take out config-requested useflags ('chrome_internal',) for public/partner builds, grunt-android-pi-pre-flight-branch-tryjob cannot find Chrome prebuilts (probably due to USE flag mismatch)
...
21:42:04: ERROR: When we take out config-requested useflags ('chrome_internal',) for public/partner builds, grunt-paladin cannot find Chrome prebuilts (probably due to USE flag mismatch)
...
21:42:04: ERROR: When we take out config-requested useflags ('chrome_internal',) for public/partner builds, grunt-pi-android-pfq cannot find Chrome prebuilts (probably due to USE flag mismatch)
...
21:42:04: ERROR: When we take out config-requested useflags ('chrome_internal',) for public/partner builds, grunt-postsubmit cannot find Chrome prebuilts (probably due to USE flag mismatch)
...
ERROR
[chromite.cbuildbot.binhost_test] PrebuiltCompatibilityTest.testDumping
Verify Chrome prebuilts exist for all configs that build Chrome. ... ok

======================================================================
ERROR: [chromite.cbuildbot.binhost_test] PrebuiltCompatibilityTest.testChromePrebuiltsPresent
Verify all builds that use Chrome have matching Chrome PFQ configs.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 276, in _stacked_tearDown
    target(obj)
  File "/mnt/host/source/chromite/cbuildbot/binhost_test.py", line 67, in tearDown
    self.fail('Fatal errors found in this test')
AssertionError: Fatal errors found in this test

----------------------------------------------------------------------
Ran 2 tests in 97.179s

FAILED (errors=1)
21:42:04: ERROR: 
return code: 1; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpXCLWzL' -- /mnt/host/source/chromite/cbuildbot/binhost_test -v '--log-level=debug' NoIncremental
cmd=['/b/swarming/w/ir/cache/cbuild/repository/chromite/bin/cros_sdk', 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpXCLWzL', '--', '/mnt/host/source/chromite/cbuildbot/binhost_test', '-v', '--log-level=debug', 'NoIncremental'], cwd=/b/swarming/w/ir/cache/cbuild/repository, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmpXCLWzL'}
 
agawronska@, i saw you are the Chrome Gardener. Could you please take a look on the master-chromium-pfq failure. The link can be found from the issue description. If you're not the owner, please let me know. Thanks
Cc: gmeinke@chromium.org shchen@chromium.org
cc MTV Sheriffs.
The error started from the last 7 build. It looks like the failure is caused by the same reason.

Comment 4 Deleted

Summary: BinhostTest failure: master-chromium-pfq test failed (was: BinhostTest failure: master-chromium-pfq build failed)
Cc: cjmcdonald@chromium.org manojgupta@chromium.org vapier@chromium.org
Labels: OS-Chrome
I will take a look. Thank you!
Cc: nverne@chromium.org
Labels: Pri-1 Type-Bug
we'll either need to add a grunt Chrome PFQ config/bot (prob what we want to do), or revert this CL:
  https://chromium-review.googlesource.com/1313382
How soon could we have grunt PFQ config/bot added and who could I assign this to? 
I would do the revert first and then reland the change after we have bot/config in place.
I made the reversion CL: crrev/c/1342707

There already is a grunt-chrome-pfq, but it's not clear to me how to get it to pick up the USE flag that's causing the incompatibility behind this failure. Any advice here on what a fix should look like?
Thank you for the revert! This should unblock the PFQ.
One way (though I am not sure if it should be recommended) to enable the USE flag per board is via chromite which would bypass binhost test issues:
E.g. CL that enables a flag on elm-release builder
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1275347/2/config/chromeos_config.py
Labels: Build-PFQ-Failures
Status: Verified (was: Untriaged)
Verified that this passed on the latest PFQ run.
Project Member

Comment 16 by bugdroid1@chromium.org, Nov 28

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4274496685fee71a9f32c124e75adbd9f4895169

commit 4274496685fee71a9f32c124e75adbd9f4895169
Author: Chris McDonald <cjmcdonald@chromium.org>
Date: Wed Nov 28 19:15:13 2018

chromeos_config: Add grunt_march USE flag to grunt-release builder

TEST=None
BUG= chromium:906572 

Change-Id: I2c2bc3596826e7724a475a28d2a3ecadd9670c38
Reviewed-on: https://chromium-review.googlesource.com/1344729
Commit-Ready: Chris McDonald <cjmcdonald@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Aga Wronska <agawronska@chromium.org>

[modify] https://crrev.com/4274496685fee71a9f32c124e75adbd9f4895169/config/chromeos_config.py
[modify] https://crrev.com/4274496685fee71a9f32c124e75adbd9f4895169/config/config_dump.json

Sign in to add a comment