New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 864873 link

Starred by 1 user
Status: Fixed
Merged: issue 700813
Owner:
vapier@chromium.org
Closed: Jul 23
Cc:
█ yungleem@chromium.org
nsanders@chromium.org
chromeos-continuous-integration-fire-hose@google.com
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 700813


Sign in to add a comment

Build failure on buddy factory factory-auron-6772.B including other boards and pre-flight

Project Member Reported by marcochen@chromium.org, Jul 18 
[ADD relevant info inline]
buddy factory builder failed to build since Oct 27 2017

Link to build or pfq page.
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/buddy%20factory%20factory-auron-6772.B

build # for that buildbot.
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/buddy%20factory%20factory-auron-6772.B/builds/62

Snippet of log that contains the failure.
************************************************************
** Start Stage BuildPackages - Fri, 13 Jul 2018 10:01:29 -0700 (PDT)
** 
** Build Chromium OS packages.
************************************************************
10:01:29: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpWTlf_p' -- /home/chrome-bot/trunk/chromite/bin/cros_extract_deps '--format=deps' virtual/target-sdk chromeos-base/chromite --output-path /home/chrome-bot/trunk/chroot/tmp/tmpH5VZzg in /b/c/cbuild/repository
10:01:36: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpv6r7RZ' -- /home/chrome-bot/trunk/chromite/bin/cros_extract_deps --board buddy '--format=deps' virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all --output-path /home/chrome-bot/trunk/chroot/tmp/tmpydzPss in /b/c/cbuild/repository
10:01:45: INFO: Recording packages under test
10:01:45: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk '--chrome_root=/b/c/cbuild/repository/.cache/distfiles/target/chrome-src-internal' 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpNs7Gxc' 'USE=-cros-debug -highdpi chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' 'FEATURES=separatedebug' -- ./build_packages '--board=buddy' '--accept_licenses=@CHROMEOS' --skip_chroot_upgrade --nousepkg --reuse_pkgs_from_local_boards --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all in /b/c/cbuild/repository
ERROR   cros_sdk:enter_chroot: Not mounting chrome source: could not find CHROME_ROOT/src dir.
ERROR   cros_sdk:enter_chroot: Full path we tried: /b/c/cbuild/repository/.cache/distfiles/target/chrome-src-internal/src
ERROR   cros_sdk:enter_chroot: Fri Jul 13 10:01:45 PDT 2018
ERROR   cros_sdk:enter_chroot: script called:  '--chroot' '/b/c/cbuild/repository/chroot' '--cache_dir' '/b/c/cbuild/repository/.cache' '--chrome_root' '/b/c/cbuild/repository/.cache/distfiles/target/chrome-src-internal' '--' 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpNs7Gxc' 'USE=-cros-debug -highdpi chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' 'FEATURES=separatedebug' './build_packages' '--board=buddy' '--accept_licenses=@CHROMEOS' '--skip_chroot_upgrade' '--nousepkg' '--reuse_pkgs_from_local_boards' '--noworkon' 'virtual/target-os' 'virtual/target-os-dev' 'virtual/target-os-test' 'chromeos-base/chromeos-installshim' 'chromeos-base/chromeos-factory' 'chromeos-base/chromeos-hwid' 'chromeos-base/autotest-factory-install' 'chromeos-base/autotest-all'
ERROR   cros_sdk:enter_chroot: Backtrace:  (most recent call is last)
ERROR   cros_sdk:enter_chroot:  enter_chroot.sh:496:main(), called: setup_env  
ERROR   cros_sdk:enter_chroot:  enter_chroot.sh:493:setup_env(), called: die 'setup_env failed' 
ERROR   cros_sdk:enter_chroot: 
ERROR   cros_sdk:enter_chroot: Error was:
ERROR   cros_sdk:enter_chroot:   setup_env failed
Running {/b/c/cbuild/repository/src/scripts/sdk_lib/enter_chroot.sh --chroot /b/c/cbuild/repository/chroot --cache_dir /b/c/cbuild/repository/.cache --chrome_root /b/c/cbuild/repository/.cache/distfiles/target/chrome-src-internal -- 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpNs7Gxc' 'USE=-cros-debug -highdpi chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' 'FEATURES=separatedebug' ./build_packages '--board=buddy' '--accept_licenses=@CHROMEOS' --skip_chroot_upgrade --nousepkg --reuse_pkgs_from_local_boards --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all} failed with exit code 1
10:01:45: ERROR: 
return code: 1
Failed command "/b/c/cbuild/repository/chromite/bin/cros_sdk '--chrome_root=/b/c/cbuild/repository/.cache/distfiles/target/chrome-src-internal' 'PARALLEL_EMERGE_STATUS_FILE=/home/chrome-bot/trunk/chroot/tmp/tmpNs7Gxc' 'USE=-cros-debug -highdpi chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' 'FEATURES=separatedebug' -- ./build_packages '--board=buddy' '--accept_licenses=@CHROMEOS' --skip_chroot_upgrade --nousepkg --reuse_pkgs_from_local_boards --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test chromeos-base/chromeos-installshim chromeos-base/chromeos-factory chromeos-base/chromeos-hwid chromeos-base/autotest-factory-install chromeos-base/autotest-all", cwd=/b/c/cbuild/repository, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/home/chrome-bot/trunk/chroot/tmp/tmpNs7Gxc', 'USE': '-cros-debug -highdpi chrome_internal', 'IGNORE_PREFLIGHT_BINHOST': '1', 'CHROME_ORIGIN': 'LOCAL_SOURCE', 'FEATURES': 'separatedebug'}
10:01:45: ERROR: ./build_packages failed (code=1)
10:01:45: INFO: Running cidb query on pid 1363, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f6b7db16dd0>
10:01:45: INFO: Running cidb query on pid 1363, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f6b7db16490>
************************************************************
** Finished Stage BuildPackages - Fri, 13 Jul 2018 10:01:46 -0700 (PDT)
************************************************************

Comment 1 by marcochen@chromium.org, Jul 18

Cc: nsanders@chromium.org yungleem@chromium.org
Owner: vapier@chromium.org

Comment 2 by marcochen@chromium.org, Jul 18 

The OEM of Buddy project needs new factory shim in order to get fix by CL:1136691.

Comment 3 by jclinton@google.com, Jul 18 

This has been broken since Oct 2015 and the logs for the breakage at that time are long gone. It's unlikely that we'll be able to fix it.

Comment 4 by jclinton@google.com, Jul 18 

 https://crbug.com/624177  might be one of the issues affecting this branch but I suspect that there are at least 5 different issues, given the time frame. Looking back, it appears the branch was created in early 2015 making this more than 3 years old.

Comment 5 by nsanders@chromium.org, Jul 18 

3 years is a pretty expected lifespan of a factory branch. It's created 3mo before the launch of lead project, the last follower is 12-18 months after, plus a year or two of RMA support by the OEMs

Comment 6 Deleted

Comment 7 by vapier@chromium.org, Jul 18

Mergedinto: 700813
Status: Duplicate (was: di)

Comment 8 by nsanders@chromium.org, Jul 18

Blockedon: 700813
Status: Assigned (was: Duplicate)
Bhaskar and Jason requested one bug per branch, since each needs to be fixed individually. Marked as blocked rather than dup.

Comment 9 by vapier@chromium.org, Jul 18 

sounds fine, thanks

Comment 10 by marcochen@chromium.org, Jul 19 

re#3,

Although the build didn't go through all stages well, some of them still can reach to archive stage. Therefore partners still can get new images (ex: [1]). 

But all recent builds failed to come out images so we are blocked by this issue to provide new factory shim.

[1] https://uberchromegw.corp.google.com/i/chromeos.branch/builders/buddy%20factory%20factory-auron-6772.B/builds/56

Comment 11 by nsanders@chromium.org, Jul 19

Labels: -Pri-3 Pri-2
Project Member

Comment 12 by bugdroid1@chromium.org, Jul 21

Labels: merge-merged-factory-auron-6772.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/00952228b60d8245bc77607b168d11134390bbf7

commit 00952228b60d8245bc77607b168d11134390bbf7
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 21 08:40:57 2018

sync_chrome: actually run the sync function

The previous change in here stopped returning the functor from
SyncChrome, but didn't execute it.  This caused the sync logic
to not actually sync anything.

BUG= chromium:658504 ,  chromium:864873 
TEST=None

Change-Id: I2616657d745d9666237fa2956361940458b5499c
Reviewed-on: https://chromium-review.googlesource.com/1146049
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/00952228b60d8245bc77607b168d11134390bbf7/scripts/sync_chrome.py
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 23 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/38e8d58131d947e4011d88f15139620a4e27afa8

commit 38e8d58131d947e4011d88f15139620a4e27afa8
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Jul 23 15:47:52 2018

cbuildbot_config: disable unittests for factory configs

The unittests on this ancient branch is failing.  We know some older
versions aren't as hermetic as they should be (and has been addressed
in master), so disable them here.

Practically speaking for factory images, unittests don't have as much
value as we only care whether Chrome runs and can run some stuff.

BUG= chromium:864873 
TEST=None

Change-Id: I713128ae46366962c6f923a4b5ae1b3e8a8b057b
Reviewed-on: https://chromium-review.googlesource.com/1146875
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/38e8d58131d947e4011d88f15139620a4e27afa8/cbuildbot/config_dump.json
[modify] https://crrev.com/38e8d58131d947e4011d88f15139620a4e27afa8/cbuildbot/cbuildbot_config.py
Project Member

Comment 14 by bugdroid1@chromium.org, Jul 23 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/78a7d52cf6c2b6525d1410891902c7b33ae3c8bf

commit 78a7d52cf6c2b6525d1410891902c7b33ae3c8bf
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Jul 23 18:25:39 2018

cbuildbot_config: disable unittests for factory pre-flight configs too

Same as previous commit in here.

BUG= chromium:864873 
TEST=None

Change-Id: Ia9163da0b3ea39ec0ae01116709692b6232ea1a0
Reviewed-on: https://chromium-review.googlesource.com/1147145
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/78a7d52cf6c2b6525d1410891902c7b33ae3c8bf/cbuildbot/config_dump.json
[modify] https://crrev.com/78a7d52cf6c2b6525d1410891902c7b33ae3c8bf/cbuildbot/cbuildbot_config.py

Comment 15 by vapier@chromium.org, Jul 23

Status: Fixed (was: Assigned)
the Chrome sync failure was caused by this CL:
  https://chromium-review.googlesource.com/411444

looks like YH's attempt to backport logic from newer branches went wrong.  my change above should fix it.  i spot checked all the other branches which had the sync logic backported and they looked OK ... it seems like only this one branch didn't make it.

after that i disabled unittests as it looks like autotest/chromite are unhappy in some way, and i don't think anyone actually cares.

preflight builder is now green.
►

Sign in to add a comment