New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 839331 link

Starred by 4 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

mod_for_recovery failure: portage/dbapi/vartree.py: KeyError(mycpv): KeyError: u'sys-apps/flashrom-0.9.9-r539'

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, May 3 2018

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of coconutruben@google.com

/mod_image_for_recovery.sh failed in this builder, and subsequently no recovery image is found. I added the two error patterns I find in the logs below, I'm not sure if these are both symptoms of the same issue. 

One error pattern in the logs is:

  File "/usr/lib64/python2.7/site-packages/portage/dbapi/vartree.py", line 1177, in _populate
    hash_value = hash_pkg(cpv)
  File "/usr/lib64/python2.7/site-packages/portage/dbapi/vartree.py", line 1145, in _hash_pkg
    cpv, ["COUNTER", "_mtime_"])
  File "/usr/lib64/python2.7/site-packages/portage/dbapi/vartree.py", line 710, in aux_get
    raise KeyError(mycpv)
KeyError: u'sys-apps/flashrom-0.9.9-r539'
ERROR   : Wed May  2 13:55:34 PDT 2018
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 10: ./mod_image_for_recovery.sh '--board=celes' '--image=/mnt/host/source/src/build/images/celes/R68-10640.0.0/tmpp67FmL/chromiumos_base_image.bin'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  mod_image_for_recovery.sh:404:main(), called: emerge_custom_kernel '/build/celes/factory-root' 
ERROR   :  common.sh:1231:emerge_custom_kernel(), called: die 'Cannot emerge chromeos-initramfs' 
ERROR   : 
ERROR   : Error was:
ERROR   :   Cannot emerge chromeos-initramfs

The other pattern I found is:

!!! Invalid binary package: '/build/celes/packages/dev-only-extras.tbz2'
!!! Missing metadata key(s): CATEGORY, PF, SLOT. This binary package is not
!!! recoverable and should be deleted.
[02/May/2018:13:53:47] BUILD Filtering install mask from chromeos-base/chromeos-chrome-68.0.3416.0_rc-r1
pbzip2: *WARNING: Trailing garbage after EOF ignored!
13:54:02: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk -- equery-celes list 'sys-kernel/*kernel*' in /b/c/cbuild/repository
13:54:08: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk -- strip_package --board celes sys-kernel/chromeos-kernel-3_18-3.18-r2107 in /b/c/cbuild/repository
!!! Invalid binary package: '/build/celes/packages/dev-only-extras.tbz2'
!!! Missing metadata key(s): CATEGORY, PF, SLOT. This binary package is not
!!! recoverable and should be deleted.
[02/May/2018:13:54:11] BUILD Filtering install mask from sys-kernel/chromeos-kernel-3_18-3.18-r2107
pbzip2: *WARNING: Trailing garbage after EOF ignored!
13:54:14: INFO: RunCommand: tar --sparse -I cat -cf /b/c/cbuild/repository/buildbot_archive/celes-release/R68-10640.0.0/stripped-packages.tar /b/c/cbuild/repository/chroot/build/celes/stripped-packages/chromeos-base/chromeos-chrome-68.0.3416.0_rc-r1.tbz2 /b/c/cbuild/repository/chroot/build/celes/stripped-packages/sys-kernel/chromeos-kernel-3_18-3.18-r2107.tbz2
tar: Removing leading `/' from member names

celes-release:2165 failed

Builders failed on: 
- celes-release: 
  http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947616999678169728


 
Cc: -coconutruben@google.com dgarr...@chromium.org coconutruben@chromium.org henryhsu@chromium.org agicsaki@chromium.org warx@chromium.org
Components: Infra>Client>ChromeOS
Labels: Hotlist-CrOS-Sheriffing

Comment 2 by warx@chromium.org, May 3 2018

Cc: vapier@chromium.org minch@chromium.org
Mike, could you help triage?

Comment 3 by warx@chromium.org, May 3 2018

Labels: OS-Chrome
Labels: -Pri-1 Pri-3 Type-Bug
Summary: mod_for_recovery failure: portage/dbapi/vartree.py: KeyError(mycpv): KeyError: u'sys-apps/flashrom-0.9.9-r539' (was: mod_for_recovery failure: celes-release:2165 failed)
i'd guess it's either a race condition in portage, or in the overall cbuildbot stages that might be mutating the vdb while Archive is running

if it hasn't come up again, i'd make it pretty low priority
build.log
31.5 KB View Download

Comment 5 by warx@chromium.org, May 7 2018

Cc: -warx@chromium.org -minch@chromium.org malaykeshav@chromium.org sammiequon@chromium.org
Cc: -coconutruben@chromium.org philipchen@chromium.org adurbin@chromium.org
+phillipchen, +adurbin (this week's sheriffs) if you guys can keep a look out for this pattern
Cc: -sammiequon@chromium.org -malaykeshav@chromium.org derat@chromium.org
Cc: evgreen@chromium.org teravest@chromium.org
This happened recently on asuka-release:
https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8939492252620876224/+/steps/Archive/0/stdout

  File "/usr/lib64/python2.7/site-packages/_emerge/FakeVartree.py", line 284, in _pkg
    self._real_vardb.aux_get(cpv, self._db_keys)),
  File "/usr/lib64/python2.7/site-packages/portage/dbapi/vartree.py", line 710, in aux_get
    raise KeyError(mycpv)
KeyError: u'chromeos-base/chromeos-storage-info-0.0.1-r18'
ERROR   : Tue Jul 31 06:11:32 PDT 2018

...

ERROR   : Error was:
ERROR   :   Cannot emerge chromeos-initramfs
Issue 891726 has been merged into this issue.

Sign in to add a comment