New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 823959
Owner:
Last visit > 30 days ago
Closed: Apr 5
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

various builds failing sporadically in BuildImage

Project Member Reported by akes...@chromium.org, Apr 2

Issue description

See for instance

https://luci-milo.appspot.com/buildbot/chromeos/lakitu-st-paladin/2706
https://luci-milo.appspot.com/buildbot/chromeos/lakitu-gpu-paladin/3177

both failed with 

IOError: [Errno 30] Read-only file system: '/mnt/host/source/src/build/images/lakitu-gpu/R67-10538.0.0-rc1/rootfs/var/cache/edb/vdb_metadata.pickle.2873'
INFO    : Unmounting image from /mnt/host/source/src/build/images/lakitu-gpu/R67-10538.0.0-rc1/stateful and /mnt/host/source/src/build/images/lakitu-gpu/R67-10538.0.0-rc1/rootfs
INFO    : Unmounting temporary rootfs /mnt/host/source/src/build/images/lakitu-gpu/R67-10538.0.0-rc1/rootfs//build/rootfs.
rmdir: failed to remove '/mnt/host/source/src/build/images/lakitu-gpu/R67-10538.0.0-rc1/rootfs//build': Directory not empty
An error occurred in your build so your latest output directory is invalid.
Running in non-interactive mode so deleting output directory.
 
Cc: lakitu-dev@google.com
Owner: wonderfly@google.com
Labels: -Pri-1 Pri-2
lakitu-gpu-paladin failed once for the linked reason in the last 100 runs. Same was true for lakitu-st-paladin. They both failed four or five more times in the last 100 runs for different reasons (sometimes because of a bad CL in CQ), but considering the rate I'm going to mark this a P2.
Cc: cra...@chromium.org
Status: WontFix (was: Assigned)
I suspect this has to do with the recent changes that stopped installing vmlinuz to rootfs but I can't be sure. FWIW, I see this error message in the BuildImage output since those changes: "/mnt/host/source/src/scripts/bin/cros_make_image_bootable: line 67: portageq-: command not found", which apparently was because the FLAGS_board was referenced before it was parsed, but the same error also appears in passing runs so I am sure it was the cause.

Since the builder only failed once and is continuously passing, I am going to silence this until it fails again, in which case we'll hopefully have more logs to debug effectively.
Labels: -Pri-2 Pri-3
Status: Unconfirmed (was: WontFix)
How about we leave the bug open for ease of discovery in case it recurs.
Hey wonderfly@, I'm curious, why do yo think this is related to https://chromium-review.googlesource.com/c/chromiumos/platform/crosutils/+/969449 ?

This should definitely be considered a risky change, but I'm not sure I see why the failure is related?

Hi Craig, I think I tried to convey that idea that "I wasn't sure but that CL seemed suspicious". My main lead was the error message "cros_make_image_bootable: line 67: portageq-: command not found" that started showing in the BuildImage output at the same time (approximately) the lakitu-gpu build failed.

But again, I didn't spend a ton of time on something that seemed intermittent so I could totally be wrong. Wanted to bring that up any way because the portageq- thing did seem a bug.
The same problem just popped up on caroline-arcnext so it doesn't seem to be limited to lakitu: https://luci-milo.appspot.com/buildbot/chromeos/caroline-arcnext-paladin/119
Owner: cra...@chromium.org
Status: Assigned (was: Unconfirmed)
OK then. There is already a CL that's reverting the vmlinuz change because it has caused update engine failures in other cases, and I believe Craig is on top of them, so I'll just fold this into his pocket. :)
Summary: various builds failing sporadically in BuildImage (was: lakitu builds failing sporadically in BuildImage)
Mergedinto: 823959
Status: Duplicate (was: Assigned)

Sign in to add a comment