Running ./build_packages --board=lakitu on ToT on my local machine is OK.

lakitu-st-paladin now passed, so either this was a flake or it was after all caused by one of the pending patches (and wasn't applied this time around).

New failures for same reason:

https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-st-paladin/builds/1269

https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-gpu-paladin/builds/1727

Several kernel CLs were among the two RED runs but they look innocent...

common CLs are set([717451, 717452, 717453, 717454])

FYI, this is independently tracked in b/68053870.

My CLs all passed build_packages in these tryjobs:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/63027
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/63028

I'm innocent, I tell you!

Could these set of changes be related: https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/713459 ?

Issue 777277 has been merged into this issue.

In bug #777277 I was asserting that the build machine was behaving very strangely.  Can anyone check if maybe it just has a bad disk or something?

It's certainly possible that the kernel build cache could cause weird issues, but it's strange that kevin and other kernel 4.4 users aren't seeing that.  I'd also note that the kernel has passed on some of the future runs, but that we saw different totally weird error messages...

I think build185-m2 does indeed have some disk issues.
That said, the other lakitu builders are still failing to build kernel, and on other build slaves. So that's a separate issue.

If you indeed seeing messages as per the original report then it's plausible that CLs Guenter pointed at (those for  bug #767073 ) are related.  It doesn't explain why the kevin builder isn't seeing this, but I could sorta believe that somehow we're creating a cache file too early and it's confusing the builder.

Can folks repro this locally?  Can you figure out what exactly is in the directory that is not empty?

I reproduced it locally. When I install dbus package, the owner of the temp folder is root/root, instead of my account.

Reproduce step:

1. Enter chroot
2. $ ./setup_board --board lakitu
3. $ ./build_packages --board lakitu
4. $ emerge-lakitu sys-apps/dbus 
Same error messsage here

$ ls -la /build/lakitu/tmp/portage/sys-apps/dbus-1.10.12-r5/work                              
total 12
drwxr-xr-x 3 root     root    4096 10月 24 14:01 .
drwxrwxr-x 6 akahuang portage 4096 10月 24 14:01 ..
drwxr-xr-x 2 root     root    4096 10月 24 14:01 dbus-1.10.12

I tried other board and the permission is:
$ ls -la /build/poppy/tmp/portage/sys-apps/dbus-1.10.12-r5/work
total 16
drwxr-xr-x  4 akahuang portage 4096 10月 24 14:02 .
drwxrwxr-x  7 akahuang portage 4096 10月 24 14:02 ..
drwxr-xr-x 11 akahuang portage 4096 10月 24 14:02 dbus-1.10.12
drwxr-xr-x  2 akahuang portage 4096 10月 24 14:03 dbus-1.10.12-abi_x86_64.amd64

you're hitting a bug in the linux-info.eclass that was fixed here:
  https://sources.gentoo.org/gentoo-x86/eclass/linux-info.eclass?r1=1.103&r2=1.104

posted https://chromium-review.googlesource.com/735061

Re #17,

Thanks vapier!! I successfully emerge dbus after cherry-pick your CL. But sys-fs/lvm2 still failed to emerge.


============== Log =================
$ emerge-lakitu sys-fs/lvm2
Calculating dependencies... done!

>>> Emerging (1 of 1) sys-fs/lvm2-2.02.97-r4::portage-stable for /build/lakitu/
 * LVM2.2.02.97.tgz SHA256 SHA512 WHIRLPOOL size ;-) ...                                                                                                                               [ ok ]
 * Running stacked hooks for pre_pkg_setup
 *    sysroot_build_bin_dir ...                                                                [ ok ]
 * Determining the location of the kernel source code
 * Found kernel source directory:
 *     /build/lakitu/usr/src/linux
 * Found sources for kernel version:
 *     4.4.92
 * Unable to check for the following kernel config options due
 * to absence of any configured kernel sources or compiled
 * config:
 *  - SYSVIPC - CONFIG_SYSVIPC: is not set (required for udev sync)
 * 
 * You're on your own to make sure they are set if needed.
 * Running stacked hooks for post_pkg_setup
 *    python_eclass_hack ...                                                                   [ ok ]
 * Running stacked hooks for pre_src_unpack
 *    python_multilib_setup ...                                                                [ ok ]
>>> Unpacking source...
>>> Unpacking LVM2.2.02.97.tgz to /build/lakitu/tmp/portage/sys-fs/lvm2-2.02.97-r4/work
tar: LVM2.2.02.97/README: Cannot open: Permission denied
...

Hmm it's ok after I run "./setup_board --board lakitu --force" again. False alarm :p

yeah you prob would need to `sudo rm -rf /build/lakitu/tmp/portage/*` first

Re #17: Any idea why only lakitu is hitting this, and why only since last weekend?

lakitu has a custom kernel, and i think they just upgraded it, so i suspect it's related to that.  i didn't check any commit timelines to double check.

> lakitu has a custom kernel, and i think they just upgraded it, so i suspect it's related to that.  i didn't check any commit timelines to double check.

Its not upgraded yet. We are still on the 4.4 kernel. Just that the kernel config is different. But that hasn't changed recently either.

FWIW, we ran various lakitu trybots with https://crrev.com/c/735061 which is supposed to be potential fix.

lakitu-incremental: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/incremental/builds/641
lakitu-paladin: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/4047

both failed at BuildPackages stage. This can be fixed by clobbering the builders.

But lakitu-release: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/16587/steps/Archive/logs/stdio
which does a clean build, still fails with at mod_image_for_recovery.sh script

(from https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/16587/steps/Archive/logs/stdio)

>>> Unpacking source...
fatal: destination path '/build/lakitu/tmp/portage/sys-kernel/lakitu-kernel-4_4-4.4.92-r1122/work/lakitu-kernel-4_4-4.4.92' already exists and is not an empty directory.

The change has not been merged but our paladin suddenly becomes green (https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-paladin ). Strange...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/41461f50e12cf5fa005a428f567fb6fd42de7bc3

commit 41461f50e12cf5fa005a428f567fb6fd42de7bc3
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Oct 25 00:36:35 2017

linux-info: pull in latest version

This fixes kernel version checks in early pkg_setup phases.

BUG= chromium:776919 
TEST=lakitu builds again

Change-Id: Iada30f7f10af5737db5d8c058c84f4665b838b7b
Reviewed-on: https://chromium-review.googlesource.com/735061
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Chih-Yu Huang <akahuang@chromium.org>

[modify] https://crrev.com/41461f50e12cf5fa005a428f567fb6fd42de7bc3/eclass/linux-info.eclass

After vapier's CL is merged, sys-apps/dbus package can be built, but go/gloud and sys-fs/lvm2 still failed to build. vapier do you have any clue about it ?

Looks like gocloud issue will be resolved when http://crrev.com/c/733854 is merged.

lvm2 issue still seems related to linux-info.eclass.

Regarding sys-fs/lvm2 problem, lvm2-2.02.97-r4.ebuild's pkg_setup() calls check_extra_config() which is in linux-info.eclass. And that eventually invokes getfilevar() in linux-info.eclass.

In that function, ${EBUILD_PHASE_FUNC} is empty. Is that because lvm2-2.02.97-r4.ebuild has EAPI=4 ?

most likely. let me have a look.

Upping priority. I see that this is making consistent progress, but we _really_ need those paladins back in the important set. Priority to reflect that.

For me failure mode for sys-fs/lvm2 is same as #16 i.e permission denied (may bcoz lvm2 is EAPI=4 as mentioned above). I cross-checked commit mentioned in #25 is present. I locally upgraded to EAPI=5 and was able to build lakitu image

The simplest might to upgrade lvm2. Upgrading it to gentoo stable fixed the build failure for me and lakitu-paladin passed. Uploaded https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/738796

lets do the EAPI bump for low risk.  last time i tried lvm2 upgrade it did not go well.  but maybe things have improved since.  plus, we can still do the upgrade after the EAPI bump.

Now chromeos-base/metrics failed to build in lakitu* board. Should be caused by this CL:
https://chromium-review.googlesource.com/#/c/chromiumos/platform2/+/731364/

Adding "libsession_manager-client" dependency should fix this.

Re #34: The latest lakitu*paladins have passed. I didn't see any failures due to the CL you mentioned, nor do I see a fix / revert. Did you do anything for this?

My plan is to wait for the current CQ to finish. Whether or not https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/738328 is merged, we will have confidence that this doesn't break any boards.

At that point, I'm going to chump that CL if needed and mark lakitu boards important again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/48e32bb8e1a75a58465da5df001aaae4169b77a8

commit 48e32bb8e1a75a58465da5df001aaae4169b77a8
Author: Pradeep Sawlani <sawlani@google.com>
Date: Thu Oct 26 16:43:00 2017

sys-fs/lvm2: Bump EAPI version to 5.

BUG= chromium:776919 
TEST=emerge-lakitu sys-fs/lvm2
TEST=build lakitu board image with and without this patch

Change-Id: I5d9a960b6a2bb686e086b5f9c1a062204a0ae13f
Signed-off-by: Pradeep Sawlani <sawlani@google.com>
Reviewed-on: https://chromium-review.googlesource.com/738328
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/48e32bb8e1a75a58465da5df001aaae4169b77a8/sys-fs/lvm2/lvm2-2.02.97-r1.ebuild
[rename] https://crrev.com/48e32bb8e1a75a58465da5df001aaae4169b77a8/sys-fs/lvm2/lvm2-2.02.97-r5.ebuild

lakitu is important in the CQ again. Lowering to P1.
Please continue to drive the rest / wait for the fix to make it to release builders and let me know if you need anything from me. Removing from my attention set.

Re #35, I created a CL for it:
https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/737773/

Now the lakitu CQ has passed, I think we can close this issue after this CL is merged.

Can't close yet. kernel builds are still broken on lakitu paladins.
See https://b.corp.google.com/issues/68053870#comment22

That bug was closed this morning.  The last comment there was:

Closing this, since builders were all green over the weekend. Today morning one build failed which seems to be different issue (b/68494029)

...presumably we don't need this bug open any more?

Yeah all of our builders are back to green. Thanks to everyone who helped triage and fixing the issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/c0c9ae0c5e4dc8cc48f53f22e88287910bbb0ff5

commit c0c9ae0c5e4dc8cc48f53f22e88287910bbb0ff5
Author: Chih-Yu Huang <akahuang@google.com>
Date: Tue Oct 31 03:51:30 2017

metrics: add session_manager-client dependency.

In CL:731364 metrics added session_manager-client dependency. This CL
added the dependency at ebuild file as well.

BUG= chromium:776919 , chromium:774036
TEST=emerge-lakitu chromeos-base/metrics

Change-Id: I119df265a2cb35a979e63797a7f802fdcd91a87b
Reviewed-on: https://chromium-review.googlesource.com/737773
Tested-by: Chih-Yu Huang <akahuang@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Sergey Poromov <poromov@chromium.org>
Commit-Queue: Chih-Yu Huang <akahuang@chromium.org>

[modify] https://crrev.com/c0c9ae0c5e4dc8cc48f53f22e88287910bbb0ff5/chromeos-base/metrics/metrics-9999.ebuild