This is because it finds unexpected new binaries with suid permission at /opt/google/containers/android/rootfs/root/system/xbin/procmem & /opt/google/containers/android/rootfs/root/system/xbin/su.

It fails again trying to dump system status.

Android PFQ starts to fail for the same reason as well, starting from https://luci-milo.appspot.com/buildbot/chromeos/betty-arcnext-pi-android-pfq/382.

Let me add the baseline file.

We marked this experimental for the CQ.

It looks like this is the fix: crrev.com/c/1101460

Once that goes in we can unmark the build.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/db9430362105585278a605544b2734de3265b902

commit db9430362105585278a605544b2734de3265b902
Author: Garfield Tan <xutan@google.com>
Date: Thu Jun 14 22:10:24 2018

SuidBinaries: add betty-arcnext baselines

betty-arcnext is running PI version of userdebug image.

BUG= chromium:852803 
TEST=test_that betty-arcnext security_SuidBinaries

Change-Id: I8ca91eb30ac4eadd8de6872aa68a9adecfc1253f
Reviewed-on: https://chromium-review.googlesource.com/1101460
Commit-Queue: Garfield Tan <xutan@chromium.org>
Tested-by: Garfield Tan <xutan@chromium.org>
Trybot-Ready: Garfield Tan <xutan@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[add] https://crrev.com/db9430362105585278a605544b2734de3265b902/client/site_tests/security_SuidBinaries/baseline.betty-arcnext.suid
[add] https://crrev.com/db9430362105585278a605544b2734de3265b902/client/site_tests/security_SuidBinaries/baseline.betty-arcnext.sgid

Ilja, the newest betty-arcnext PFQ is still failing for the same reason. Is there need to push new test suite to bots to make them effective?

FYI, latest PFQ run is still red with the same reason.

I am investigating. There is no need to push anything. The CQ/Paladin is happy though. Both Android and Chrome PFQ are unhappy, which is rather confusing.

For completeness links

Release builder runs VMTest and passes
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arcnext-release

Paladin/CQ runs VMTest and passes
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arcnext-paladin

Android PFQ fails VMTest
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arcnext-pi-android-pfq

Chrome PFQ fails VMTest
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arcnext-chrome-pfq

The odd part is that both PFQs fail it. (It might be independent and legitimate failures. It might be a race in the test due to performance of the builder.)

Android PFQ fails with
TestFail: New suid binaries: /opt/google/containers/android/rootfs/root/system/xbin/su, /opt/google/containers/android/rootfs/root/system/xbin/procmem

The Chrome PFQ fails with the same. I can't explain how both PFQs would hit that if they are independent, but I'll just add them to the list.

They are already part of https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/security_SuidBinaries/baseline.vm.arc_userdebug.suid, and https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/security_SuidBinaries/baseline.betty-arcnext.suid is just a symlink to it. All betty boards are set up in this way.

More importantly we added these cases yesterday. Looking at the images used in the PFQs now.

Not sure if related though, there is an Android checkout of autotest at //external/autotest [1], which doesn't have that change yet. Not sure if we also need to change that as well.

[1]: https://cs.corp.google.com/pi-arc-dev/external/autotest/client/site_tests/security_SuidBinaries/

Not related.

I am going to take a look at the builder.

The builder build172-m2.golo had the right chroot checkout including the change. But the files built by the chroot for use by autotest did not contain the change. See attachment. Maybe a problem with ebuilds + incremental builds?

Deleted: test-security_SuidBinaries.tar.bz2 2.7 KB

test-security_SuidBinaries.tar.bz2 2.7 KB Download

Hmm... Really can't get ebuild right. It might have something to do with incremental builds, as the ebuild file for autotest-tests-security doesn't change, so its version won't be auto-bumped.

Maybe we need to manually bump ebuild version for it?

OK... chrome-bot just bumped security test's ebuild number in [1]. The last time it did it was in Jun. 7. Let's see if PFQ's will become green this time.

[1]: https://chromium.git.corp.google.com/chromiumos/overlays/chromiumos-overlay/+/7fdb85d4465857948a47e270bcee933c0813ab05/chromeos-base/autotest-tests-security/autotest-tests-security-0.0.1-r3223.ebuild

I hope that will do the trick.

I'm not really familiar with gentoo output
But looking at the output of the build_package phase of AndroidPFQ #387

It seems autotest-tests-security was built.
[binary  N     ] chromeos-base/autotest-tests-security-0.0.1-r3222::chromiumos

And from the man page of emerge
$ man emerge
[ebuild N ] app-games/qstat-25c
   Qstat is New to your system, and will be emerged for the first time.

Oops I didn't noticed that one is binary and one is ebuild. 
Please ignore my comment #21. 

Ah... That can explain something. TIL. Thanks pwang@. The new ebuild # is 3223. A new rebuild is necessary.

On the positive side the next chrome-pfq run has the upreved package. (It builds it from scratch though, even though I thought it should use the prebuild.)

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fbetty-arcnext-chrome-pfq%2F563%2F%2B%2Frecipes%2Fsteps%2FBuildPackages%2F0%2Fstdout

[ebuild  N     ] chromeos-base/autotest-tests-security-0.0.1-r3223::chromiumos to /build/betty-arcnext/ ...


The next android PFQ build is not there yet, but I am optimistic this is finally going to be fixed.
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arcnext-pi-android-pfq/builds/390

The first VMTest of Android PFQ#391 passed the SuidBinaries but failed on other tests.

/tmp/cbuildbotmNJ_8a/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-14-security_SuidBinaries                                                                                 [  PASSED  ]
/tmp/cbuildbotmNJ_8a/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-14-security_SuidBinaries/security_SuidBinaries.fscap                                                     [  PASSED  ]
/tmp/cbuildbotmNJ_8a/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-14-security_SuidBinaries/security_SuidBinaries.sgid                                                      [  PASSED  ]
/tmp/cbuildbotmNJ_8a/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-14-security_SuidBinaries/security_SuidBinaries.suid                                                      [  PASSED  ]

That's tracked at b/110264684. Closing this one.