Here is the failure message:

cheets_PlayStoreTest                                  [ FAILED ]
cheets_PlayStoreTest                                    FAIL: Unhandled Exception: Can not find SEARCH image
cheets_PlayStoreTest                                    retry_count: 2
Suite timings:

There are some SELinux failures that are probably related:
https://stainless.corp.google.com/browse/chromeos-autotest-results/204667999-chromeos-test/
type=1400  avc:  denied  { read }  comm="Binder:644_4" name="/" dev="tmpfs" ino=299441 scontext=u:r:priv_app:s0:c512,c768 tcontext=u:object_r:device:s0 tclass=dir permissive=0
type=1400  avc:  denied  { search }  comm="Binder:644_2" name="23" dev="proc" ino=299630 scontext=u:r:priv_app:s0:c512,c768 tcontext=u:r:zygote:s0 tclass=dir permissive=0
type=1400  avc:  denied  { read }  comm="Binder:644_2" name="/" dev="tmpfs" ino=299441 scontext=u:r:priv_app:s0:c512,c768 tcontext=u:object_r:device:s0 tclass=dir permissive=0

The failure has hit two CQ runs in a row:
    https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/18759
    https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/18758

It appears to affect all boards running arc-bvt.

There are no CLs in common between the runs, so this is likely a bug
in the tree.

Escalating, since we expect the CQ to fail until this is addressed

This is also failing in the Android PFQ, for instance:
    https://uberchromegw.corp.google.com/i/chromeos/builders/samus-nyc-android-pfq/builds/2155

This moves the test to suite:bvt-perbuild

https://chrome-internal-review.googlesource.com/#/c/chromeos/autotest-cheets/+/634547

This also fails in the canaries, for instance:
    https://uberchromegw.corp.google.com/i/chromeos/builders/kevin-release/builds/2266

The canary that first failed doesn't include a new Chrome, so this isn't
caused by a Chrome change.

The failures in the Android PFQ and the canaries are close together,
time-wise, so it's hard to say which one picked up the change first.

So, this could be caused either by a change in Android, or in the OS
source base.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/a9f978bc773e13afd8a7911239668a6fb36932ad

commit a9f978bc773e13afd8a7911239668a6fb36932ad
Author: Allen Webb <allenwebb@google.com>
Date: Fri Jun 01 20:28:11 2018

This workaround needs to be reverted after this is fixed for real:
https://chrome-internal-review.googlesource.com/c/chromeos/autotest-cheets/+/634547

Since this test is non-hermetic, we'll leave it in bvt-perbuild.

This seems to be happening randomly on the 68 branch, whom is the right owner for this sort of test failure?

Actually it failed three times in a row on Caroline 
https://stainless.corp.google.com/search?view=matrix&col=test&first_date=2018-06-18&last_date=2018-07-02&suite=%5CQbvt-arc%5CE&exclude_au=false&exclude_retried=false&waterfall=chromeos(_release%7C)&owner=chromeos-test&model=caroline&row=build&test=cheets_PlayStoreTest

And minnie
https://stainless.corp.google.com/search?view=matrix&col=test&first_date=2018-06-18&last_date=2018-07-02&suite=%5CQbvt-arc%5CE&exclude_au=false&exclude_retried=false&waterfall=chromeos%28_release%7C%29&owner=chromeos-test&model=veyron_minnie&row=build&test=cheets_PlayStoreTest

It seems to be ok on other boards, and was flaky on the prior build.


Elijah, do you know whom would be the right owner for this kind of failure? It looks like this test gets patched by a variety of people...

This issue is marked as a release blocker with no OS labels associated. Please add an appropriate OS label.

All release blocking issues should have OS labels associated to it, so that the issue can tracked and promptly verified, once it gets fixed.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

All of the failures have the same play error in the screenshot " [RPC:S-13:AEC-13 ...]": https://00e9e64bac8fdd26765c3938cafeee4e2e415ceefd665a801d-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/212183282-chromeos-test%2Fchromeos6-row1-rack23-host19%2Fcheets_PlayStoreTest%2Fresults%2FPlayStoreTest-1..png?qk=AD5uMEsOw3yJkPJiLxc-Z_6zJyEyG0TjCnmlmH8X6bTLAsTnVVMMAQpFnRUK0UnxzuXfH-pZAJGJfi4-PrgCO3CUiNmknSAeNpNBC7NEXm5WrYwaq86qfTzR2Axi67ZUuTGYS1_QoyxtewO58_ahjLnvoI08OB3I0huJ2nxQzy9iTzmRvUiTFbl1MBqKfn8XocVqCZP2eJqh0fVVKLKs3qWQ0NbCEvX-mjV6XMnuRQzlqUAwzyhkdNoxtsXrKGyFDSE9_7JQw7c-O-_ft-J6GzV5NsOB4f-PkUSALWKijK6N8EuPaWygXoLV2aops4oByljEXpjURZLjXPYmqMYJxGu0kfzL1qGIgXXGM-cJJ8sjhpCPpzkAFmlMwU6_HRxJyc5a91Z49Dle-sV8meumLKjy-RArZm083wWQ9YeBGVPj1IoYBuB0fKFtSQGmj0VfQHqIV1UeMa2EUQXbQHKcKDulHoTCgfdQAaTLSaLJLaF6IBN7S0MG9Hd7Mz8_PtweJdU1b1hx06rzV5MG0fk05411InRZT1lkIv0ssYsS29IeE7O7IGcpOvvCX-BLfx5Aef6XaF1Xfqqchz0_ZFC7uef0hPh-rVZBVSlKTTpcCuclAKkmgijArFgWg8v0nbLMe-6_mEag8IKh98DHTcUETDEFLgodKVMZbXSFSY2QiLeY1pBPX4DBmyJnCqHD_dhP_tCJFOPhVOzIcd1o8YBcQI4C3-U_HuNXPK4f_i6H286JlSyGui7IxfNEhCSTLFse1x9uF9BdfPjzIEzvaSQrKJW7MT2gc_Z1SfOxXQhDKBw7K3bg-Kqzbox_3QPXzvAbjzc60KmJhAherqYwNgEABX6_RmVXw31ItdU_6NHVW8CFJFsYFUBBcJgjs3YJ_G0R6qF7EVwKYCS_

I believe this is a recent real regression in Play, as reported also by GMS team: https://buganizer.corp.google.com/issues/110209819

We've moved the test to perbuild because it's not hermetic, not much we can do until it's fixed on the Play side.  BTW, I think it's just random that it fails on some devices, even on some success runs I saw failures of the same sort with retry that succeeded.

Thanks! I need to find where the screen shots end up for these tests to triage these better...

If this is a known issue on the GMS side and is just a flake I think we can live with it on 68. 

This is probably dupe worthy with the internal bug against Play, but since we cannot dupe, we can WontFix.

again we are seeing the same problem in https://cros-goldeneye.corp.google.com/chromeos/console/qaRelease?releaseName=M68-BETA-CHROMEOS-4 (screen shot 
https://storage.cloud.google.com/chromeos-autotest-results/216089406-chromeos-test/chromeos4-row10-rack7-host21/cheets_PlayStoreTest/results/PlayStoreTest-0..png)

This is affecting multiple boards on R68-10718.50.0 beta:

https://stainless.corp.google.com/search?view=list&first_date=2018-07-11&last_date=2018-07-11&owner=chromeos-test&waterfall=chromeos%28_release%7C%29&suite=bvt-arc&test=cheets_PlayStoreTest&build=R68-10718.50.0&status=FAIL&status=ERROR&status=ABORT&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

I don't think there is anything to do here still, it's a server problem with Play Store.  They have narrowed it down to a degenerate case of database calls timing out due to heavy volume (test accounts): https://buganizer.corp.google.com/issues/110896823

So until they fix it, or even if they do, this test remains in bvt-perbuild.

Are test suite definitions taken from the branch they are built from?  the change in comment#8 landed after M-68 branched, so is it possible it's in the critical path in bvt-arc instead for M-68?  If so, we should think about merging this change to M-68 and ignoring the failure for now in qualifying a release

Allen, can you review the comments in #24?  My goal is to unblock the active M68 release without incurring risk by skipping something that shouldn't potentially be skipped.   

I have no problem with this being merged to 68.

Is there really nothing better we can do here? Removing this test from the blocking suite means the test team needs to *manually* check each build for a basic test: "can we install something from the play store". If we really can't install anything (it has happened a bunch of times and was the reason I moved the test to bvt-arc) it wastes a lot of time for a lot of testers setting up for a broken build 

What makes this hard is the play store is maintained independently of Chrome OS and ARC so changes or flakyness there should not break our build and release process. Ideally if the test worked as intended it would save tester time, but instead since it is broken not only do testers have to spend time manually verifying the functionality, but also developers waste time because their CLs get blocked by a bad test, and the release process is held up figuring out whether or not the test results can be ignored.

I agree the test owner/maintainer should look into a better way of checking this functionality, but it shouldn't reintroduce the problems with this original test---blocking everyone else because of issues with an external service.

This test failure observed on M67 as well for coral boards on 10575.58.2, 67.0.3396.99 stable channel.

Stainless logs:
https://stainless.corp.google.com/search?view=list&first_date=2018-07-09&last_date=2018-07-23&owner=chromeos-test&waterfall=chromeos%28_release%7C%29&suite=%5CQbvt-arc%5CE&test=cheets_PlayStoreTest&build=%5ER67%5C-10575%5C.58%5C.2%24&status=FAIL&status=ERROR&status=ABORT&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=false&exclude_acts=true&exclude_retried=false&exclude_non_production=false

This is getting pretty bad on 68, roughly half of the boards in the latest beta failed this test at least 5 times without passing. 

We do have a few that luckily pass, but at this rate of failure the test is meaningless, so I just brought back the CL from comment 8 which should make this not visible in 68 anymore.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/ff6055bd0eb81b9ff8c1caa92c1ef7f9524b0553

commit ff6055bd0eb81b9ff8c1caa92c1ef7f9524b0553
Author: Allen Webb <allenwebb@google.com>
Date: Fri Jul 20 16:05:56 2018

Looks like some query timed out inside Play server.  I've filed an internal bug 111835195 to the team.

I believe the original issue has been fixed, but cheets_PlayStoreTest is still failing for other reasons today.
b/113197541 is the tracking bug. Closing this one.