looking

Yeah, something's definitely up with out screen detection. Swarming's telling me that my N5's screen is on, yet I'm looking at a black screen right now.

Errr, scratch that. I just had a '!' in the wrong place in my local tests. Still looking...

 Issue 640580  has been merged into this issue.

Dumping what I've found so far before signing off for the day:

The chrome_public_test isolate that made it through the CQ (https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=5f8389d7ce277311efafdaece3a18b051c2ebad1) fails when the screen is locked, but succeeds when the screen is unlocked.

A chrome_public_test isolate that ran, and failed, on the main waterfall (https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=6fac87d3cf57600b4a5c40448dded64d3e68c90d) fails when the screen is locked, and **also** fails when the screen is unlocked.

This leads me to believe that swarming's device provisioning isn't failing when it comes to device screen locking. But the fact that the swarmed N5X bot (https://build.chromium.org/p/chromium.android/builders/Android%20N5X%20Swarm%20Builder?numbuilds=200) failed while the non swarmed N5X bot (https://build.chromium.org/p/chromium.android/builders/Marshmallow%2064%20bit%20Tester?numbuilds=200) passed all these tests is very suspicious. Maybe we're doing something wrong when we trigger/isolate. I'll look at this more next week.

Also bumping pri since this seems like a regression from the switch from buildbot to swarming.

Any further updates on this? I might reland my patch with the offending test disabled, because it's blocking another patch and both of them should be merged to M54.

I dug a bit deeper and was unable to reproduce the failures. Given what I mentioned in #5, I still doubt it was device provisioning. And since most of the original logs have expired, I'm not sure how easily I can continue investigation.

Feel free to reland, but try launching an individual tryjob on both linux_android_rel_ng and linux_android_dbg_ng before clicking the cq box. That might rule out any swarming weirdness.

What's preventing us from mirroring the sqlite stuff we do with locksettings.db in non-swarming provisioning?

Re #8: nothing. But I'm still not convinced it would help in this case. If you think otherwise, do me one favor. Take an N5 running KTU84P that you think is *correctly provisioned* and run the following command and tell me what you find:

python src/tools/swarming_client/run_isolated.py -s 6fac87d3cf57600b4a5c40448dded64d3e68c90d -I isolateserver.appspot.com

Following up here, I've run my CL through the trybots multiple times (including explicitly adding the android bots), with no issues (https://codereview.chromium.org/2254763002/#).

I will reland the CL as is first thing next week.

Relanding still ending up breaking the Android Tests bot: https://uberchromegw.corp.google.com/i/chromium.linux/builders/Android%20Tests/builds/31799

I am simply going to disable the tests and reland. Can we please try and work out what infra issues are happening here? My tests happily pass locally and on the trybots.

And yet again it passed the cq before failing elsewhere.

... what is the tryserver doing that the main waterfall isn't?

#12: there shouldn't be a difference.

Ben pointed out offline that I was wrong in #13: the try bots run w/ dcheck_always_on where Android Tests doesn't. This may present a problem in https://codereview.chromium.org/2254763002/diff/240001/chrome/browser/media/webrtc/media_stream_devices_controller.cc, where in MediaStreamDevicesController::GetPermissionTypeForContentSettingsType, PermissionsUtil::GetPermissionType is called in a DCHECK.

Ahah. I'll try fixing that and resubmitting. Strange - I thought all the trybots built without DCHECKS.....

Thanks for the spot - my CL now passes the bot in question! :)

I think this bug can be closed now. It's good to know that the trybots do have DCHECK_ALWAYS_ON, because I previously thought they didn't.

Sorry about the red herring, Ben.

My swarming device provisioning lives to fight another day!