Please help triage.

Detected 3 new flakes for test/step "gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_2DCanvasWebGL". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyVAsSBUZsYWtlIklncHVfdGVzdHMucGl4ZWxfaW50ZWdyYXRpb25fdGVzdC5QaXhlbEludGVncmF0aW9uVGVzdC5QaXhlbF8yRENhbnZhc1dlYkdMDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

The images grabbed from the screen are much larger than they should be in these tryjobs:

https://ci.chromium.org/buildbot/tryserver.chromium.android/android_optional_gpu_tests_rel/17615
https://ci.chromium.org/buildbot/tryserver.chromium.android/android_optional_gpu_tests_rel/17634
https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/639887

See these shards:

https://chromium-swarm.appspot.com/task?id=3b65c14c3c991310&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3b6639b7b4fb4710&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3b62e9ba9ceb2b10&refresh=10&show_raw=1

and these results:

http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?f237d4c82eb90f438e51331dc896b95fa4cadd29_android_optional_gpu_tests_rel_telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?a27c5abe3a8600fe89968b5befebb458cff6a816_android_optional_gpu_tests_rel_telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?3ac0c0fb7b5663ea6e737fd619e7dcda610e2cfd_mac_chromium_rel_ng_telemetry

This isn't an Android-specific problem; something's going wrong with the size of the browser's window.

ccameron@: this sounds similar to the bug we've discussed today in Canary on Mac where the browser is getting the scale factor wrong during resizing. Could there be a race condition that affects the bots in this way? Are all of the fixes you had in mind in progress? Should this be duplicated into another bug, and should things re-stabilize?

The bug we discussed was specific to Mac (and is has been fixed in Canary for a few days) -- it wouldn't be flaky -- it was just wrong logic.

+ericrk for Android insights, but I don't have any ideas offhand.

+junov, +zakerinasab for canvas.

Detected 3 new flakes for test/step "gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_2DCanvasWebGL". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyVAsSBUZsYWtlIklncHVfdGVzdHMucGl4ZWxfaW50ZWdyYXRpb25fdGVzdC5QaXhlbEludGVncmF0aW9uVGVzdC5QaXhlbF8yRENhbnZhc1dlYkdMDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Marked test as flaky.

zmo@: Assigning to you since you are owner of the test and gpu/. Please take a look or reassign to a better owner.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/df09d5f47b07f6b8f17b3be1116b2b5a76b35fa9

commit df09d5f47b07f6b8f17b3be1116b2b5a76b35fa9
Author: Guido Urdaneta <guidou@chromium.org>
Date: Wed Feb 07 16:38:23 2018

Mark several pixel-integration tests as flaky on Android.

Bug:  809884 , 809868 , 809846 , 807370 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I07cbbc7d0b4d182e44d3cf8f2c5b9aef437aabc9
TBR: kbr@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/905645
Reviewed-by: Guido Urdaneta <guidou@chromium.org>
Cr-Commit-Position: refs/heads/master@{#535025}
[modify] https://crrev.com/df09d5f47b07f6b8f17b3be1116b2b5a76b35fa9/content/test/gpu/gpu_tests/pixel_expectations.py

 Issue 809846  has been merged into this issue.

 Issue 809868  has been merged into this issue.

 Issue 810006  has been merged into this issue.

Taking a look at a few uploaded result images. It looks like the rendering is correct, only the screenshot is larger than expected. It seems the ref images are rendered in phone's vertical mode, but the failing tests run in horizontal mode.

This is not new and it has been flaky on and off since last year.

Yuly: do you really think that this is caused by the Android devices sometimes being in landscape mode? See comment https://bugs.chromium.org/p/chromium/issues/detail?id=807370#c4 where this is happening on Mac too.

+bpastene in case there is some race condition in setting the Android devices to portrait mode.

kbr also mentioned this might related with the high dpi.

I won't be able to look into this, so unassign myself.

Re #14: I'm pretty certain Android failures are due to landscape mode.
Just ran the tests locally in landscape orientation and getting exactly the same diffs as in #4.
bpastene@, could it be that "set portrait orientation" code got broken, some devices remember that they were fixed in portrait mode, or just are physically portrait oriented, but other devices reverted to default auto-rotate?

Mac failure is probably a different issue.
On Android, pretty much all the tests fail:
10 tests passed in 421.6s, 10 skipped, 24 failures.
I think the 10 passing are the ones which have Fail expectation.

But on Mac, only 1 test fails:
49 tests passed in 127.3s, 3 skipped, 1 failure.

bpastene@: assigning to you and raising to P1 because this looks like a potentially serious regression in the Android testing infrastructure.

ynovikov@, thanks for investigating and confirming the likely root cause.

I'll look into it, but I imagine a great many more things would be failing if our device provisioning logic broke.

Here's another example of a build that failed *all* pixel tests because the device *or* browser was clearly in landscape mode:

https://ci.chromium.org/buildbot/chromium.gpu.fyi/Android%20Release%20%28Nexus%205X%29/15944

In my opinion, either:

1) There is a problem in the device's provisioning, or:

2) There is a bug in the browser where it isn't correctly detecting the device's orientation.

In this particular case, the browser came up with the wrong orientation *multiple times*. This test harness restarts the browser after every failure.

I still lean toward there being some newly introduced flakiness in device provisioning. Ben, please look at any recent code changes in that area, or OS upgrades that might have affected it. Thanks.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/8038b30ede1bd32c6af22c657fc6bd4c16937ae1

commit 8038b30ede1bd32c6af22c657fc6bd4c16937ae1
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 09 23:21:05 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/2d6ca780eac10914a1f9f9037b96f4192c54ceba

commit 2d6ca780eac10914a1f9f9037b96f4192c54ceba
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Mon Feb 12 19:06:05 2018

Has there been any recent occurrences of similar test failures? I tried viewing recent builds for the builder in #21 but am getting a 404:
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Android%20Release%20%28Nexus%205X%29/

We have renamed it.
Here is a recent failure:
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Android%20FYI%20Release%20%28Nexus%205X%29/20

Hrmmm, something's not right. When provisioning devices, we do two things:
set accelerometer_rotation to 0: https://chrome-internal.googlesource.com/infradata/config/+/a483776fa4dfa62fe3b86309a6a91b4085526b8c/configs/chromium-swarm/scripts/android.py#245
set user_rotation to 0: https://chrome-internal.googlesource.com/infradata/config/+/a483776fa4dfa62fe3b86309a6a91b4085526b8c/configs/chromium-swarm/scripts/android.py#254

With those two settings set to 0, the phone is stuck in portrait mode by default. After the task in #26 failed, the bot's settings were:
accelerometer_rotation = 0
user_rotation = 1

So somehow user_rotation got set to 1. According to https://developer.android.com/reference/android/view/Surface.html#ROTATION_0, a value of 1 is a 90 degree clockwise rotation, ie: landscape mode. So that would explain why the test failed.

As for how user_rotation got changed, either:
a) Our initial provisioning failed.
b) Some test is tweaking it.

I'll add some logging to rule-out option a, but regardless the solution will likely look the same: ensure system settings are correct before and/or after tasks. I'll get on that...

 Issue 809123  has been merged into this issue.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/28fd8fcd8fc8ecda591e33128987d837a405d8e9

commit 28fd8fcd8fc8ecda591e33128987d837a405d8e9
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Feb 21 23:43:34 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/3275e8f93e0c1789e22de2a8a92bfbfa4dbbd547

commit 3275e8f93e0c1789e22de2a8a92bfbfa4dbbd547
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Thu Feb 22 18:41:50 2018

After the change in #32, device system settings should be corrected in-between tasks. So the failure mode in #28 should only affect the tests that are messing with the settings in the first place (if there are any).

Let me know if you see any more test failures due to orientation problems.

Thank you Ben!

I hope / think that Telemetry isn't changing the device's orientation, but maybe it was. Hopefully it was a previous test run on the device which switched it to landscape orientation. Wranglers (cblume/fjhenigman right now), can you please keep an eye on the gpu.fyi waterfall via sheriff-o-matic and see if any similar failures show up? Thanks.

Post weekend ping: have we noticed any recent failures? I'd really like to see one if we have.

I didn't see one yet. The frequency before was pretty low, once in ~60 builds.
We can wait a bit more, or just close this and reopen if we see it happening again.

Still no failures, let's call the Android bug fixed.
Last Mac failure was on Jan 31st, let's open a new bug if it reoccurs.
https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/639887

Thanks for fixing, Ben!

FWIW, I think I found the source of the mysterious landscape-causing-tests. I've noticed that after every VR test on devices (ie: chrome_public_test_vr_apk) bot logs complained about "user_rotation" being the wrong value. And it's only after vr tests. I think that setting gets changed when the vr app opens (since I don't think portrait VR'ing is supported...)

Given that the bots can now heal themselves when this happens, it's not much of a problem. So I don't think there's a really a follow-up; just posting this for posterity.

+klausw, billorr, and bajones so that they know the VR tests seem to have left the bots in this state.

Thank you Ben for fixing this and making the infrastructure more robust.

Brian, Michael - if we are leaving the phone in landscape and breaking future tests could we be leaving the phone in landscape and breaking user scenarios after exiting VR?

I don't think so - my guess as to what's happening is that we're setting landscape orientation here on VR entry https://cs.chromium.org/chromium/src/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/VrShellDelegate.java?q=orientation+file:%5Esrc/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/+package:%5Echromium$&dr=CSs&l=1171, and we're supposed to set it back to the previous value during VR shutdown here https://cs.chromium.org/chromium/src/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/VrShellDelegate.java?q=orientation+file:%5Esrc/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/+package:%5Echromium$&dr=CSs&l=1191.

However, since Chrome gets killed at the end of a test, we don't ever actually call shutdownVR (and by extension restoreWindowMode) if we're in VR at the end of the test.

We could try to add code to always exit VR at the end of a test before Chrome gets killed if this is still problematic, but I don't think this should be an issue for actual users.

We do call shutdownVR when the Activity is destroyed, so we should be restoring the window mode. At any rate, when the Activity is destroyed Android shouldn't be preserving its orientation preference anyways...

I tried adding logging to shutdownVR, but it never got run during WebVrTransitionTest#testRequestPresentEntersVr (one of the tests that finishes while still in VR).

Should this test no longer be marked as flaky in pixel_expectations.py if this is fixed?

Yes, thanks for pointing that out - it should be unmarked. Essentially https://chromium-review.googlesource.com/905645 should be reverted. I'll remove them.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0c4f27e2b3c4f9da5086b2b4d74fe7d8c67e9f71

commit 0c4f27e2b3c4f9da5086b2b4d74fe7d8c67e9f71
Author: Kenneth Russell <kbr@chromium.org>
Date: Wed Mar 28 00:41:24 2018

Remove flaky expectations for several GPU pixel tests.

Device provisioning has been made more robust and these should no
longer be flaky.

TBR=enne@chromium.org, ynovikov@chromium.org, khushalsagar@chromium.org
NOTRY=true

Bug:  807370 ,  809846 ,  809868 ,  810006 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I4ddd9d7e3282e8430ec0c2c4a7225657e0442ebe
Reviewed-on: https://chromium-review.googlesource.com/982522
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#546339}
[modify] https://crrev.com/0c4f27e2b3c4f9da5086b2b4d74fe7d8c67e9f71/content/test/gpu/gpu_tests/pixel_expectations.py