GPU's the first to have P automation.

(though I have successfully run tests against P locally)

I have tried it locally with the same OS version, same gn args and almost the same run_gpu_integration_test.py args, and it worked.
I don't know how the swarming environment differs.

Thanks Yuly for trying this out.

How can we proceed? Maybe the Swarming bots do something in their setup which doesn't quite work on P, and it breaks Chromium's timers, or similar.

Perhaps I was to early to celebrate. Looks like the tests start failing at some point even locally. Unfortunately, I didn't log into a file, so I only had logs of last ~90 tests out of ~2700 when I came in today, so I don't have good info.

I've started running the suite again, and some tests are failing, but the suite seems to recover after that, even though I get "Chrome keeps crashing" window on the phone. One pattern that worries me a bit is this:

  [host]> /usr/local/google/work/clank/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s FA8121A02969 forward --no-rebind tcp:35123 localabstract:chrome_devtools_remote
  AndroidReverseForwarder started between 127.0.0.1:35123 and localabstract:chrome_devtools_remote
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Websocket at ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote) not yet ready: [Errno 104] Connection reset by peer
  Got devtools config: ws://127.0.0.1:35123/devtools/browser (remote=localabstract:chrome_devtools_remote)
  Successfully connected to org.chromium.chrome DevTools client

do note that the devil version of adb is *very* out of date at this point -- iirc, I tried updating it to 1.0.36 or 1.0.39 a while back only to have it get reverted after breaking some subset of telemetry tests. dunno if that's responsible, but it's worth revisiting once I upgrade platform-tools in chromium.

I just finished a second local run, and didn't see as many failures as before.
Other than the errors in #6, didn't see anything else in the logs.
One other concern I have is that P doesn't seem to like us restarting the browser - it think that the application keeps crashing and shows a pop-up asking what to do with it. Also seems to navigate to the App settings where the Uninstall button is. Hope this doesn't interfere with the testing.

Thanks for re-testing. We should report these issues to the Android team. Do you have any appropriate contacts?

Raising to P2, because we should start addressing these issues now.

dansnyder@ found the same problem was happening on some Cast devices and proposed a patch which I've uploaded for him at https://chromium-review.googlesource.com/1142238 .

Linking to related bugs.

John, is there a bug for #7? Sounds like updating adb is something we should be doing sooner rather than later.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0ebb714c18a2ec435ac6ed9532435330dcbef44d

commit 0ebb714c18a2ec435ac6ed9532435330dcbef44d
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Jul 20 22:03:35 2018

Handle negative times in TimeClamper::ClampTimeResolution.

Patch from dansnyder@.

Negative times are being seen on certain platforms (the Android P
preview and some Cast devices, in particular). Handle them gracefully
rather than asserting.

Bug:  856858 
Tbr: jochen@chromium.org
Change-Id: Ib05cb8534bec9e6a95df2f989a9d65fedfeece0b
Reviewed-on: https://chromium-review.googlesource.com/1142238
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577005}
[modify] https://crrev.com/0ebb714c18a2ec435ac6ed9532435330dcbef44d/gin/v8_platform.cc
[modify] https://crrev.com/0ebb714c18a2ec435ac6ed9532435330dcbef44d/third_party/blink/renderer/platform/time_clamper.cc
[modify] https://crrev.com/0ebb714c18a2ec435ac6ed9532435330dcbef44d/third_party/blink/renderer/platform/time_clamper_test.cc

Trying running WebGL 2.0 conformance on Android P again after the above fix:

https://chromium-review.googlesource.com/1115802
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/gpu-manual-try-android-p-pixel-2-32/3

Looks like it worked!
All that's left is to suppress the 43 failing tests :(

It's a relief that the browser now runs via Telemetry on this device. Let's track the suppression of the new test failures in  Issue 858879 .

Woo hoo, thanks for the hard work pioneering Telemetry tests on this device!