File "/b/swarm_slave/w/irtkTCz_/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 732, in _recv
raise WebSocketTimeoutException(e.message)
TimeoutException:

Hmm, looks like network timeout? Might be that the network setup for the containers doesn't agree with the tests. I'm taking build245-m4 offline for a bit to debug.

I'm using blink_perf.paint_slimmingpaintinvalidation to debug since it keeps failing on on swarming, but always passes on buildbot: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5X%20Perf%20%283%29

It passes on my local N5X.
It fails inside 00e7a97549912611's container on build245-m4.
It fails in the same way on 00e7a97549912611 outside any container.
Maybe something's just wrong with 00e7a97549912611 on the bot...?

Hmmmm, I reflashed 00e7a97549912611 and now blink_perf.paint_slimmingpaintinvalidation is passing on it when running outside and inside a container.

Maybe swarming's post-task device cleanup isn't doing a good enough job?

Anyway, I'll reflash the remaining devices and throw build245 back in the pool and see if that helps.

Good news is: the latest build didn't have any real test failures: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/127

All the red tests are due to the swarming tasks expiring. I can slim down the swarming overhead for these bots, which should increase the turn-around time for each task.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/8264839b25a1dba6efbb4521bec04149ab7bd5c1

commit 8264839b25a1dba6efbb4521bec04149ab7bd5c1
Author: bpastene <bpastene@chromium.org>
Date: Fri Jan 06 18:36:18 2017

swarming: Roll py-adb to 75477ebf4fb8b906707f35e5fb385a9203256bef

75477ebf4fb8b906707f35e5fb385a9203256bef python-adb: Make FindDevicesSafe true to its name.
42558a58e1be016f84b2cbf43a10511e1248ded9 python-adb: Ignore devices that don't expose a serial number.

TBR=stip@chromium.org
BUG= 670879 ,  678046 

Review-Url: https://codereview.chromium.org/2618043002

[modify] https://crrev.com/8264839b25a1dba6efbb4521bec04149ab7bd5c1/appengine/third_party/python-adb/README.swarming
[modify] https://crrev.com/8264839b25a1dba6efbb4521bec04149ab7bd5c1/appengine/third_party/python-adb/adb/common.py
[modify] https://crrev.com/8264839b25a1dba6efbb4521bec04149ab7bd5c1/appengine/third_party/python-adb/adb/contrib/high.py

Down to 5 failures: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/133

loading.mobile: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/133/steps/loading.mobile%20on%20Android/logs/stdio
smoothness.top_25_smooth: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/133/steps/smoothness.top_25_smooth%20on%20Android/logs/stdio
v8.top_25_smooth: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/133/steps/v8.top_25_smooth.reference%20on%20Android/logs/stdio
webrtc.peer_connection: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/133/steps/webrtc.peerconnection%20on%20Android/logs/stdio
webrtc.peer_connection.reference:
https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/133/steps/webrtc.peerconnection.reference%20on%20Android/logs/stdio

Note that the regression sheriffs pointed out a regression in loading.mobile so that timeout shouldn't be happening, it is specific to these sets of bots. 

https://chromeperf.appspot.com/group_report?sid=47cc26a57499d845efc76abdc3177ff3838733b0da7f043cdca2809e8d5dcfb2

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6c327310f57ea5fd9bb692b5c565917191ce4e3

commit c6c327310f57ea5fd9bb692b5c565917191ce4e3
Author: bpastene <bpastene@chromium.org>
Date: Fri Jan 13 19:08:41 2017

Add custom swarming timeouts for chromium.perf.* tests.

BUG= 680174 ,  678046 

Review-Url: https://codereview.chromium.org/2621383003
Cr-Commit-Position: refs/heads/master@{#443633}

[modify] https://crrev.com/c6c327310f57ea5fd9bb692b5c565917191ce4e3/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/c6c327310f57ea5fd9bb692b5c565917191ce4e3/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/c6c327310f57ea5fd9bb692b5c565917191ce4e3/tools/perf/generate_perf_json.py

Latest build has 3 failures: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/161
smoothness.scrolling_tough_ad_cases
power.tough_ad_cases
blink_perf.canvas

All these are real failures and are also failing on the non-swarming buildbot builders:
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/4280
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5X%20Perf%20%282%29/builds/2828

We've reached buildbot-parity!

\o/

https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester/builds/195 :D

Now if I can just keep it green. It seems the main waterfall N5X bots are having similar problems, so I won't look too deeply into the current failures.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/c3747dd4f63eeb124b054e972a19f78727accf7b

commit c3747dd4f63eeb124b054e972a19f78727accf7b
Author: Benjamin Pastene <bpastene@google.com>
Date: Wed Feb 22 01:36:03 2017

I think this is pretty much done. All failing tests are also failing on the main waterfall N5X bots.

Note the giant string of purple recently was me accidentally moving the bots to another swarming server.