Now there seems to be cred/oauth issues along with the hanging downloads:
https://chromium-swarm.appspot.com/task?id=36742e2214c12c10
https://chromium-swarm.appspot.com/task?id=367419f323941f10
https://chromium-swarm.appspot.com/task?id=3673e84a821ec410

We have tools/perf/fetch_benchmark_deps.py script that fetch all the WPR archives used by Telemetry, however, I need to be able to conditionally invokes it only when GYP_DEFINES define fetch_telemetry_dependencies=1.

Look like  issue 570091  gonna take a while, so I gonna whip up something simple for now.

FWIW, the root cause for a lot of the failures yesterday was bug 727945. That said, it was particularly bad here due to telemetry_perf_unittests' reliance on google storage.

Thanks! I plan to make Telemetry fetches all the dependencies during the gclient sync step, so that we wouldn't need to rely on network during test time.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ad2bd774f996150e57e1fc2ef26939f268bf6c37

commit ad2bd774f996150e57e1fc2ef26939f268bf6c37
Author: nednguyen <nednguyen@google.com>
Date: Sat Jun 03 05:28:50 2017

Refactor fetch_benchmark_deps.py script to allow fetching all benchmark deps

After this:
fetch_benchmark_deps.py <benchmark foo>: fetch deps of benchmark foo

fetch_benchmark_deps.py: fetch all deps of all benchmarks

fetch_benchmark_deps.py -f: fetch all deps of all benchmarks without
prompt question

BUG= 725516 
TBR=eyaich@chromium.org

Review-Url: https://codereview.chromium.org/2914843002
Cr-Commit-Position: refs/heads/master@{#476891}

[modify] https://crrev.com/ad2bd774f996150e57e1fc2ef26939f268bf6c37/tools/perf/fetch_benchmark_deps.py
[modify] https://crrev.com/ad2bd774f996150e57e1fc2ef26939f268bf6c37/tools/perf/fetch_benchmark_deps_unittest.py

Issue 728990 has been merged into this issue.

The situation is very severe on linux_android_rel_ng. Since we already have coverage on n5x_android_rel_ng, I will disabling the whole telemetry_perf_unittest on linux_android_rel_ng bot for now.

These two logs so what went wrong on linux_android_rel_ng:

linux_android_rel_ng:
https://chromium-swarm.appspot.com/task?id=36934cfec660f510&refresh=10&show_raw=1 
...
INFO:root:All WPR archives are downloaded, took 359.283438921 seconds.
...
INFO:root:All WPR archives are downloaded, took 705.20665884 seconds.

android_n5x_swarming_rel:https://chromium-swarm.appspot.com/task?id=36936f8a888d5010&refresh=10&show_raw=1https://chromium-swarm.appspot.com/task?id=36936f8a888d5010&refresh=10&show_raw=1
...
INFO:root:All WPR archives are downloaded, took 26.4820959568 seconds.
...
INFO:root:All WPR archives are downloaded, took 36.1329550743 seconds.
...

So look like the network speed of linux_android_rel_ng regressed recently (we still shouldn't fetch live network during test time anyway).

 Issue 729864  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b51378065a1d1ede626abeab2122e95191f49db4

commit b51378065a1d1ede626abeab2122e95191f49db4
Author: nednguyen <nednguyen@google.com>
Date: Tue Jun 06 01:29:12 2017

Stop running telemetry_perf_unittest on linux_android_rel_ng

BUG= 725516 
NOTRY=true

Review-Url: https://codereview.chromium.org/2928463003
Cr-Commit-Position: refs/heads/master@{#477145}

[modify] https://crrev.com/b51378065a1d1ede626abeab2122e95191f49db4/testing/buildbot/chromium.linux.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/534722a4d48581f5216bdf98d98d283ae478849a

commit 534722a4d48581f5216bdf98d98d283ae478849a
Author: nednguyen <nednguyen@google.com>
Date: Tue Jun 06 18:02:43 2017

Refactor how we fetch Telemetry deps conditionally

Before this, we conditionally fetch Telemetry deps through checking GYP_DEFINES
variable in https://github.com/catapult-project/catapult/blob/master/telemetry/bin/fetch_telemetry_binary_dependencies#L18

This refactors it by adding tools/perf/conditionally_execute which allows conditioning
executing any script on GYP_DEFINES flag.

BUG= 725516 

Review-Url: https://codereview.chromium.org/2921353002
Cr-Commit-Position: refs/heads/master@{#477339}

[modify] https://crrev.com/534722a4d48581f5216bdf98d98d283ae478849a/DEPS
[add] https://crrev.com/534722a4d48581f5216bdf98d98d283ae478849a/tools/perf/conditionally_execute

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/acf10c1ba215cfe40d6ebe587fb9078f884402ac

commit acf10c1ba215cfe40d6ebe587fb9078f884402ac
Author: agrieve <agrieve@chromium.org>
Date: Tue Jun 06 20:18:39 2017

Stop running telemetry_perf_unittest on Android Tests (dbg)

Temporary until related bug is fixed.

BUG= 725516 
NOTRY=true
TBR=dpranke

Review-Url: https://codereview.chromium.org/2931483002
Cr-Commit-Position: refs/heads/master@{#477386}

[modify] https://crrev.com/acf10c1ba215cfe40d6ebe587fb9078f884402ac/testing/buildbot/chromium.linux.json

It seems like this bot is hit by this: https://build.chromium.org/p/chromium.mac/builders/Mac10.11%20Tests

This seems like a bug in typ: 
https://chromium-swarm.appspot.com/task?id=36ba31dcf9cb4810&refresh=10&show_raw=1

Dirk: can you take a look/triage?

[6/27] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu failed unexpectedly 0.0000s:
  Failed to load "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu" in run_one_test
    loadTestsFromName("benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu") failed: 'module' object has no attribute 'SystemHealthBenchmarkSmokeTest'
    Traceback (most recent call last):
      File "/b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py", line 822, in _run_one_test
        suite = child.loader.loadTestsFromName(test_name)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
        parent, obj = obj, getattr(obj, part)
    AttributeError: 'module' object has no attribute 'SystemHealthBenchmarkSmokeTest'

...
Retrying failed tests (attempt #3 of 3)...

[1/1] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu failed unexpectedly 0.0000s:
  Failed to load "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu" in run_one_test
    loadTestsFromName("benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu") failed: 'module' object has no attribute 'SystemHealthBenchmarkSmokeTest'
    Traceback (most recent call last):
      File "/b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py", line 822, in _run_one_test
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
        parent, obj = obj, getattr(obj, part)
    AttributeError: 'module' object has no attribute 'SystemHealthBenchmarkSmokeTest'
    
    
    load_via_load_tests("benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu") returned 0 tests

I don't understand how the code loads the test in question. Where is "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:baidu" defined? 

From the stack trace, it looks like the load_tests() protocol failed to actually load that individual test, but I can't tell why. Assuming that test should've existed, is it possible that load_tests() in system_health_benchmark_smoke_test.py raised an exception?

Did we see this test fail anywhere else? Or did we any other similar-looking failures?

Can you please look into these things first? typ hasn't changed in quite some time and there's nothing particularly special about the load_test() code so if it's working at all I'd expect it to be working completely. I'm not saying there isn't a bug in typ, but it's not the first place I'd look.

Also, given that this crash happened on Mac10.11 and appears to have little to nothing to do w/ the WPR downloads on android, this seems like the wrong bug for this, so I'd move the stuff in comments #15 and #16 to a new bug.

 Issue 693078  has been merged into this issue.

 Issue 739455  has been merged into this issue.

This also hangs on Mac10.11 Tests:
https://build.chromium.org/p/chromium.mac/builders/Mac10.11%20Tests/builds/15209
https://chromium-swarm.appspot.com/task?id=375701f1ed354110&refresh=10&show_raw=1

Agree with dpranke@ that comments 15-16 as well as 21 happen on mac and therefore seems like does not belong here.   Bug 739455  is a mac bug and was marked as duplicate of this one.  I will move comments 15, 16, and 21 to the latter bug and reopen it.

Going back to this, I will try to reenable this suite on linux_android_rel_ng

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bee340f1f3b0a00baadb422e1939541618b79ba6

commit bee340f1f3b0a00baadb422e1939541618b79ba6
Author: nednguyen <nednguyen@google.com>
Date: Mon Nov 13 22:12:26 2017

Enable telemetry_perf_unittest on linux_android_rel_ng

Back then, telemetry_perf_unittest was removed from linux_android_rel_ng because it occasionally get stuck while fetching dependent files from GS.
Two things have changed since then:
1) linux_android_rel_ng is swarmed, which mean we don't run telemetry_perf_unittest in multi-process mode.
2) Dependent telemetry files are prefetched during "gclient sync" ( crbug.com/780967 )

This CL reenables telemetry_perf_unittest on linux_android_rel_ng to get coverage on Nexus5 devices.

Bug:  725516 
Change-Id: Ib5ff018be5d702c3d60ee3a09cf5fdfb36b605d5
Reviewed-on: https://chromium-review.googlesource.com/764570
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#516075}
[modify] https://crrev.com/bee340f1f3b0a00baadb422e1939541618b79ba6/testing/buildbot/chromium.android.json
[modify] https://crrev.com/bee340f1f3b0a00baadb422e1939541618b79ba6/tools/perf/page_sets/system_health/expectations.py