FYI, I had some issues with this step earlier, which I assumed flake with https://chromium.googlesource.com/chromium/src/+/80d2dd066eff7bac3dc0060bfe89067db48d17cf

Popping up on windows, too: https://luci-milo.appspot.com/buildbot/chromium.win/Win7%20Tests%20%281%29/70011

On https://uberchromegw.corp.google.com/i/chromium.linux/builders/Linux%20Tests/ the last 3 runs have the same test timing out:

benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.memory.long_running_idle_gmail_background_tbmv2

rnephew's found the culprit & is uploading a fix.

well, identified *a* culprit. The timing isn't quite right, though...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2c52141a3817cf635e60e5306891f637bc45cafc

commit 2c52141a3817cf635e60e5306891f637bc45cafc
Author: rnephew <rnephew@chromium.org>
Date: Wed Aug 09 23:18:56 2017

[Telemetry] Add ability to disable by test name to telemetry_perf smoke tests.

It also disables memory.long_running_idle_gmail_background_tbmv2

TBR=jbudorick@chromium.org, nednguyen@google.com

No-Try: true
Bug:  754011 
Change-Id: I3a90c4ab42f8777746c4f284c4e6e2efc5dafae4
Reviewed-on: https://chromium-review.googlesource.com/608583
Reviewed-by: rnephew <rnephew@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: rnephew <rnephew@chromium.org>
Cr-Commit-Position: refs/heads/master@{#493173}
[modify] https://crrev.com/2c52141a3817cf635e60e5306891f637bc45cafc/tools/perf/benchmarks/benchmark_smoke_unittest.py

It looks like there might be errors in the test runner code, not the tests themselves?

Just had a bunch of failures come in:
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/521609
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/521603
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/521531
 
I inspected the failure stdout of the telemetry tests and found an uncaught Python exception at the bottom (https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F521603%2F%2B%2Frecipes%2Fsteps%2Ftelemetry_perf_unittests__with_patch_%2F0%2Fstdout)

  Call site info:   File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 278, in Close
      self._browser_backend.Close()
    File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py", line 657, in Close
      shutil.rmtree(self._tmp_profile_dir, ignore_errors=True)
  
  rmtree is invoked with arguments: ('/b/s/w/itYWoXpm/tmpCeYJrr',) {'ignore_errors': True}
  Call site info:   File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 278, in Close
      self._browser_backend.Close()
    File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py", line 665, in Close
      shutil.rmtree(self._tmp_minidump_dir, ignore_errors=True)
  
  Browser is closed.
  rmtree is invoked with arguments: ('/b/s/w/itYWoXpm/tmpxARkiV',) {}
  Call site info:   File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/results/page_test_results.py", line 194, in PopulateHistogramSet
      vinn_result = convert_chart_json.ConvertChartJson(chart_json_path)
    File "/b/s/w/ir/third_party/catapult/tracing/tracing/value/convert_chart_json.py", line 24, in ConvertChartJson
      js_args=[os.path.abspath(chart_json)])
    File "/b/s/w/ir/third_party/catapult/third_party/vinn/vinn/_vinn.py", line 188, in RunFile
      shutil.rmtree(temp_dir)
  
  Traceback (most recent call last):
    File "/b/s/w/ir/tools/perf/benchmarks/benchmark_smoke_unittest.py", line 85, in BenchmarkSmokeTest
      msg='Failed: %s' % benchmark)
  AssertionError: Failed: <class 'benchmarks.tab_switching.TabSwitchingTypical25'>

On https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/521609

The shard 0 failure looks like it is the test failing, and then timing out during retries:
[1/1] benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.tab_switching.typical_25 failed unexpectedly 222.2724s:

The shard 5 failure has no failure messages but is just timing out. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6a4b7b66071f53cc86cf077f0bfde99e53203e7

commit c6a4b7b66071f53cc86cf077f0bfde99e53203e7
Author: rnephew <rnephew@chromium.org>
Date: Thu Aug 10 00:09:52 2017

[Telemetry] Disable tab_switching.typical_25 during telemetry_perf_unittest.

TBR=jbudorick@chromium.org, nednguyen@google.com

No-Try: true
Bug:  754011 
Change-Id: I15476311d83305f96d397606b2f836a2088c893e
Reviewed-on: https://chromium-review.googlesource.com/609475
Reviewed-by: rnephew <rnephew@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: rnephew <rnephew@chromium.org>
Cr-Commit-Position: refs/heads/master@{#493185}
[modify] https://crrev.com/c6a4b7b66071f53cc86cf077f0bfde99e53203e7/tools/perf/benchmarks/benchmark_smoke_unittest.py

w/ the CLs in #6 and #9, this appears to be fixed.

#8 started timing out again:

https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.11%20Tests/builds/16429

#11: please file a separate bug for that.

This is a new and unrelated problem I think. I'm probably not the proper owner, but I can put some insight into whats going on. Please CC me to the new bug.

 Issue 753542  has been merged into this issue.