This looks quit flaky. The problem went away after 4 builds, but I already found happening at build #19026 (the same error message), so I say the blame range is unclear.

See:
https://chromium-swarm.appspot.com/task?id=391750ccecc9fb10&refresh=10&show_raw=1

Tentatively reassigning to performance (bot) sheriffs, as I don't think it's actionable by off-the-shelf generic sheriffs.

Sorry, bunch of typos: quite*, found it*, I'd*

+nednguyen, SystemHealthBenchmarkSmokeTest does seem quite flaky on this platform. I see a number of failures in the Mac10.11 Tests history.

Sample run:
https://chromium-swarm.appspot.com/task?id=392efb375d562010

No longer happening

This seems to still be happening:
https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.11%20Tests/builds/19506

[3/10] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:news:qq failed unexpectedly 11.2861s:
  Traceback (most recent call last):
    File "/b/s/w/ir/tools/perf/benchmarks/system_health_smoke_test.py", line 110, in RunTest
      story_set = benchmark_class().CreateStorySet(options)
    File "/b/s/w/ir/tools/perf/benchmarks/system_health.py", line 113, in CreateStorySet
      take_memory_measurement=True)
    File "/b/s/w/ir/tools/perf/page_sets/system_health/system_health_stories.py", line 33, in __init__
      self.AddStory(story_class(self, take_memory_measurement))
    File "/b/s/w/ir/tools/perf/page_sets/system_health/accessibility_stories.py", line 25, in __init__
      extra_browser_args=['--force-renderer-accessibility'])
    File "/b/s/w/ir/tools/perf/page_sets/system_health/system_health_story.py", line 71, in __init__
      extra_browser_args=extra_browser_args)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 47, in __init__
    File "/b/s/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 413, in GetIfChanged
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
      return self.gen.next()
    File "/b/s/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 266, in _FileLock
    File "/b/s/w/ir/third_party/catapult/common/py_utils/py_utils/__init__.py", line 148, in WaitFor
  TimeoutException: Timed out while waiting 10s for <lambda>.
Failed to delete /b/s/w/ir (3 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 2 seconds.
Succeeded.

The starting error here is:

[3/26] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:pinterest failed unexpectedly 0.0000s:
  Failed to load "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:pinterest" in run_one_test
    loadTestsFromName("benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:pinterest") failed: 'module' object has no attribute 'SystemHealthBenchmarkSmokeTest'
    Traceback (most recent call last):
      File "/b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py", line 853, in _run_one_test
        suite = child.loader.loadTestsFromName(test_name)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
        parent, obj = obj, getattr(obj, part)
    AttributeError: 'module' object has no attribute 'SystemHealthBenchmarkSmokeTest'
    
    
    load_via_load_tests("benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:pinterest") returned 0 tests

(from https://chromium-swarm.appspot.com/task?id=39613f5598761310&refresh=10&show_raw=1)

I think this has to do with the logic of generating tests in SystemHealthBenchmarkSmokeTest

Dirk:
system_health_smoke_test generate it test cases through https://cs.chromium.org/chromium/src/tools/perf/benchmarks/system_health_smoke_test.py?rcl=1dea5ee048fa7c57e2af0c47ac989ba5b4b25a64&l=163 

What would make typ fail to load "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:pinterest" flakily? Does typ invoke load_tests() once before running all the tests in parallel, or does it invoke this for every test process?

If this is the latter, we probably can fix the flake by just switching telemetry_perf_unittests on  chromium.mac/Mac10.11 Tests to use --jobs=1 (related to issue 753495)

 Issue 739455  has been merged into this issue.

It does both, first to load the full list of tests, and then second to load each individual test in the appropriate worker subprocess. Using --jobs=1 might help, but I'm not 100% sure (we should still call load_tests() twice).

This has not happened in the last 100 builds. Suggest it's because my switch to "--jobs=1" (issue 753495)