Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug


Sign in to add a comment
"telemetry_perf_unittests (with patch)" is flaky due to whole suite timed out
Project Member Reported by chromium...@appspot.gserviceaccount.com, Aug 15 2016 Back to list
"telemetry_perf_unittests (with patch)" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 4 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA.



This flaky test/step was previously tracked in issue 635255.
 
Project Member Comment 1 by chromium...@appspot.gserviceaccount.com, Aug 16 2016
Detected 18 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Owner: bcwh...@chromium.org
Status: Started
Cc: gab@chromium.org
Owner: asvitk...@chromium.org
Status: Assigned
This has apparently been happening since at least mid-May.

According to https://codereview.chromium.org/2236383002 ...
Some attempts have been made to fix this but a real fix is coming from asvitkine.

Labels: -Sheriff-Chromium
I don't think it's caused by the same issue. What's the stack trace?
Also, I'm ooo until August 28 - so if you want someone to look at this before then, i suggest finding a different owner.
Project Member Comment 7 by chromium...@appspot.gserviceaccount.com, Aug 18 2016
Detected 13 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 8 by chromium...@appspot.gserviceaccount.com, Aug 21 2016
Detected 11 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 9 by chromium...@appspot.gserviceaccount.com, Aug 25 2016
Detected 46 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Cc: asvitk...@chromium.org
Labels: Sheriff-Chromium
Owner: ----
Status: Available
Marking available and putting the Sheriff-Chromium label back.

I don't think this is the same issue as https://bugs.chromium.org/p/chromium/issues/detail?id=635255 unless there's evidence otherwise.

These tests can fail if there's a startup crash. So sheriff need to get the stack trace for that to figure out who should own this. 
Components: Infra>Client>Android
I perused some of these failures during my sheriff stint today. There are various errors. Today the flakes stopped happening on non-android. I am not very familiar with android so I'm hoping someone from Infra can triage these.

https://chromium-swarm.appspot.com/user/task/30efee9b3f25b210
ctrl-f 232/
(1) "Failed to install test certificate authority on target platform. Browsers may fall back to ignoring certificate errors."
(2) Cascades to: Traceback (most recent call last):
    _RunStoryAndProcessErrorIfNeeded at /b/swarm_slave/w/irUDEYK1/third_party/catapult/telemetry/telemetry/internal/story_runner.py:104
      None
    DidRunStory at /b/swarm_slave/w/irUDEYK1/third_party/catapult/telemetry/telemetry/page/shared_page_state.py:159
      None
  AttributeError: 'NoneType' object has no attribute 'credentials'
(3) Then many following tests fail in story_set.py, like:
 File "/b/swarm_slave/w/irUDEYK1/tools/perf/measurements/v8_gc_times_unittest.py", line 93, in testWithGarbageCollectionEvents
    File "/b/swarm_slave/w/irUDEYK1/third_party/catapult/telemetry/telemetry/testing/page_test_test_case.py", line 45, in CreateEmptyPageSet
    File "/b/swarm_slave/w/irUDEYK1/third_party/catapult/telemetry/telemetry/story/story_set.py", line 41, in __init__
  ValueError: Invalid directory path of base_dir: /b/swarm_slave/w/irUDEYK1/third_party/catapult/telemetry/telemetry/internal/testing

Maybe the scripts don't handle test-CA installation failures very gracefully? And who knows why the test CAs weren't installed correctly in the first place.

https://chromium-swarm.appspot.com/user/task/30ede4911aed9e10
Tests were disabled in issue 637938, so yay.

Will look again tomorrow.
Project Member Comment 12 by chromium...@appspot.gserviceaccount.com, Aug 30 2016
Detected 7 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 13 by chromium...@appspot.gserviceaccount.com, Aug 31 2016
Detected 9 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Comment 14 by gab@chromium.org, Sep 1 2016
Cc: sullivan@chromium.org
I don't think these are the same flakes anymore.
Cc: nednguyen@chromium.org
Cc: petrcermak@chromium.org
I filed issue 643320 about recent flakes of telemetry_perf_unittest
Project Member Comment 18 by chromium...@appspot.gserviceaccount.com, Sep 1 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 19 by chromium...@appspot.gserviceaccount.com, Sep 3 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
For the last 10 flakes:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/292253: issue 629434
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/135553: CommandTimeoutError (https://chromium-swarm.appspot.com/user/task/31059fbddb439510)
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/273907: issue 643320
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/286799: issue 643320
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/286427: issue 643320
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/273453: issue 643320
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/291109: issue 629434
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/273350: issue 629434
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/134827: CommandTimeoutError (https://chromium-swarm.appspot.com/user/task/310075195a440410)
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/134822: AdbCommandFailedError (https://chromium-swarm.appspot.com/user/task/310073af23ee1510)

There seem to be three causes of failures:

  1. issue 629434 (3/10)
  2. issue 643320 (4/10)
  3. Android device issues (3/10)
Labels: -Sheriff-Chromium
The very last failure is timeout on Android (https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/136335)
Comment 22 by gab@chromium.org, Sep 6 2016
Cc: -gab@chromium.org
Removing self per #14.
Blockedon: 629434 643320
Blockedon: 644270
Project Member Comment 25 by chromium...@appspot.gserviceaccount.com, Sep 6 2016
Labels: Sheriff-Chromium
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Project Member Comment 26 by chromium...@appspot.gserviceaccount.com, Sep 7 2016
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
All of the last 5 failures seem to be due to issue 643320.
Project Member Comment 28 by chromium...@appspot.gserviceaccount.com, Sep 8 2016
Detected 7 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 29 by bugdroid1@chromium.org, Sep 10 2016
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4b1e75d9334ca63343d5e44768c246d796c8386

commit c4b1e75d9334ca63343d5e44768c246d796c8386
Author: nednguyen <nednguyen@google.com>
Date: Sat Sep 10 12:00:05 2016

Manually roll src/third_party/catapult/ da6d44e4a..c0b988891 (3 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/da6d44e4a2a5..c0b988891fd1

$ git log da6d44e4a..c0b988891 --date=short --no-merges --format='%ad %ae %s'
2016-09-09 erikchen Add a new cpu time TBMv2 system health metric.
2016-09-09 nednguyen Update the lock operation in cloud_storage with better implementation using py_utils.lock
2016-09-09 rnephew [BattOr] Kill BattOr shell if there is a problem with communicating during clock sync.

BUG=640312,637904,645106,645720

TBR=catapult-sheriff@chromium.org, jbudorick@chromium.org

Review-Url: https://codereview.chromium.org/2326063004
Cr-Commit-Position: refs/heads/master@{#417833}

[modify] https://crrev.com/c4b1e75d9334ca63343d5e44768c246d796c8386/DEPS
[modify] https://crrev.com/c4b1e75d9334ca63343d5e44768c246d796c8386/build/android/test_runner.pydeps

Project Member Comment 30 by chromium...@appspot.gserviceaccount.com, Sep 10 2016
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Labels: -Sheriff-Chromium
Two recent failures:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/291856
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/140017
shows problems in the SystemHealthBenchmarkSmokeTest tests:

[7/23] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:social:facebook failed unexpectedly 0.0310s:
  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
  Traceback (most recent call last):
    File "e:\b\swarm_slave\w\irv8mnl9\tools\perf\core\path_util.pyc\..\..\benchmarks\system_health_smoke_test.py", line 85, in RunTest
      self.assertEqual(0, SinglePageBenchmark().Run(options),
    File "e:\b\swarm_slave\w\irv8mnl9\third_party\catapult\telemetry\telemetry\benchmark.py", line 91, in Run
      return story_runner.RunBenchmark(self, finder_options)
    File "e:\b\swarm_slave\w\irv8mnl9\third_party\catapult\telemetry\telemetry\internal\story_runner.py", line 304, in RunBenchmark
      stories = benchmark.CreateStorySet(finder_options)
    File "e:\b\swarm_slave\w\irv8mnl9\tools\perf\core\path_util.pyc\..\..\benchmarks\system_health_smoke_test.py", line 69, in CreateStorySet
      story_set = super(SinglePageBenchmark, self).CreateStorySet(options)
    File "e:\b\swarm_slave\w\irv8mnl9\tools\perf\core\path_util.pyc\..\..\benchmarks\system_health.py", line 103, in CreateStorySet
      take_memory_measurement=True)
    File "e:\b\swarm_slave\w\irv8mnl9\tools\perf\page_sets\system_health\system_health_stories.py", line 31, in __init__
      self.AddStory(story_class(self, take_memory_measurement))
    File "e:\b\swarm_slave\w\irv8mnl9\tools\perf\page_sets\system_health\system_health_story.py", line 75, in __init__
      grouping_keys={'case': case, 'group': group})
    File "e:\b\swarm_slave\w\irv8mnl9\third_party\catapult\telemetry\telemetry\page\__init__.py", line 43, in __init__
      cloud_storage.GetIfChanged(credentials_path, credentials_bucket)
    File "e:\b\swarm_slave\w\irv8mnl9\third_party\catapult\common\py_utils\py_utils\cloud_storage.py", line 339, in GetIfChanged
      with _FileLock(file_path):
    File "e:\b\depot_tools\python276_bin\lib\contextlib.py", line 17, in __enter__
      return self.gen.next()
    File "e:\b\swarm_slave\w\irv8mnl9\third_party\catapult\common\py_utils\py_utils\cloud_storage.py", line 243, in _FileLock
      fd = open(pseudo_lock_path, 'w')
  IOError: [Errno 13] Permission denied: 'e:\\b\\swarm_slave\\w\\irv8mnl9\\tools\\perf\\page_sets\\system_health\\../data/credentials.json.pseudo_lock'

Is this an infra problem?

Android one:
[143/284] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_mobile.load:games:lazors failed unexpectedly 41.9931s:
  [ RUN      ] load:games:lazors
  Traceback (most recent call last):
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 78, in _RunStoryAndProcessErrorIfNeeded
      test.WillRunStory(state.platform)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 274, in WillRunStory
      platform.tracing_controller.StartTracing(self._tbm_options.config)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 43, in StartTracing
      self._tracing_controller_backend.StartTracing(tracing_config, timeout)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 91, in StartTracing
      if agent.StartAgentTracing(config, timeout):
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 113, in StartAgentTracing
      self._platform_backend.SetGraphicsMemoryTrackingEnabled(True)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 238, in SetGraphicsMemoryTrackingEnabled
      '-d'], as_root=True, check_return=True)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
      retry_if_func=retry_if_func)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/utils/timeout_retry.py", line 157, in Run
      error_log_func=error_log_func)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
      self._JoinAll(watcher, timeout)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
      thread.ReraiseIfException()
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
      self._ret = self._func(*self._args, **self._kwargs)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda>
      child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
      return f(*args, **kwargs)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/device_utils.py", line 904, in RunShellCommand
      output = handle_large_output(cmd, large_output)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/device_utils.py", line 882, in handle_large_output
      return handle_large_command(cmd)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/device_utils.py", line 864, in handle_large_command
      return handle_check_return(cmd)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/device_utils.py", line 855, in handle_check_return
      return run(cmd)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/device_utils.py", line 851, in run
      return self.adb.Shell(cmd)
    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 494, in Shell
      command, output, status=status, device_serial=self._device_serial)
  AdbShellCommandFailedError: (device: 060f09f513c8594a) shell command run via adb failed on the device:
    command: /data/local/tmp/profilers/memtrack_helper -d
    exit status: 1
    output:
    - FATAL: /data/local/tmp/profilers/memtrack_helper must be run as root!
  
  
  [  FAILED  ] load:games:lazors (7307 ms)


Cc: jbudorick@chromium.org
To #31, the failure on win is tracked in issue 643320.

The failure on Android seems like an infra failure with us not having root on the device.
#31/#32: I don't think this is an infra failure. We do have root on the device, and we're attempting to run the command as root:

    File "/b/swarm_slave/w/ir0OxkXR/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 238, in SetGraphicsMemoryTrackingEnabled
      '-d'], as_root=True, check_return=True)

I'm guessing that we think we're running as root though we aren't, so we don't use su? Need to look into this a bit more.
er, actually, we may not have root on the device. If that's the case, though, devil is still doing a poor job of messaging.
These, on the other hand, are more indicative of possible device failure:

CRITICAL:root:(TimeoutThread-1-for-delete_temporary_file(060f09f513c8594a)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-db4f5fe7fb44f'], check_error=False, retries=2, device_serial=060f09f513c8594a, timeout=30), attempt 1 of 3: AdbCommandFailedError("(device: 060f09f513c8594a) adb shell 'rm -f /data/local/tmp/temp_file-db4f5fe7fb44f': failed with exit status 255 and output:\n- error: protocol fault (couldn't read status): Success\n",)
INFO:root:[host]> /b/swarm_slave/w/ir0OxkXR/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 060f09f513c8594a shell 'rm -f /data/local/tmp/temp_file-db4f5fe7fb44f'
CRITICAL:root:(TimeoutThread-2-for-delete_temporary_file(060f09f513c8594a)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-db4f5fe7fb44f'], check_error=False, retries=2, device_serial=060f09f513c8594a, timeout=30), attempt 2 of 3: AdbCommandFailedError("(device: 060f09f513c8594a) adb shell 'rm -f /data/local/tmp/temp_file-db4f5fe7fb44f': failed with exit status 255 and output:\n- error: device '060f09f513c8594a' not found\n",)

...

  CommandFailedError: (device: 060f09f513c8594a) pm path returned: 'Error: Could not access the Package Manager.  Is the system running?'
Project Member Comment 36 by chromium...@appspot.gserviceaccount.com, Sep 13 2016
Labels: Sheriff-Chromium
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Comment 37 by grt@chromium.org, Sep 14 2016
Labels: -Sheriff-Chromium
The last 5 flakes are issue 643320, which is unrelated to device problem. Popping out of sheriff queue.
Project Member Comment 38 by chromium...@appspot.gserviceaccount.com, Sep 19 2016
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Blockedon: 649167
Project Member Comment 40 by chromium...@appspot.gserviceaccount.com, Sep 22 2016
Detected 34 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Blockedon: 649216
Blockedon: 649217
Labels: -Sheriff-Chromium
The failures in #38 and #40:

* Timeout in benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_mobile.load:social:twitter on android_n5x_swarming_rel. Only seen once.
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/32924

* "FATAL:navigation_handle_impl.cc(507)] Check failed: state_ != WILL_REDIRECT_REQUEST || next_index_ == 0". Only seen once.
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/32913

* Issue 649167.

* Windows failures: issue 643320, afaict.

nednguyen@: Can you verify that the Windows failures are the same as that issue?
issue 643320 only happen once on win recently. Instead, I see issue 649217 & issue 649216 are the major cause of telemetry_perf_unittest' on win
Project Member Comment 45 by chromium...@appspot.gserviceaccount.com, Sep 23 2016
Labels: Sheriff-Chromium
Detected 37 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Project Member Comment 46 by chromium...@appspot.gserviceaccount.com, Sep 23 2016
Cc: stale-flakes-reports@google.com
Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.
Seems the problem is the infrastructure can't tell which specific part is flaky. Ideally, it should be able to file more targeted issues like it does for other test binaries.
Cc: st...@chromium.org
+Shuotao: do you know why the flakiness dashboard fails to show which specific tests in telemetry_perf_unittest suite are flaky? 
#48: because the step doesn't report its results to test-results in the gtest JSON format.
Comment 50 by st...@chromium.org, Sep 23 2016
Cc: kbr@chromium.org estaab@chromium.org
#48: I think John is right.

Probably we'd better switch telemetry test_runner to use gtest JSON format and gain the benefit automatically here.
Blockedon: 649762
Sounds good to me. I will file a different bug about that (issue 649762)
Cc: -stale-flakes-reports@google.com
Thank you for looking into this and filing a separate bug.
Project Member Comment 53 by chromium...@appspot.gserviceaccount.com, Sep 24 2016
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 54 by chromium...@appspot.gserviceaccount.com, Sep 26 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 55 by chromium...@appspot.gserviceaccount.com, Sep 26 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 56 by chromium...@appspot.gserviceaccount.com, Sep 27 2016
Detected 68 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 57 by chromium...@appspot.gserviceaccount.com, Sep 28 2016
Detected 9 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Cc: stip@chromium.org
Majority of the flakes of linux_android_rel_ng recently seems like infra/lab issue to me:
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/148010
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/147899

John, Mike: do we have a tracking bug for it?
Project Member Comment 59 by chromium...@appspot.gserviceaccount.com, Sep 29 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 60 by chromium...@appspot.gserviceaccount.com, Oct 1 2016
Detected 13 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 61 by chromium...@appspot.gserviceaccount.com, Oct 3 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Some of the Friday night flakes on the Android bots were due to me messing up the forwarder binary upgrade.

#58: that was https://bugs.chromium.org/p/chromium/issues/detail?id=650141
Project Member Comment 63 by chromium...@appspot.gserviceaccount.com, Oct 5 2016
Detected 38 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Cc: eyaich@chromium.org
Cc: -petrcermak@chromium.org
Cc: -petrcermak@chromium.org
Project Member Comment 67 by chromium...@appspot.gserviceaccount.com, Oct 6 2016
Detected 55 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 68 by chromium...@appspot.gserviceaccount.com, Oct 8 2016
Detected 35 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 69 by chromium...@appspot.gserviceaccount.com, Oct 10 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 70 by chromium...@appspot.gserviceaccount.com, Oct 11 2016
Detected 25 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Labels: -Sheriff-Chromium
I am chromium sheriff today.

It's difficult to follow this discussion. Let me remove Sheriff-Chromium since it looks there are already filed bugs other than this.

Please feel free to add Sheriff-Chromium again if my understanding is wrong.
Project Member Comment 72 by chromium...@appspot.gserviceaccount.com, Oct 12 2016
Labels: Sheriff-Chromium
Detected 8 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
Project Member Comment 74 by chromium...@appspot.gserviceaccount.com, Oct 13 2016
Labels: Sheriff-Chromium
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Cc: -estaab@chromium.org
Labels: -Sheriff-Chromium
Same comment as in #71. Not sure who to assign this issue to. Those of you who know the topic and history, please assign to suitable owner.
Blockedon: 649762
Owner: nedngu...@google.com
Status: Assigned
I am working on addressing the flakiness reporting mechanism in issue 649762 so that we report which tests are failing, not the whole suite.
Project Member Comment 78 by chromium...@appspot.gserviceaccount.com, Oct 14 2016
Detected 10 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 79 by chromium...@appspot.gserviceaccount.com, Oct 17 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 80 by chromium...@appspot.gserviceaccount.com, Oct 17 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 81 by chromium...@appspot.gserviceaccount.com, Oct 18 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Comment 82 by kbr@chromium.org, Oct 20 2016
Is any progress being made on the flakes on the various platforms? See for example this random failure from an unrelated CL:
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/164497

Ken, most of the flakes here don't happen very frequently, the one you just have is probably the more serious issue in issue 619189
Project Member Comment 84 by chromium...@appspot.gserviceaccount.com, Oct 21 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 85 by chromium...@appspot.gserviceaccount.com, Oct 22 2016
Detected 15 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 86 by chromium...@appspot.gserviceaccount.com, Oct 24 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 87 by chromium...@appspot.gserviceaccount.com, Oct 26 2016
Detected 10 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 88 by chromium...@appspot.gserviceaccount.com, Oct 27 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 89 by chromium...@appspot.gserviceaccount.com, Oct 28 2016
Detected 13 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 90 by chromium...@appspot.gserviceaccount.com, Oct 29 2016
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 91 by chromium...@appspot.gserviceaccount.com, Oct 30 2016
Detected 7 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 92 by chromium...@appspot.gserviceaccount.com, Oct 31 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 93 by chromium...@appspot.gserviceaccount.com, Nov 1 2016
Detected 8 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 94 by chromium...@appspot.gserviceaccount.com, Nov 3 2016
Detected 21 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Comment 95 by kbr@chromium.org, Nov 3 2016
Blockedon: 662021
Ok so I got a chance to look at this closer this afternoon ( I was waiting for https://codereview.chromium.org/2472123002 to land to see what it did) and it seems that cleared up the json errors, but we are still seeing some pretty consistent flakes.  

For windows most commonly it is smoke test system_health.memory_desktop.browse:news:nytimes

Example failing: https://chromium-swarm.appspot.com/user/task/324943564d061210

An snippet of the failing log: 
      expr, context_id=context_id, timeout=timeout)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 36, in inner
      return func(inspector_backend, *args, **kwargs)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 199, in ExecuteJavaScript
      self._runtime.Execute(expr, context_id, timeout)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_runtime.py", line 21, in Execute
      self.Evaluate(expr + '; 0;', context_id, timeout)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_runtime.py", line 45, in Evaluate
      res = self._inspector_websocket.SyncRequest(request, timeout)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 110, in SyncRequest
      res = self._Receive(timeout)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 166, in _Receive
      self._HandleNotification(result)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 179, in _HandleNotification
      self._domain_handlers[domain_name](result)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 365, in _HandleInspectorDomainNotification
      raise exception
  DevtoolsTargetCrashException: Devtools target crashed
  ********************************************************************************
  (e:\b\swarm_slave\w\irhz58th\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:418 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed.
  *******************************************************************************

The tab seems to be crashing but I can't tell if it is random failure or if something else is causing it. 


Here is an example of a browser log of one of these failures: 
 [8176:4916:1104/093737:ERROR:memory_mapped_file.cc(52)] Couldn't open e:\b\swarm_slave\w\irsz8uni\out\Release_x64\chrome_200_percent.pak
  [8176:4916:1104/093737:ERROR:data_pack.cc(79)] Failed to mmap datapack
  [8176:4916:1104/093737:WARNING:chrome_browser_main_win.cc(419)] Command line too long for RegisterApplicationRestart
  [7932:7784:1104/093737:ERROR:angle_platform_impl.cc(33)] ANGLE Display::initialize error 4: Could not create D3D11 device.
  [7932:7784:1104/093737:ERROR:gl_surface_egl.cc(630)] eglInitialize D3D11 failed with error EGL_NOT_INITIALIZED, trying next display type
  [5168:4348:1104/093737:ERROR:angle_platform_impl.cc(33)] ANGLE Display::initialize error 4: Could not create D3D11 device.
  [5168:4348:1104/093737:ERROR:gl_surface_egl.cc(630)] eglInitialize D3D11 failed with error EGL_NOT_INITIALIZED, trying next display type
  [7932:7784:1104/093738:ERROR:angle_platform_impl.cc(33)] ANGLE Display::initialize error 4: Renderer does not support PS 3.0.aborting!
  [7932:7784:1104/093738:ERROR:gl_surface_egl.cc(630)] eglInitialize D3D9 failed with error EGL_NOT_INITIALIZED
  [7932:7784:1104/093738:ERROR:gl_initializer_win.cc(272)] GLSurfaceEGL::InitializeOneOff failed.
  [7932:7784:1104/093738:INFO:media_foundation_video_encode_accelerator_win.cc(290)] Windows versions earlier than 8 are not supported.
  [7932:7784:1104/093738:ERROR:gpu_child_thread.cc(328)] Exiting GPU process due to errors during initialization
  [8176:4880:1104/093738:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process.
  [5168:4348:1104/093738:ERROR:angle_platform_impl.cc(33)] ANGLE Display::initialize error 4: Renderer does not support PS 3.0.aborting!
  [5168:4348:1104/093738:ERROR:gl_surface_egl.cc(630)] eglInitialize D3D9 failed with error EGL_NOT_INITIALIZED
  [5168:4348:1104/093738:ERROR:gl_initializer_win.cc(272)] GLSurfaceEGL::InitializeOneOff failed.
  [5168:4348:1104/093738:INFO:media_foundation_video_encode_accelerator_win.cc(290)] Windows versions earlier than 8 are not supported.
  [5168:4348:1104/093738:ERROR:gpu_child_thread.cc(328)] Exiting GPU process due to errors during initialization

I am not sure if these errors are typical or if one of them could be causing it.




Android failures on linux_android_rel_ng seem to be different tests and different reasons, the most recent related to adb failures: 

https://chromium-swarm.appspot.com/user/task/32495f84f5938310

    File "/b/swarm_slave/w/ir3qKHC9/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
      self._ret = self._func(*self._args, **self._kwargs)
    File "/b/swarm_slave/w/ir3qKHC9/third_party/catapult/devil/devil/utils/timeout_retry.py", line 152, in <lambda>
      child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
    File "/b/swarm_slave/w/ir3qKHC9/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
      return f(*args, **kwargs)
    File "/b/swarm_slave/w/ir3qKHC9/third_party/catapult/devil/devil/android/device_utils.py", line 657, in Install
      reinstall=reinstall, permissions=permissions)
    File "/b/swarm_slave/w/ir3qKHC9/third_party/catapult/devil/devil/android/device_utils.py", line 743, in _InstallInternal
      base_apk.path, reinstall=reinstall, allow_downgrade=allow_downgrade)
    File "/b/swarm_slave/w/ir3qKHC9/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 697, in Install
      cmd, output, device_serial=self._device_serial)
  AdbCommandFailedError: (device: 0609d298003bc315) adb install /b/swarm_slave/w/ir3qKHC9/out/Release/apks/ChromePublic.apk: failed and output:
  - 4752 KB/s (58587431 bytes in 12.038s)
  - 	pkg: /data/local/tmp/ChromePublic.apk
  - Failure [INSTALL_FAILED_UID_CHANGED]
  

I didn't get very far in my investigations but wondered if anyone had any thoughts on the browser log errors.  Otherwise I will keep digging.

Project Member Comment 97 by chromium...@appspot.gserviceaccount.com, Nov 4 2016
Detected 50 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Comment 98 by kbr@chromium.org, Nov 4 2016
jbudorick@ would be the best person to help diagnose the Android failures.

It would be good to prove or disprove whether the GPU process launch failures are the cause of these test failures. Are they seen on any passing test runs?

I'll look into the INSTALL_FAILED_UID_CHANGED errors on Monday.
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
So I am not sure how to look at the browser log of a passing test to see if they are seen. 

It is interesting to note that we often seen nytimes failing with the same stack trace but then succeeding on a retry so that is why it is still green.  

For instance the most recent run of windowns_rel_ng: 

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/326793

Here is a link to the telemetry_perf_unittests swarming task: 
https://chromium-swarm.appspot.com/user/task/325825ba3c80ca10

You see the stack trace of the failing nytimes test with the same browser output, but then you see it succeeed on retry so the builder is green. 

Where as in another recent build: 
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/326777

You see from the swarming task that it succeeded the first time: 
https://chromium-swarm.appspot.com/user/task/32580b0e52b2c810

and you never see that stack trace.

Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Cc: bpastene@chromium.org
Looking a bit at the android flakes, a lot of the recent failures are all localized to a single swarming bot: https://chromium-swarm.appspot.com/restricted/bot/build3-b4. It's been consistently failing all of its telemetry_perf_unittests runs for the same reason: "INSTALL_FAILED_UID_CHANGED"

That usually means a previous version of an app, in this case ChromePublic.apk, wasn't properly removed from the device before reinstalling. I'll reflash all phones on that host, that should clear it up.

Though, we should look at how we're cleaning these apps from the phones and see if we can't avoid the errors without having to trigger a reflash.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 105 by chromium...@appspot.gserviceaccount.com, Nov 12 2016
Detected 13 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 106 by chromium...@appspot.gserviceaccount.com, Nov 13 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 107 by chromium...@appspot.gserviceaccount.com, Nov 14 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
All recent flakes are due to timeouts. Most likely caused by https://codereview.chromium.org/2493203002/ which increased the number of tests ran. I'll bump the timeout on the suite.
Project Member Comment 109 by bugdroid1@chromium.org, Nov 14 2016
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1ebc90106e99c94b774f9de0f10197185500de0e

commit 1ebc90106e99c94b774f9de0f10197185500de0e
Author: bpastene <bpastene@chromium.org>
Date: Mon Nov 14 23:27:14 2016

Bump timeout of telemetry_perf_unittests on android_n5x bot to 20 min.

BUG=637904

Review-Url: https://codereview.chromium.org/2498903004
Cr-Commit-Position: refs/heads/master@{#431966}

[modify] https://crrev.com/1ebc90106e99c94b774f9de0f10197185500de0e/testing/buildbot/chromium.android.json

Project Member Comment 110 by chromium...@appspot.gserviceaccount.com, Nov 19 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 111 by chromium...@appspot.gserviceaccount.com, Nov 24 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 112 by chromium...@appspot.gserviceaccount.com, Nov 26 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 113 by chromium...@appspot.gserviceaccount.com, Nov 29 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 114 by chromium...@appspot.gserviceaccount.com, Nov 30 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 14 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 18 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Almost all the new flakes are caused by issue 671018
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 121 by chromium...@appspot.gserviceaccount.com, Dec 12 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 122 by chromium...@appspot.gserviceaccount.com, Dec 13 2016
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 123 by chromium...@appspot.gserviceaccount.com, Dec 14 2016
Detected 20 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 124 by chromium...@appspot.gserviceaccount.com, Dec 15 2016
Detected 9 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 125 by chromium...@appspot.gserviceaccount.com, Dec 16 2016
Detected 9 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 126 by chromium...@appspot.gserviceaccount.com, Dec 17 2016
Detected 29 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 127 by chromium...@appspot.gserviceaccount.com, Dec 19 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 128 by chromium...@appspot.gserviceaccount.com, Dec 21 2016
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 129 by chromium...@appspot.gserviceaccount.com, Dec 22 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 130 by chromium...@appspot.gserviceaccount.com, Dec 23 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 131 by chromium...@appspot.gserviceaccount.com, Dec 26 2016
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 132 by chromium...@appspot.gserviceaccount.com, Dec 29 2016
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Cc: -nednguyen@chromium.org
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 8 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 138 by chromium...@appspot.gserviceaccount.com, Jan 10 2017
Detected 14 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 139 by chromium...@appspot.gserviceaccount.com, Jan 11 2017
Detected 16 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Almost all the new flakiness are due to telemetry_perf_unittest timed out on android_n5x_swarming after adb keeps failing. Filed issue 680141 on this.
Blockedon: 680141
Those aren't adb failures. Those are timeouts. The timing out is the problem.
Project Member Comment 143 by chromium...@appspot.gserviceaccount.com, Jan 12 2017
Detected 13 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 144 by chromium...@appspot.gserviceaccount.com, Jan 13 2017
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 145 by chromium...@appspot.gserviceaccount.com, Jan 16 2017
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 146 by chromium...@appspot.gserviceaccount.com, Jan 17 2017
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 147 by chromium...@appspot.gserviceaccount.com, Jan 18 2017
Detected 35 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 148 by chromium...@appspot.gserviceaccount.com, Jan 19 2017
Detected 48 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 149 by chromium...@appspot.gserviceaccount.com, Jan 20 2017
Detected 16 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 150 by chromium...@appspot.gserviceaccount.com, Jan 21 2017
Detected 14 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 151 by chromium...@appspot.gserviceaccount.com, Jan 22 2017
Detected 8 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 152 by chromium...@appspot.gserviceaccount.com, Jan 23 2017
Detected 60 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Comment 153 by kbr@chromium.org, Jan 23 2017
Blockedon: 669242
60 flakes in a day is a huge number. It looks to me like the most recent source of crashes might be a V8 bug. See Issue 669242.

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/376089 has this failure log:

Retrying failed tests (attempt #3 of 3)...

[1/1] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:news:nytimes failed unexpectedly 133.5537s:
[ RUN ] browse:news:nytimes
Traceback (most recent call last):
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 94, in _RunStoryAndProcessErrorIfNeeded
state.RunStory(results)
File "/b/s/w/irhu4lH5/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 299, in RunStory
self._current_page.Run(self)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/page/__init__.py", line 111, in Run
self.RunPageInteractions(action_runner)
File "/b/s/w/irhu4lH5/tools/perf/page_sets/system_health/system_health_story.py", line 112, in RunPageInteractions
self._DidLoadDocument(action_runner)
File "/b/s/w/irhu4lH5/tools/perf/page_sets/system_health/browsing_stories.py", line 80, in _DidLoadDocument
self._NavigateBack(action_runner)
File "/b/s/w/irhu4lH5/tools/perf/page_sets/system_health/browsing_stories.py", line 49, in _NavigateBack
action_runner.NavigateBack()
File "/b/s/w/irhu4lH5/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 183, in NavigateBack
self.ExecuteJavaScript2('window.history.back()')
File "/b/s/w/irhu4lH5/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 216, in ExecuteJavaScript2
return self._tab.ExecuteJavaScript2(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 152, in ExecuteJavaScript2
return self._inspector_backend.ExecuteJavaScript2(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 37, in inner
return func(inspector_backend, *args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 216, in ExecuteJavaScript2
self._runtime.Execute(statement, context_id, timeout)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 21, in Execute
self.Evaluate(expr + '; 0;', context_id, timeout)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 45, in Evaluate
res = self._inspector_websocket.SyncRequest(request, timeout)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 110, in SyncRequest
res = self._Receive(timeout)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 166, in _Receive
self._HandleNotification(result)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 179, in _HandleNotification
self._domain_handlers[domain_name](result)
File "/b/s/w/irhu4lH5/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 456, in _HandleInspectorDomainNotification
raise exception
DevtoolsTargetCrashException: Devtools target crashed
********************************************************************************
(/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:509 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed.
********************************************************************************
(/b/s/w/irhu4lH5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:510 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:36850/devtools/page/212b6012-aeb5-444f-8364-5c35ead4f0c8
Found Minidump: True
Stack Trace:
********************************************************************************
Operating system: Linux
0.0.0 Linux 3.13.0-46-generic #79~precise1-Ubuntu SMP Tue Mar 10 20:25:33 UTC 2015 x86_64
CPU: amd64
family 6 model 62 stepping 4
8 CPUs

GPU: UNKNOWN

Crash reason: SIGSEGV
Crash address: 0x0
Process uptime: not available

Thread 0 (crashed)
0 chrome!StringCharacterPosition [objects-inl.h : 3987 + 0x0]
rax = 0x6a00001e9b7c2023 rdx = 0x00007f9c7b91e2c0
rcx = 0x000007d6367c6088 rbx = 0x0000000000000000
rsi = 0x0000000000000000 rdi = 0x0000161a1b068191
rbp = 0x00007fffd6666b58 rsp = 0x00007fffd6666a60
r8 = 0x00000000000003ff r9 = 0x000007d6367a4000
r10 = 0x0000000000000008 r11 = 0x00007f9c75306f9f
r12 = 0x0000000000000000 r13 = 0x00007fffd6666b18
r14 = 0x00007fffd6666b50 r15 = 0x00007fffd6666b68
rip = 0x00007f9c7d0cd57a
Found by: given as instruction pointer in context
1 chrome!CheckStackGuardState [regexp-macro-assembler.cc : 204 + 0x9]
rbx = 0x0000000000000015 rbp = 0x00007fffd6666b58
rsp = 0x00007fffd6666a70 r12 = 0x0000000000000000
r13 = 0x00007fffd6666b18 r14 = 0x00007fffd6666b50
r15 = 0x00007fffd6666b68 rip = 0x00007f9c7d0cd75e
Found by: call frame info
2 chrome!CheckStackGuardState [regexp-macro-assembler-x64.cc : 1222 + 0xb]
rbx = 0x00000abc3f0778c0 rbp = 0x00007fffd6666b70
rsp = 0x00007fffd6666b00 r12 = 0x0000161a1b0681bd
r13 = 0x0000161a1b0681a8 r14 = 0x000019797d778839
r15 = 0x000007d636754bf4 rip = 0x00007f9c7d30b90e
Found by: call frame info
3 0xabc3f077cc7
rbx = 0x00000abc3f0778c0 rbp = 0x00007fffd6666b70
rsp = 0x00007fffd6666b20 r12 = 0x0000161a1b0681bd
r13 = 0x0000161a1b0681a8 r14 = 0x000019797d778839
r15 = 0x000007d636754bf4 rip = 0x00000abc3f077cc7
Found by: call frame info
4 chrome!Execute [regexp-macro-assembler.cc : 283 + 0x1d]
rsp = 0x00007fffd6666b80 rip = 0x00007f9c7d0cdabf
Found by: stack scanning
5 chrome!Match [regexp-macro-assembler.cc : 257 + 0x11]
rbx = 0x0000000000000004 rbp = 0x0000000000000000
rsp = 0x00007fffd6666be0 r12 = 0x0000000000000015
r13 = 0x000007d63674f000 r14 = 0x0000161a1b068191
r15 = 0x0000000000000000 rip = 0x00007f9c7d0cd96c
Found by: call frame info
6 chrome!IrregexpExecRaw [jsregexp.cc : 500 + 0x17]
rbx = 0x0000000000000000 rbp = 0x00007fffd6666d78
rsp = 0x00007fffd6666c40 r12 = 0x000007d63674f000
r13 = 0x00007fffd6666d80 r14 = 0x000007d6367c6070
r15 = 0x00000abc3f077841 rip = 0x00007f9c7d0b43c4
Found by: call frame info
7 chrome!IrregexpExec [jsregexp.cc : 592 + 0x15]
rbx = 0x000007d63674f000 rbp = 0x000007d636754bf4
rsp = 0x00007fffd6666c90 r12 = 0x00007fffd6666d78
r13 = 0x00007fffd6666d80 r14 = 0x00007fffd6666cf8
r15 = 0x0000000000000004 rip = 0x00007f9c7d0b2925
Found by: call frame info
8 chrome!Exec [jsregexp.cc : 198 + 0x8]
rbx = 0x00007fffd6666cf8 rbp = 0x000007d636756000
rsp = 0x00007fffd6666ce0 r12 = 0x000007d63676c000
r13 = 0x000007d63674f0c8 r14 = 0x000007d63674f000
r15 = 0x00007fffd6666d80 rip = 0x00007f9c7d0b2356
Found by: call frame info
9 chrome!__RT_impl_Runtime_RegExpExec [runtime-regexp.cc : 822 + 0xb]
rbx = 0x00007fffd6666d78 rbp = 0x000007d636756000
rsp = 0x00007fffd6666cf0 r12 = 0x000007d63676c000
r13 = 0x000007d63674f0c8 r14 = 0x000007d63674f000
r15 = 0x00007fffd6666d80 rip = 0x00007f9c7d170e15
Found by: call frame info
10 0xabc3df0426e
rbx = 0x00007f9c7d170960 rbp = 0x00007fffd6666d58
rsp = 0x00007fffd6666d40 r12 = 0x000007d63676c000
r13 = 0x000007d63674f0c8 r14 = 0x0000000000000004
r15 = 0x00007fffd6666d80 rip = 0x00000abc3df0426e
Found by: call frame info
11 0xabc3df9519c
rbp = 0x00007fffd6666e20 rsp = 0x00007fffd6666d68
rip = 0x00000abc3df9519c
Found by: previous frame's frame pointer
12 0xabc3e7964f6
rbp = 0x00007fffd6666e60 rsp = 0x00007fffd6666e30
rip = 0x00000abc3e7964f6
Found by: previous frame's frame pointer
13 0xabc3f0774e5
rbp = 0x00007fffd6666ea8 rsp = 0x00007fffd6666e70
rip = 0x00000abc3f0774e5
Found by: previous frame's frame pointer
14 0xabc3e62eb5f
rbp = 0x00007fffd6666f18 rsp = 0x00007fffd6666eb8
rip = 0x00000abc3e62eb5f
Found by: previous frame's frame pointer
15 0xabc3f7e4479
rbp = 0x00007fffd6666f58 rsp = 0x00007fffd6666f28
rip = 0x00000abc3f7e4479
Found by: previous frame's frame pointer
16 0xabc3df06555
rbp = 0x00007fffd6666f88 rsp = 0x00007fffd6666f68
rip = 0x00000abc3df06555
Found by: previous frame's frame pointer
17 0xabc3e62e1f2
rbp = 0x00007fffd6666fd0 rsp = 0x00007fffd6666f98
rip = 0x00000abc3e62e1f2
Found by: previous frame's frame pointer
18 0xabc3f785687
rbp = 0x00007fffd6667028 rsp = 0x00007fffd6666fe0
rip = 0x00000abc3f785687
Found by: previous frame's frame pointer
19 0xabc3f7866de
rbp = 0x00007fffd6667088 rsp = 0x00007fffd6667038
rip = 0x00000abc3f7866de
Found by: previous frame's frame pointer
20 0xabc3f784564
rbp = 0x00007fffd66670c0 rsp = 0x00007fffd6667098
rip = 0x00000abc3f784564
Found by: previous frame's frame pointer
21 0xabc3df06555
rbp = 0x00007fffd66670f0 rsp = 0x00007fffd66670d0
rip = 0x00000abc3df06555
Found by: previous frame's frame pointer
22 0xabc3f5aca28
rbp = 0x00007fffd6667128 rsp = 0x00007fffd6667100
rip = 0x00000abc3f5aca28
Found by: previous frame's frame pointer
23 0xabc3dfcbec3
rbp = 0x00007fffd6667158 rsp = 0x00007fffd6667138
rip = 0x00000abc3dfcbec3
Found by: previous frame's frame pointer
24 0xabc3df34981
rbp = 0x00007fffd66671c0 rsp = 0x00007fffd6667168
rip = 0x00000abc3df34981
Found by: previous frame's frame pointer
25 chrome!Invoke [execution.cc : 144 + 0x15]
rsp = 0x00007fffd66671d0 rip = 0x00007f9c7ce642fb
Found by: stack scanning
26 chrome!CallInternal [execution.cc : 180 + 0x1e]
rbx = 0x000007d63674f068 rbp = 0x000007d63674f000
rsp = 0x00007fffd6667280 r12 = 0x000007d6367c6008
r13 = 0x00007fffd6667308 r14 = 0x0000000000000000
r15 = 0x0000000000000000 rip = 0x00007f9c7ce63c91
Found by: call frame info
27 chrome!v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 0x14
rbx = 0x00007fffd6667308 rbp = 0x0000000000000000
rsp = 0x00007fffd66672e0 r12 = 0x000007d6367c6028
r13 = 0x000007d6367c6008 r14 = 0x0000000000000000
r15 = 0x0000000000000004 rip = 0x00007f9c7ce63bb4
Found by: call frame info
28 chrome!Call [api.cc : 5101 + 0x14]
rbx = 0x000007d63674f000 rbp = 0x0000000000000000
rsp = 0x00007fffd6667300 r12 = 0x000007d6367c6028
r13 = 0x000007d6367c6008 r14 = 0x0000000000000000
r15 = 0x0000000000000004 rip = 0x00007f9c7ca9c6ed
Found by: call frame info
29 chrome!callFunction [V8ScriptRunner.cpp : 650 + 0x16]
rbx = 0x000015baa136a9c0 rbp = 0x000007d6367c6008
rsp = 0x00007fffd66673e0 r12 = 0x0000000000000000
r13 = 0x000007d6367c6028 r14 = 0x000007d63674f000
r15 = 0x0000000000000000 rip = 0x00007f9c80695b09
Found by: call frame info
30 chrome!execute [ScheduledAction.cpp : 135 + 0xe]
rbx = 0x000015baa136a9c0 rbp = 0x000015baa136a9c0
rsp = 0x00007fffd6667610 r12 = 0x00003e2e5a1f8248
r13 = 0x00007fffd66681b0 r14 = 0x00002274a7b61f50
r15 = 0x000007d6367c6008 rip = 0x00007f9c82b05645

Project Member Comment 154 by chromium...@appspot.gserviceaccount.com, Jan 25 2017
Detected 23 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 155 by chromium...@appspot.gserviceaccount.com, Jan 27 2017
Detected 31 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 156 by chromium...@appspot.gserviceaccount.com, Jan 28 2017
Detected 23 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 157 by chromium...@appspot.gserviceaccount.com, Jan 29 2017
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 158 by chromium...@appspot.gserviceaccount.com, Jan 30 2017
Detected 17 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Project Member Comment 159 by chromium...@appspot.gserviceaccount.com, Jan 31 2017
Detected 33 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 36 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 58 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
According to the flakiness dashboard, most of the new flakes are due to benchmarks/system_health_smoke_test/SystemHealthBenchmarkSmokeTest/system_health/memory_mobile/browse:news:washingtonpost. 

Track the disabling & fixing it in issue 688190
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 12 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 10 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Cc: -stip@chromium.org
Detected 12 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 20 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 18 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 11 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 14 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 9 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 10 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 11 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 12 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 8 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Blockedon: 713048
#194 is tracked in issue 713048
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 6 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 15 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Things were looking really good there for a while. These recent failures all came from Android and look related to devices going offline. Are these being looked into under some other bug? Here's the backtrace which seems to be taking down the harness:

Traceback (most recent call last):
  File "/b/swarm_slave/w/ir/third_party/catapult/third_party/typ/typ/pool.py", line 159, in _loop
    context_after_pre = pre_fn(host, worker_num, context)
  File "/b/swarm_slave/w/ir/third_party/catapult/third_party/typ/typ/runner.py", line 775, in _setup_process
    child.context_after_setup = child.setup_fn(child, child.context)
  File "/b/swarm_slave/w/ir/third_party/catapult/telemetry/telemetry/testing/run_tests.py", line 290, in _SetUpProcess
    android_devices[child.worker_num-1].guid)
IndexError: list index out of range
 <traceback object at 0x7f49103ede60>

of the 15 failures reported in #204, there were:
 - 2 device issues
 - 2 oauth issues
 - 11 timeouts

There certainly appears to be something worth investigating here -- perhaps more than one thing -- but it's at least not only device issues.
also, we appear to be doing a lot of unnecessary looking for battors on those trybots...
Also, what happened to the work that had been done to have the infrastructure report which actual tests are being flaky, rather than just the top level "telelemtry_perf_unittests". I thought a bunch of patches landed towards that a couple months back - but seems it's still reporting under this top level bug (which now has >200 comments).
#208: when the whole suite failed due to timed out, it will still report under this top level bug.

We have individual tests flakiness report in https://bugs.chromium.org/p/chromium/issues/detail?id=725386 for example
Ah, great! (Wonder if we could change the report title of these cases to say something like "failed due to time out" or similar if that's the event where the top level bug will happen.)
Filed issue 725516 about the timed out issue. 
Summary: "telemetry_perf_unittests (with patch)" is flaky due to whole suite timed out (was: "telemetry_perf_unittests (with patch)" is flaky)
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 7 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 10 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 11 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 17 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 5 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 4 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Detected 7 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
#228 is likely due to https://bugs.chromium.org/p/chromium/issues/detail?id=742463
Detected 3 new flakes for test/step "telemetry_perf_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.
Sign in to add a comment