Detected 5 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app.

Detected 9 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app.

https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/666766

    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/page_run_end_to_end_unittest.py", line 271, in testTrafficSettings
      self.assertGreater(latencies_by_page_in_ms['slow'],
  KeyError: 'slow'

Detected 6 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app.

[mac triage] assigning to a catapult owner

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/6eaec901b9866f7831296a6047a1ce2f8db020eb

commit 6eaec901b9866f7831296a6047a1ce2f8db020eb
Author: Nghia Nguyen <nednguyen@google.com>
Date: Mon Mar 12 04:55:05 2018

Add more logging to debug flaky failure of ActualPageRunEndToEndTests.testTrafficSettings

Bug:  chromium:817906 
Change-Id: I3f4ec1557efc4ac7bd1d09007583759a62921413
TBR=perezju@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/958803
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/6eaec901b9866f7831296a6047a1ce2f8db020eb/telemetry/telemetry/page/page_run_end_to_end_unittest.py

Detected 3 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Hi Ned, seems like this is still happening?

HI rogerta, I am adding more log to debug this. The flaky rate is fairly low: ~1 per day, so hopefully it's ok?

The new error message is:
  Traceback (most recent call last):
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/page_run_end_to_end_unittest.py", line 269, in testTrafficSettings
      self.assertFalse(results.had_failures)
  AssertionError: True is not false


Ethan: it looks like I will need a way to query the error stack of the page_test_results's failure to understand why sometimes the test run in https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/page/page_run_end_to_end_unittest.py?rcl=f73167a6894a911709d1d6c88f1bc97cb133de33&l=269 fails. 

Do we still capture the string error stack of each failure in page_test_results?

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/ee4c7175f67bca581b731c129a14423ed0e06b8e

commit ee4c7175f67bca581b731c129a14423ed0e06b8e
Author: Nghia Nguyen <nednguyen@google.com>
Date: Thu Mar 15 18:42:07 2018

Store the failure message in story_run object for debugging failure reason

This is needed for debugging failure in ActualPageRunEndToEndTests.testTrafficSettings
(see https://bugs.chromium.org/p/chromium/issues/detail?id=817906#c17)

Bug:  chromium:817906 
Change-Id: I6b08cc5f93950ef47ea34af9fa01d33642b5d44a
Reviewed-on: https://chromium-review.googlesource.com/964588
Reviewed-by: Ethan Kuefner <eakuefner@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/ee4c7175f67bca581b731c129a14423ed0e06b8e/telemetry/telemetry/internal/results/page_test_results.py
[modify] https://crrev.com/ee4c7175f67bca581b731c129a14423ed0e06b8e/telemetry/telemetry/internal/results/story_run_unittest.py
[modify] https://crrev.com/ee4c7175f67bca581b731c129a14423ed0e06b8e/telemetry/telemetry/internal/results/story_run.py
[modify] https://crrev.com/ee4c7175f67bca581b731c129a14423ed0e06b8e/telemetry/telemetry/page/page_run_end_to_end_unittest.py

Detected 3 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 5 new flakes for test/step "telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testTrafficSettings". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyZQsSBUZsYWtlIlp0ZWxlbWV0cnkucGFnZS5wYWdlX3J1bl9lbmRfdG9fZW5kX3VuaXR0ZXN0LkFjdHVhbFBhZ2VSdW5FbmRUb0VuZFRlc3RzLnRlc3RUcmFmZmljU2V0dGluZ3MM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Now we have the error message!
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 289, in WaitForJavaScriptCondition
      return py_utils.WaitFor(IsJavaScriptExpressionTrue, timeout)
    File "/b/s/w/ir/third_party/catapult/common/py_utils/py_utils/__init__.py", line 136, in WaitFor
      res = condition()
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 286, in IsJavaScriptExpressionTrue
      return self._EvaluateJavaScript(condition, context_id, timeout)
    File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 38, in Inner
      return func(inspector_backend, *args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 524, in _EvaluateJavaScript
      self._runtime.Crash(context_id, timeout)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 83, in Crash
      raise exceptions.EvaluateException(res['error']['message'])
  EvaluateException: Page has pending navigations, not killing
  

Looks like the new Mac bots are too slow, so the test is timing out. I will change this from 2G to 4G to speed it up.

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/fa0f0f2dd7e9ee09a5bed38bba060750d92c2ef2

commit fa0f0f2dd7e9ee09a5bed38bba060750d92c2ef2
Author: Nghia Nguyen <nednguyen@google.com>
Date: Fri Mar 16 23:04:40 2018

change ActualPageRunEndToEndTests.testTrafficSettings to use good3g for slow case

Bug:  chromium:817906 
Change-Id: I0df31acd6357f61330f71c3f3611bd773085b5c0
TBR=perezju@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/967357
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/fa0f0f2dd7e9ee09a5bed38bba060750d92c2ef2/telemetry/telemetry/page/page_run_end_to_end_unittest.py

No longer happening, so this is probably fixed!