All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=658872

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZykQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImNDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL21lZGlhLmFuZHJvaWQudG91Z2hfdmlkZW9fY2FzZXMvYXBwbGljYXRpb25fZW5lcmd5X2NvbnN1bXB0aW9uX213aC9yZWYMCxINU3RvcHBhZ2VBbGVydBj28BkM,agxzfmNocm9tZXBlcmZysgELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoMBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2Qub2dnX3R5cGVfYXVkaW8MCxINU3RvcHBhZ2VBbGVydBj28BkM,agxzfmNocm9tZXBlcmZytgELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IocBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2Qub2dnX3R5cGVfYXVkaW9fcmVmDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZysgELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoMBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2Qud2F2X3R5cGVfYXVkaW8MCxINU3RvcHBhZ2VBbGVydBj28BkM,agxzfmNocm9tZXBlcmZytgELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IocBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2Qud2F2X3R5cGVfYXVkaW9fcmVmDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZyqgELEhNTdG9wcGFnZUFsZXJ0UGFyZW50InxDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL21lZGlhLmFuZHJvaWQudG91Z2hfdmlkZW9fY2FzZXMvYXBwbGljYXRpb25fZW5lcmd5X2NvbnN1bXB0aW9uX213aC92aWRlby5odG1sP3NyY19jcm93ZDEwODAub2d2DAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZytwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IogBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwLm9ndl9jYW52YXNfdHJ1ZQwLEg1TdG9wcGFnZUFsZXJ0GPbwGQw,agxzfmNocm9tZXBlcmZyuwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IowBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwLm9ndl9jYW52YXNfdHJ1ZV9yZWYMCxINU3RvcHBhZ2VBbGVydBj28BkM,agxzfmNocm9tZXBlcmZyrwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoABQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwLm9ndl9yZWYMCxINU3RvcHBhZ2VBbGVydBj28BkM,agxzfmNocm9tZXBlcmZysAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoEBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwX3ZwOS53ZWJtDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZyvAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Io0BQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwX3ZwOS53ZWJtX2NhbnZhc190cnVlDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZywAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IpEBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwX3ZwOS53ZWJtX2NhbnZhc190cnVlX3JlZgwLEg1TdG9wcGFnZUFsZXJ0GPbwGQw,agxzfmNocm9tZXBlcmZytAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoUBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2QxMDgwX3ZwOS53ZWJtX3JlZgwLEg1TdG9wcGFnZUFsZXJ0GPbwGQw,agxzfmNocm9tZXBlcmZyrwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoABQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2Q3MjBfdnA5LndlYm0MCxINU3RvcHBhZ2VBbGVydBj28BkM,agxzfmNocm9tZXBlcmZyswELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoQBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfY3Jvd2Q3MjBfdnA5LndlYm1fcmVmDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZywAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IpEBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfc21wdGVfMzg0MHgyMTYwXzYwZnBzX3ZwOS53ZWJtX3JlZgwLEg1TdG9wcGFnZUFsZXJ0GPbwGQw,agxzfmNocm9tZXBlcmZyswELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoQBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfdHVsaXAyLm00YV90eXBlX2F1ZGlvDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZytwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IogBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfdHVsaXAyLm00YV90eXBlX2F1ZGlvX3JlZgwLEg1TdG9wcGFnZUFsZXJ0GPbwGQw,agxzfmNocm9tZXBlcmZyswELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IoQBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfdHVsaXAyLm1wM190eXBlX2F1ZGlvDAsSDVN0b3BwYWdlQWxlcnQY9vAZDA,agxzfmNocm9tZXBlcmZytwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IogBQ2hyb21pdW1QZXJmL2FuZHJvaWQtbmV4dXM1WC9tZWRpYS5hbmRyb2lkLnRvdWdoX3ZpZGVvX2Nhc2VzL2FwcGxpY2F0aW9uX2VuZXJneV9jb25zdW1wdGlvbl9td2gvdmlkZW8uaHRtbD9zcmNfdHVsaXAyLm1wM190eXBlX2F1ZGlvX3JlZgwLEg1TdG9wcGFnZUFsZXJ0GPbwGQw


Bot(s) for this bug's original alert(s):

android-nexus5X

I suspect the catapult roll, because a similar earlier regression only contained a catapult roll. Both catapult rolls contained this line:

Update android stable chrome reference builds.

Aiolos, is this something your change (or some other catapult change) could have caused? If not, feel free to assign back to me.

Updating the reference build should only affect benchmarks run on the reference build, which this isn't. Although the reference version of this metric (media.android.tough_video_cases.reference) is also failing much of the time, it doesn't look related to the update. Looking at the logs on the waterfall, this appears to be the failure:

Traceback (most recent call last):
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 311, in RunStory
    self._current_page.Run(self)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 109, in Run
    self.RunPageInteractions(action_runner)
  File "/b/rr/tmpCj0JQM/w/src/tools/perf/page_sets/tough_video_cases.py", line 504, in RunPageInteractions
    action_timeout_in_seconds=120)
  File "/b/rr/tmpCj0JQM/w/src/tools/perf/page_sets/tough_video_cases.py", line 40, in SeekBeforeAndAfterPlayhead
    label='seek_cold')
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 718, in SeekMedia
    log_time=log_time, label=label))
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 53, in _RunAction
    action.RunAction(self._tab)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/internal/actions/seek.py", line 48, in RunAction
    self._timeout_in_seconds)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/internal/actions/media_action.py", line 34, in WaitForEvent
    timeout=timeout_in_seconds)
  File "/b/rr/tmpCj0JQM/w/src/third_party/catapult/telemetry/telemetry/core/util.py", line 94, in WaitFor
    (timeout, GetConditionString()))
TimeoutException: Timed out while waiting 120s for util.WaitFor(lambda:
                     self.HasEventCompletedOrError(tab, selector, event_name),
                 timeout=timeout_in_seconds).


from https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5%20Perf%20%282%29/builds/4167/steps/media.android.tough_video_cases/logs/stdio


You can start a bisect on that metric over the revisions it started failing on to try to pinpoint the cause.