Based on flakiness dashboard data, it looks like this should be an easy bisect because the story went from passing 100% of the time to failing 100% of the time. Going to disable now and kick off a bisect.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/118bae79240000

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e75efc7e1b58c07a3ac58c213041a1098ef40b50

commit e75efc7e1b58c07a3ac58c213041a1098ef40b50
Author: Charlie Andrews <charliea@chromium.org>
Date: Mon Jun 11 15:46:50 2018

Disable blotspot_pinch on Android

It's failing on all platforms right now.

NOTRY=true
TBR=eyaich@chromium.org
CC=bokan@chromium.org,ericrk@chromium.org

Bug:  851499 
Change-Id: I5b4c3f6cff9a747dff1c28e0e21e117535857e93
Reviewed-on: https://chromium-review.googlesource.com/1094993
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566014}
[modify] https://crrev.com/e75efc7e1b58c07a3ac58c213041a1098ef40b50/tools/perf/expectations.config

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/118bae79240000

Merge tough_pinch_zoom_cases into rendering page sets by amyqiu@google.com
https://chromium.googlesource.com/chromium/src/+/4671c00c498d74d9e4a0a9364b547bc85dda3704
0 → 1 (+1)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Bah, because I kicked off the bisect on rendering.mobile, it just found where the story was added to rendering.mobile. Going to kick off the same bisect on smoothness.tough_pinch_zoom_cases, where this story started failing between 564870 and 565003

Hmmm... it looks like that actually wasn't the reason. It looks like something started failing due to Amy's refactoring: this doesn't look like it's due to a Chrome-side change.

The stack trace for the failure is:

Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 126, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 310, in RunStory
    self._current_page.Run(self)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 95, in Run
    shared_state.page_test.RunNavigateSteps(self, current_tab)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 152, in RunNavigateSteps
    page.RunNavigateSteps(action_runner)
  File "/b/swarming/w/ir/tools/perf/page_sets/rendering/tough_pinch_zoom_cases.py", line 131, in RunNavigateSteps
    action_runner.WaitForElement(text='accessibility')
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 289, in WaitForElement
    timeout_in_seconds=timeout_in_seconds))
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 62, in _RunAction
    action.RunAction(self._tab)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/wait.py", line 22, in RunAction
    wait=True, timeout_in_seconds=self.timeout_in_seconds)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/page_action.py", line 127, in EvaluateCallbackWithElement
    tab.WaitForJavaScriptCondition(code, timeout=timeout_in_seconds)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 239, in WaitForJavaScriptCondition
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 302, in WaitForJavaScriptCondition
    e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 60s for IsJavaScriptExpressionTrue.

A link to ehf ailing log is: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf%2F1827%2F%2B%2Frecipes%2Fsteps%2Fsmoothness.gpu_rasterization.tough_pinch_zoom_cases_on_Android_device_Nexus_5%2F0%2Fstdout

Amy, could you look into what might have caused this?

Expanding the scope of this bug to account for the fact that this is also failing on rendering.mobile.

Yeah, I'll take a look at it now.

Note that blogspot_pinch_desktop_gpu_raster is also failing in rendering.mobile.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0ddf2d6eb847ad31d9f972fdb30322f138845fbb

commit 0ddf2d6eb847ad31d9f972fdb30322f138845fbb
Author: Charlie Andrews <charliea@chromium.org>
Date: Mon Jun 11 17:56:44 2018

Disabling failing blogspot_pinch cases in rendering.mobile

NOTRY=true
TBR=eyaich@chromium.org

Bug:  851499 
Change-Id: I1ef846085e0b6eb65e03fcd07367f896c1363889
Reviewed-on: https://chromium-review.googlesource.com/1095689
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566050}
[modify] https://crrev.com/0ddf2d6eb847ad31d9f972fdb30322f138845fbb/tools/perf/expectations.config

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/35cae5d909ae76ff3d3b4e36790a650712df06e8

commit 35cae5d909ae76ff3d3b4e36790a650712df06e8
Author: Amy Qiu <amyqiu@google.com>
Date: Tue Jun 12 13:53:07 2018

Fix tough_pinch_zoom_cases to always use desktop shared state

When MobileRenderingSharedState is used, the blogspot_pinch test fails
because pages in this page set are meant for mobile desktop sites.
To prevent this, make the tough_pinch_zoom page set desktop-only and set
the target_scale_factor based on platform instead of shared_page_state.

Bug:  851499 
Change-Id: I9001784c37612a7da689fa8af3bcf3f0121dafec
Reviewed-on: https://chromium-review.googlesource.com/1096154
Commit-Queue: Amy Qiu <amyqiu@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#566407}
[modify] https://crrev.com/35cae5d909ae76ff3d3b4e36790a650712df06e8/tools/perf/expectations.config
[modify] https://crrev.com/35cae5d909ae76ff3d3b4e36790a650712df06e8/tools/perf/page_sets/data/rendering_mobile.json
[modify] https://crrev.com/35cae5d909ae76ff3d3b4e36790a650712df06e8/tools/perf/page_sets/rendering/rendering_stories.py
[modify] https://crrev.com/35cae5d909ae76ff3d3b4e36790a650712df06e8/tools/perf/page_sets/rendering/tough_pinch_zoom_cases.py

This has been fixed, the blogspot_pinch story is now re-enabled in the smoothness/rendering benchmarks

Amy: we used this bug for the TODO in https://chromium.googlesource.com/chromium/src/+/35cae5d909ae76ff3d3b4e36790a650712df06e8/tools/perf/page_sets/rendering/tough_pinch_zoom_cases.py#15?

If so, can you either tweak the title of this bug to reflect that TODO, or file a different bug & update the TODO text?

Looks liked Amy decided to file another bug & update the TODO reference (issue 852373)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ca03f2a1adb42da98b015940a516ca48d5c93af4

commit ca03f2a1adb42da98b015940a516ca48d5c93af4
Author: Amy Qiu <amyqiu@google.com>
Date: Wed Jun 13 18:19:17 2018

Fix bug number referenced in tough_pinch_zoom_cases TODO

After creating new bug for including tough_pinch_zoom_cases in
rendering.mobile, fix TODO to refrence correct bug number.

Bug:  851499 
Change-Id: I8657adb33b2e3ac246702a06356871c287522e4d
Reviewed-on: https://chromium-review.googlesource.com/1099037
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Amy Qiu <amyqiu@google.com>
Cr-Commit-Position: refs/heads/master@{#566920}
[modify] https://crrev.com/ca03f2a1adb42da98b015940a516ca48d5c93af4/tools/perf/page_sets/rendering/tough_pinch_zoom_cases.py

Looks like this is still failing:

https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus5X%20Perf/1913

https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf%2F1913%2F%2B%2Frecipes%2Fsteps%2Frendering.mobile_on_Android_device_Nexus_5X%2F0%2Fstdout


Unexpected Failures:
* rendering.mobile/wordpress
 
Unexpected Skips:
* rendering.mobile/wordpress_desktop_gpu_raster
* rendering.mobile/worldjournal_mobile
* rendering.mobile/worldjournal_mobile_sync_scroll
* rendering.mobile/wow_wiki_pathological
* rendering.mobile/wowwiki_mobile
* rendering.mobile/wowwiki_mobile_sync_scroll
* rendering.mobile/yahoo_answers
* rendering.mobile/yahoo_answers_desktop_gpu_raster
* rendering.mobile/yahoo_answers_mobile
* rendering.mobile/yahoo_answers_mobile_sync_scroll
* rendering.mobile/yahoo_games
* rendering.mobile/yahoo_games_desktop_gpu_raster
* rendering.mobile/yahoo_news
* rendering.mobile/yahoo_news_desktop_gpu_raster
* rendering.mobile/yahoo_news_mobile
* rendering.mobile/yahoo_news_mobile_sync_scroll
* rendering.mobile/yahoo_sports
* rendering.mobile/yahoo_sports_desktop_gpu_raster
* rendering.mobile/yahoo_sports_pathological
* rendering.mobile/youtube
* rendering.mobile/youtube_desktop_gpu_raster
* rendering.mobile/youtube_mobile
* rendering.mobile/youtube_mobile_sync_scroll
* rendering.mobile/yuv_decoding
* rendering.mobile/yuv_decoding_gpu_rasterization_and_decoding
* rendering.mobile/zdnet_pathological
 

Hmm, it doesn't seem like this is the same issue? From the logs, the cause doesn't seem to be the test itself, but instead a timeout error due to waiting for a thread. Also, as far as I can tell, this hasn't been happening in builds since then.

Marking this as fixed since the page set has been consistently passing.