Here's the correct link to the failing bot: https://build.chromium.org/p/chromium.perf/builders/Mac%20HDD%20Perf%20(5)

Disable: https://codereview.chromium.org/1807183002/
Bisect: https://chromeperf.appspot.com/buildbucket_job_status/9017945617270392848

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f6cf0cb1501ea3e08b196943d754c3df049a052a

commit f6cf0cb1501ea3e08b196943d754c3df049a052a
Author: petrcermak <petrcermak@chromium.org>
Date: Thu Mar 17 12:33:52 2016

Disable media.tough_video_cases on Mac

The benchmark is causing lots of test failures on the "Mac HDD Perf (5)"
perf bot (see the associated bug).

BUG= 595665 
TBR=skyostil
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review URL: https://codereview.chromium.org/1807183002

Cr-Commit-Position: refs/heads/master@{#381695}

[modify] https://crrev.com/f6cf0cb1501ea3e08b196943d754c3df049a052a/tools/perf/benchmarks/media.py

The benchmark is now disabled, so I'm lowering this bug's priority and assigning it to the benchmark owner.

Note that the bisect is still running.

prabhur@ doesn't work on Videostack anymore.

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [Courgette] Clean up Disassembler; fix ELF Memory leaks.
Author  : huangs
Commit description:
  
Cleaning up code surrounding Disassembler:
- Extract AddressTranslator interface to be used across subclasses.
- Use FileOffset = size_t by context.
- Detailed comments & TODOs in DisassemblerElf32ARM.
- Fix DisassemblerElf32ARM memory leaks.
- Lots of superficial stylistic changes.

Except for AddressTranslator routines and unit tests, shying away
from control flow and logic changes.

BUG= 579206 

Committed: https://crrev.com/58b822d441f5c982e879e536fa3c1cbac8fd339a
Cr-Commit-Position: refs/heads/master@{#380881}

Review URL: https://codereview.chromium.org/1676683002

Cr-Commit-Position: refs/heads/master@{#380987}
Commit  : dda11d069213d53aef9bf3bc018b02f3aef7177b
Date    : Mon Mar 14 16:37:34 2016


===== TESTED REVISIONS =====
Revision                Exit Code   Std. Dev.   Num Values  Good?
chromium@380887         1           N/A         5           good
chromium@380937         1           N/A         5           good
chromium@380962         1           N/A         5           good
chromium@380975         1           N/A         5           good
chromium@380981         1           N/A         5           good
chromium@380984         1           N/A         5           good
chromium@380986         1           N/A         5           good
chromium@380987         1           N/A         5           bad

Bisect job ran on: mac_hdd_perf_bisect
Bug ID: 595665

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests media.tough_video_cases
Test Metric: idle_wakeups_total/video.html?src_smpte_3840x2160_60fps_vp9.webm
Relative Change: 0.00%
Score: 0.0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_hdd_perf_bisect/builds/451
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9017945617270392848


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!

Bot health sheriff ping; this test is still disabled.

According to the original post, the test was flaky but didn't fail 100% of the time, and it failed with a timeout.

If the Page36 test (see src/tools/perf/tough_video_cases.py line 488) is the only test that has problems, then perhaps it would make sense to simply remove it for now (with a TODO), so that the rest of the pages can be reenabled. The Page36 test is 4k 60fps VP9 video, so it makes sense that it's having problems. 

Removing Page36 would be as easy as deleting the line: 

self.AddStory(Page36(self))

in src/tools/perf/page_sets/tough_video_cases.py line 579. 

Another idea would be to refactor the toughest tough video cases into a separate page set, and make a new benchmark for it. Here's an example: In https://codereview.chromium.org/1612333002 they break up a pageset and a benchmark into two pagesets and two benchmarks. One possibility would be to include the easier, less likely to be flaky pages in one pageset/benchmark so that they can all be reenabled, and then include the more difficult benchmarks in the separate pageset/benchmark.

petrcermak@, qyearsley@, does this seem reasonable?

Both of those options (1, removing that page and adding a TODO and and an associated bug and then reenabling the test sounds; 2, make a separate "easier" page set) sound reasonable to me.

I guess it's up to what makes sense in terms of what we want to test in the future -- if there's not a lot of long-term value in separately measuring the "easier" page set, then the first option is probably better.

Note that removing one page will change the average, which will likely cause a false positive alert (which isn't a problem, but you should note this in the description of the CL which removes the pate)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/65d76370288406e505db5efe99298e5266b4f742

commit 65d76370288406e505db5efe99298e5266b4f742
Author: sandersd <sandersd@chromium.org>
Date: Tue May 10 19:29:26 2016

Re-enable media.tough_video_cases on Mac

This CL bumps the timeouts for 4k playback to reduce flakiness on Mac.

BUG= 595665 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review-Url: https://codereview.chromium.org/1962563005
Cr-Commit-Position: refs/heads/master@{#392681}

[modify] https://crrev.com/65d76370288406e505db5efe99298e5266b4f742/tools/perf/benchmarks/media.py
[modify] https://crrev.com/65d76370288406e505db5efe99298e5266b4f742/tools/perf/page_sets/tough_video_cases.py