From 666312, looks like the test runs fine and generates output but it isn't being picked up by the bisect. I'm not clear if we just need to change the commandline for these, or actually parse the output.
Since compare_samples looks like it already supports buildbot: https://github.com/catapult-project/catapult/blob/master/tracing/tracing/metrics/compare_samples_cmdline.html#L221 I tried downloading the captured output from this bisect run: https://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7050 ./tracing/bin/compare_samples ~/tmp/fake_bb_output1.txt ~/tmp/fake_bb_output2.txt CastV2Performance_gpu_60fps/av_sync --buildbot {"sampleA":[-41.272819],"sampleB":[-43.856306],"result":{"U":0,"p":1,"significance":"NEED_MORE_DATA"}} perf_test.py:130 does save the output specifically to parse buildbot formats, but that doesn't look like it gets saved by each revision. Think we just need to save that and fall back to buildbot if no chartjson or valuesets in bisector.py:179.
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/4393ca1fbafecd7976b6bd7458a4a52c50113120 commit 4393ca1fbafecd7976b6bd7458a4a52c50113120 Author: Simon <simonhatch@chromium.org> Date: Tue Nov 29 22:07:13 2016 Bisect - Fall back to buildbot paths if no chartjson or valuesets. Previously tests like performance_browser_tests failed bisecting since they only produce buildbot output. The compare_samples script supports this but we weren't passing it from the test, so this CL just does that. BUG= 668529 Change-Id: Ia0f71ee50d0b1ce0624e7e024221626a0a3eddf2 Reviewed-on: https://chromium-review.googlesource.com/414592 Reviewed-by: Prasad Vuppalapu <prasadv@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/4393ca1fbafecd7976b6bd7458a4a52c50113120/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py [modify] https://crrev.com/4393ca1fbafecd7976b6bd7458a4a52c50113120/scripts/slave/recipe_modules/auto_bisect_staging/revision_state.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ff6f9e8c02c4242c8fa3df7ff217756d8760a1e1 commit ff6f9e8c02c4242c8fa3df7ff217756d8760a1e1 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Thu Dec 01 18:00:44 2016 Roll src/third_party/catapult/ 852f85163..da3aa694a (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/852f85163d54..da3aa694af97 $ git log 852f85163..da3aa694a --date=short --no-merges --format='%ad %ae %s' 2016-12-01 simonhatch Dashboard - Also check for "staging_win" when modifying non-telemetry commands. BUG= 668529 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2546653004 Cr-Commit-Position: refs/heads/master@{#435649} [modify] https://crrev.com/ff6f9e8c02c4242c8fa3df7ff217756d8760a1e1/DEPS
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/a5700d90eca1e05a02bf85b8993541f5ca631fb8 commit a5700d90eca1e05a02bf85b8993541f5ca631fb8 Author: Simon <simonhatch@chromium.org> Date: Fri Dec 02 17:06:45 2016 Push bisect_staging changes to prod Bisect - Check for buildbot output tests. Bisect - Fail if no values from "Gathering Reference Values BUG= 666447 , 668529 Change-Id: Ida0ac3db2e04f5ab4ed17bb3f0910ed4af77dd0d Reviewed-on: https://chromium-review.googlesource.com/415618 Reviewed-by: Prasad Vuppalapu <prasadv@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/auto_bisect/bisector.py [add] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/auto_bisect/example.expected/basic_buildbot_bisect.json [modify] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/auto_bisect/example.expected/failed_buildbucket_get.json [add] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/auto_bisect/example.expected/gathering_references_no_values.json [modify] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/auto_bisect/example.py [modify] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/auto_bisect/revision_state.py [modify] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipe_modules/bisect_tester/perf_test.py [modify] https://crrev.com/a5700d90eca1e05a02bf85b8993541f5ca631fb8/scripts/slave/recipes/bisection/desktop_bisect.expected/basic_perf_tryjob_with_metric.json
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/4f278c2d103de31f945e419de09916a0f23d1a75 commit 4f278c2d103de31f945e419de09916a0f23d1a75 Author: Simon <simonhatch@chromium.org> Date: Tue Dec 06 17:11:03 2016 Bisect - Add check for gtest performance_browser_tests. BUG= 668529 Change-Id: I93fc67499decb68dfd7aa56be695b5dfd386004a Reviewed-on: https://chromium-review.googlesource.com/417099 Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/4f278c2d103de31f945e419de09916a0f23d1a75/scripts/slave/recipe_modules/bisect_tester/perf_test.py [modify] https://crrev.com/4f278c2d103de31f945e419de09916a0f23d1a75/scripts/slave/recipe_modules/bisect_tester_staging/perf_test.py
Comment 1 by simonhatch@chromium.org
, Nov 24 2016