All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=752409

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=de4ef6eb73431f1278c95c129b75f23fa014ff1d76161fb95bcc6fc7fcc3f013


Bot(s) for this bug's original alert(s):

android-nexus5

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8972207569239140320

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : cc_perftests
  Metric       : StructTraits serialization min_frame_serialization_time_per_quad_shared_quad_state/DelegatedFrame_ManyQuads_1_4000


To Run This Test
  src/build/android/test_runner.py gtest --release -s cc_perftests --verbose

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8972207569239140320


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8968094148931874816

I re-kicked a bisect on a wider range. But cc-ing test owner enne: this test failed when we tried to bisect, not sure what we can do about the regression.

What does failure mean here? Is this a timeout that we need to increase the limit for?

weiliangc: since you've been looking into compositor frame serialization times, can you investigate this?

Ooops, I should have read the logs more carefully. Looks like there was actually a device issue:
https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/5477
https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Ftryserver.chromium.perf%2Fandroid_nexus5_perf_bisect%2F5477%2F%2B%2Frecipes%2Fsteps%2FGathering_reference_values%2F0%2Fsteps%2FPerformance_Test_1_of_6%2F0%2Flogs%2FFailure_Output%2F0

Traceback (most recent call last):
  File "/b/c/b/android_nexus5_perf_bisect/src/build/android/test_runner.py", line 965, in main
    return RunTestsCommand(args)
  File "/b/c/b/android_nexus5_perf_bisect/src/build/android/test_runner.py", line 715, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/b/c/b/android_nexus5_perf_bisect/src/build/android/test_runner.py", line 815, in RunTestsInPlatformMode
    test_instance = test_instance_factory.CreateTestInstance(args, infra_error)
  File "/b/c/b/android_nexus5_perf_bisect/src/build/android/pylib/base/test_instance_factory.py", line 18, in CreateTestInstance
    args, device_dependencies.GetDataDependencies, error_func)
  File "/b/c/b/android_nexus5_perf_bisect/src/build/android/pylib/gtest/gtest_test_instance.py", line 307, in __init__
    error_func('Could not find apk or executable for %s' % self._suite)
  File "/b/c/b/android_nexus5_perf_bisect/src/build/android/test_runner.py", line 738, in infra_error
    sys.exit(constants.INFRA_EXIT_CODE)
SystemExit: 87

+simonhatch: is this transient, or actually about not building the right APK?

oops actually +simonhatch, see #8

Hmm I'm not seeing cc_perftests_apk/cc_perftests-debug.apk in the unzipped files list, but I see it on the swarming page for cc_perftests on the waterfall. Will try to see why it's missing.

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : cc_perftests
  Metric       : StructTraits serialization min_frame_serialization_time_per_quad_shared_quad_state/DelegatedFrame_ManyQuads_1_4000


To Run This Test
  src/build/android/test_runner.py gtest --release -s cc_perftests --verbose

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8968094148931874816


For feedback, file a bug with component Speed>Bisection

So it kinda looks like the Android Compile bot actually does build and upload it, but that's to gs://chrome-perf/Android Compile/etc. but the Android Builder doesn't. It tries to build the targets specified here:

https://cs.chromium.org/chromium/build/scripts/slave/recipes/android/builder.py?sq=package:chromium&dr=C&l=23 + chromium_builder_perf, which shouldn't do anything on Android.

This is what's uploaded to gs://chrome-perf/Android Builder/etc. which is what the bisectors use. I imagine we could just sync the targets between the tryserver list (further down in the file), or optionally could try to get chromium_builder_perf working. I'm not sure what other uses chromium_builder_perf target had besides waterfall perf/bisect, and since the waterfall is swarmed now and  Pinpoint is rolling out as we speak it might not be worth it to update that target.

+dtu/jbudorick owners of recipe

So there are a few issues here:

1. The builds in gs://chrome-perf/Android Builder/foo don't contain the right targets (example failed bisect: https://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/471). Dave mentioned these bots were actually deprecated back in Feb, and the bisect recipe should be using "Android Compile". So I tested with some changes on staging, and that run was successful (after some dashboard/other recipe changes):

https://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/469

Culprit was:

{
  "author": "Ken Rockot",
  "body": "\nThis is a reland of 0463c3dc61221f43758bb30f9a4e88b4e1e80fe0\n\nBreakage was caused by a defunct test. Test has been deleted.\n\nOriginal change's description:\n> Mojo C++ Bindings: Dynamic message allocation\n> \n> Reduces the \"prepare-to-serialize\" bindings step to handle and\n> interface collection instead of full message size measurement.\n> \n> Allows the serialized message buffer to be expanded dynamically\n> during serialization, and reworks the serialization code to support\n> writing into a buffer which may be reallocated (and thus relocated)\n> between operations.\n> \n> Bug:  742369 \n> Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel\n> Change-Id: I5cacee48343f565d68a77455872b0bf4e74e7674\n> \n> TBR=sky@chromium.org\n> \n> Change-Id: I5cacee48343f565d68a77455872b0bf4e74e7674\n> Reviewed-on: https://chromium-review.googlesource.com/580568\n> Commit-Queue: Ken Rockot <rockot@chromium.org>\n> Reviewed-by: Yuzhu Shen <yzshen@chromium.org>\n> Cr-Commit-Position: refs/heads/master@{#489210}\n\nBug: 742369\nChange-Id: Ic3a55bae2977fdeec93fbeddd95b04dc6aabe880\nCq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel\n\nTBR=yzshen@chromium.org\nTBR=sky@chromium.org\n\nChange-Id: Ic3a55bae2977fdeec93fbeddd95b04dc6aabe880\nReviewed-on: https://chromium-review.googlesource.com/584038\nCommit-Queue: Ken Rockot <rockot@chromium.org>\nReviewed-by: Ken Rockot <rockot@chromium.org>\nCr-Commit-Position: refs/heads/master@{#489326}",
  "date": "Tue Jul 25 16:26:24 2017",
  "email": "rockot@chromium.org",
  "subject": "Reland \"Mojo C++ Bindings: Dynamic message allocation\""
}

* I can dig up the full set of values if needed, in case this was a super noisy or funky result.

The catch here is that the bisect builders upload to gs://chrome-perf/Android Builder/foo, so we'll either need to change it to upload to gs://chrome-perf/Android Compile/foo, or less appealing make the bisect recipe check both. Switching to "Android Compile" right now will most likely result in every attempt to descend into other depots (v8/skia/etc.) to fail.


2. Dashboard's invocation command was out of date, changed it to match the command sent to swarming for the waterfall bots, along with some changes on the bisect recipe's side to fill in some additional params.


Lastly, we're in the middle of rolling out Pinpoint, and it's very likely that within the next month the android bisects will be running through that.

 Issue 758990  has been merged into this issue.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8958202997727243152

I kicked off another bisect for this bug since it looks like the problems with bisection should have been fixed in #13. Unassigning since I think weiliangc was looking into test failures, but the failure turned out to be on the perf builder side.

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : cc_perftests
  Metric       : StructTraits serialization min_frame_serialization_time_per_quad_shared_quad_state/DelegatedFrame_ManyQuads_1_4000


To Run This Test
  src/build/android/test_runner.py gtest --release -s cc_perftests --verbose

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8958202997727243152


For feedback, file a bug with component Speed>Bisection

Re-kicking bisect on pinpoint, to see if that helps.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14f313dc440000

Kicking again, not sure what happened last time.

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/14f313dc440000

Looks like Pinpoint can't build revisions that old.

MBErr: Builder name "Android Compile"  not found under masters[tryserver.chromium.perf] in "/b/c/b/Android_Compile/src/tools/mb/mb_config.pyl"

It's theoretically doable in the future with something like https://github.com/catapult-project/catapult/issues/3516

Doesn't look like we'll get to the bottom of this.