All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=600293

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg8JeepgoM


Bot(s) for this bug's original alert(s):

health-plan-clankium-phone

Kicked off a bisect now that the blocking bug is updated: https://chromereviews.googleplex.com/396827013

The bisect failed :(

Relevant log isn't very helpful:

https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium_nexus5_perf_bisect/builds/23/steps/run%20bisect%20perf%20regression/logs/stdio

Here is the relevant log:
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium_nexus5_perf_bisect/builds/23/steps/Results/logs/stdio


Runtime Error: An error occurred while building and running the 'bad' reference value. The bisect cannot continue without a working 'bad' revision to start from.

Error: Metric ['background-memory_mmaps_ashmem_total', 'background-memory_mmaps_ashmem_total'] was not found in the test output.


Looks like the metrics aren't being output in the test. Any idea why that is happening? Is there flake in the test?

https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium_nexus5_perf_bisect
This bot always fails recently. Could be a bot issue?

https://chromeperf.appspot.com/report?sid=b766f8fa42c24e3c564319e734651d3544dbfb169cfe7d3f2fe32cd3c5f4247c&start_rev=382242&end_rev=384837
Here is a graph that shows the same test on health-plan-clankium-low-end-phone. This graph looks relatively smooth compared to the phone bot, but we can find similar regression in similar range.

I kicked another bisect for this low-end-phone bot.

Hum... configuration looks wrong?

@@@STEP_NEST_LEVEL@1@@@
fatal: bad object 5f88c17ffe63fb0d72f34850a764c75b0dda3093
step returned non-zero exit code: 128
@@@STEP_EXCEPTION@@@

I kicked additional two bisects, that's same test between revision 383536:383575 running on Nexus4 and 5X.

https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium_nexus5x_perf_bisect/builds/4
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium_nexus4_svelte_perf_bisect/builds/26

Both fail at the "Expanding revision range" step.

https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium_nexus4_svelte_perf_bisect/builds/26/steps/Expanding%20revision%20range.for%20revisions%2022c8852e8f91c670cb7805229676b57810725d2a%3A27992628e395bc0264c7f73472011708dd1b3c72

Let me assign this to rnephew@ who is today's perfbot sheriff.

I know next to nothing about how the bisect bots work, I'm going to assign to Prasad since he knows much more. There was a problem he fixed yesterday where a config was wrong, I wonder if this is a similar issue?

[Bulk edit]

This bug is marked as a blocker for Android M51 beta, which is targeted for release on Apr 21, with the candidate build being cut on Apr 20.  As such, please fix the regression on trunk by no later than Tuesday, Apr 19 @ 5 PM PT in order to facilitate a merge.

100% sure this isn't a blocker?  Remove the ReleaseBlock-Beta label.  Think it shouldn't block, but not totally sure - or know it should block, but you won't be able to fix it in time?  Reply here and CC me.

There have been a lot of bisect fixes over the last few days, kicked off a few more:
https://chromeperf.appspot.com/buildbucket_job_status/9015303809245941920
https://chromeperf.appspot.com/buildbucket_job_status/9015303886171185632

Removing blockers from this. This is about a 23k regression on ashmem, which ultimately went down to +11k after later CLs. Not the end of the world, given the bigger problems we have. If we end up not shipping chrome because of 23k of discardable memory we should just leave and open an ice cream bar :-)

I'm leaving this open just for the sake of making sure that bisects can catch this. This seems pretty low noise, so I'd expect bisect to find them.

Just downstream bot links do not require RVG.