All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=614858

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgrLn4-AoM


Bot(s) for this bug's original alert(s):

android-nexus5

Based on bisect results, this change seems to be the culprit:

commit	4297d7d9e0cd33ac6673ec6a839b1501baabeba8	
author	ananta <ananta@chromium.org>	Wed May 25 00:39:15 2016
committer	Commit bot <commit-bot@chromium.org>	Wed May 25 00:40:52 2016
The on screen keyboard on Windows 8+ should not obscure the input field.

It looks like the bisect did not finish. Since this benchmark checks tracing overhead, it's more likely to be the other CL between good and bad, r395753, which is a catapult roll.  It has two CLs:

rnephew:
Make traces produced by telemetry combine all the trace parts
reland of: https://codereview.chromium.org/1980773002/
BUG=catapult:#2309
Review-Url: https://codereview.chromium.org/2004103002


nednguyen:
[Tracing] Export TraceConfig to model.metadata
BUG=catapult:#2336
Review-Url: https://codereview.chromium.org/2013553002

Randy, assigning to you since this was a reland and there was a revert/reland pattern in the graphs (although i didn't check that all the bumps line up)

This definilty seems like it might be that cl. 
https://codereview.chromium.org/2004103002/
Before it was only adding events from the chrome trace. Now it will go through the trace_data object and find any active trace and add them. The trace_data object can contain trace parts with the IDs BATTOR_TRACE_PART, CHROME_TRACE_PART, INSPECTOR_TRACE_PART, SURFACE_FLINGER_PART, TAB_ID_PART, and TELEMETRY_PART. 

For the battor benchmarks we need at least CHROME_TRACE_PART, BATTOR_TRACE_PART, and TELEMETRY_PART. I am not sure exactly the path forward here, I could get rid of the other parts.

Randy: if we can confirm that it's your CL that causes this "regression", there is no need to do anything besides we are all happy that our tracing_overhead is tracking the right thing & mark this bug "Working as Intended" :-)

Can you revert that patch locally & see if it indeed is the cause of the regression?

Randy: we can also look at the trace got uploaded by telemetry before & after regression point & look at the trace size stat panel.

I did a local run with and without the other traces, and the numbers are not matching up with this regression.

Probably not my CL.

Deleted: results.html 125 KB

results.html 125 KB View Download

Assign this to Ananta since the auto bisect point to your CL.

My patch cannot cause this. It should only kick in for Windows devices with touch screens.

Here are the trace before vs after regression.
Before:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_3-2016-05-24_07-43-25-86603.html
After:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_3-2016-05-24_22-33-25-69888.html

Nothing really pops out of the traces. Kicked off another bisect in https://chromeperf.appspot.com/buildbucket_job_status/9011117675501047136

This regression is still on going.  It looked like it got better for a while before getting bad again.

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9007038491138361296

There's a land/revert/reland pattern going on here, and each change has a catapult roll in the range. But there's no patching being reverted/relanded, or at least not obviously.

I don't have any other ideas here. Ned, do you?

I notice a lot of events increasing are due to V8. From the traces in #10:
Before:
V8.CompileCode	14342
V8.CompileFullCode	3351
V8.ParseLazy	3013
V8.PreParse	2368

After:
V8.CompileCode	18469
V8.CompileFullCode	4372
V8.ParseLazy	3966
V8.PreParse	2368


So out of ~11000 events increasing, these V8 events increase by 6000 events, which account for 50% of events increases.

I wonder whether this is related to v8 ignition?

Though it worth noting that ref build graph also moves in alignment with the TOT. So there is something about hardware that affecting how v8 scheduling works in both cases?

The numbers are back to normal, so I am marking this as WontFix.