All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=712471

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgwq3n9QoM


Bot(s) for this bug's original alert(s):

cros-parrot

Shuo-Peng, is the new bisect script able to do this bisection?

Marking this for now as Kernel Graphics even though that seems a Chrome issue.

Let me try the (yet-to-commit) bisect tool to see if it is Chrome issue.

I still see the increased usage at today's ToT @ 465377.

Ok, Pohsien only has a minnie where the repro is not stable. I have a sentry and there it looks pretty clean, so I will manually bisect this the old fashioned way.

Yesterday I sanity checked CrOS 9454.0.0 and 9455.0.0. And then use simplechrome to build 59.0.3065.0 and 59.0.3068.1 to cross install (new Chrome to old CrOS DUT, and vice versa). I found that the first run of Aquarium autotest on GoldenEye image, the MemUsed is consist to what ChromePerf shows. But if it runs second time or runs after deploying locally build Chrome using simplechrome flow, the MemUsed is significantly higher.

I'm still taking at look why.

I just realized that sentry is not a great repro either. So restricted to older Intel? I will try peppy.

I ran Chrome bisect on caroline-release/R59-9454.0.0 . I sanity checked two end points, I ran test three times on each commit:
Last known good: 7f44763 Delay deleting profile notification
Good commit score mean: 1093586.667  STD: 14121.068

Last known bad: 2f0cddd Incrementing VERSION to 59.0.3068.1 
Bad commit score mean: 1095237.333  STD: 3188.907

As the mean score are quite close and STD are quite large, it is unable to bisect. Will investigate what causes such big variance.

Caroline
https://chromeperf.appspot.com/report?sid=b2c0a2734105a3fdf11fd3ba27784f14a6af86e88ebe508f14dd2b80cd9b8e1d&rev=30680010945600000

daisy_spring
https://chromeperf.appspot.com/report?sid=925e14a3880c35a2cdac213c52eb35c7f54c805bfd07eb1467f2cf9140769a8d&rev=30680010945500000

It clearly is a Chrome regression, but I was set back with bisecting as it requires --internal builds (which I unfortunately didn't use).

Continue bisection.

My manual bisection on peppy with OS image 9454 and internal Chrome builds went to a dead end as 463050 first showed as a "good" build, while much later it showed as a "bad" build.

So we know 463050 is bad.

I used hctsai@ cros bisect tool to try bisecting CrOS R59-9454.0.0...R59-9456.0.0 on caroline machine.

But during sanity check, the locally built images didn't reveal the difference (MemUsed):
9454: 3 iterations Values: [823540.0, 822292.0, 850816.0]  Average: 832216.000
9456: 3 iterations Values: [801124.0, 807208.0, 841524.0]  Average: 816618.667

I have a suspicion this could be due to issue 713968. Waiting for Chrome uprev.