New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 665201 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Regression - PerformanceMonitor.HighCPU.BrowserProcess on macOS 10.12

Project Member Reported by shrike@chromium.org, Nov 14 2016

Issue description

Comparing Mac, Windows, and Linux, seems to only be a regression on the Mac. Breaking out across macOS releases seems to be 10.12 only - see screenshot and URL:

https://uma.googleplex.com/timeline_v2?sid=ddf902a9f48bed078228dfb4fe097eca

Spikes between 54.0.2840.34 and 54.0.2840.41 - change log is https://chromium.googlesource.com/chromium/src/+log/54.0.2840.34..54.0.2840.41?pretty=fuller&n=10000 . Suspecting:

Hack for AIA fetching on Mac Sierra: On >=10.12, always do FETCH_CRL_FROM_NET if adding a crl policy
https://codereview.chromium.org/2371033002

There is also Mac

Hack around Sierra autolayout bug in the certificate viewer
https://codereview.chromium.org/2363823003

but that seems unlikely given that any performance problems should be confined to the certificate viewer.


 
Screen Shot 2016-11-14 at 3.23.18 PM.png
340 KB View Download

Comment 1 by tapted@chromium.org, Nov 14 2016

Are these normalized by user count? (if not, then that's probably more concerning).

Apple only started rolling out Sierra 10.12 from Sep 20.

54.0.2840.34 and 54.0.2840.41 were beta channel releases going out Sep 21 and Sep 28. I think 54.0.2840.59 was the first stable channel release and it went out Oct 12 (https://googlechromereleases.blogspot.com.au/search/label/Stable%20updates)

Comment 2 by mattm@chromium.org, Nov 15 2016

Looks like the spike also starts around the day Sierra was officially launched (sep 20). Since these numbers aren't normalized to user counts it's unreasonable to assume it was caused by any code changes in that region.

I would be very surprised if that CL is actually responsible for keeping browser process cpu load high for two minutes straight, but if it were I would expect the numbers to start going down due to https://bugs.chromium.org/p/chromium/issues/detail?id=655612#c25.

Comment 3 by shrike@chromium.org, Nov 15 2016

Ah, good point about 10.12 shipping on the day of the spike. It looks like that's probably it, as there's an even bigger regression in Stable starting on that date.

https://uma.googleplex.com/timeline_v2?sid=fc20d46caea70b293dac50d3f2e9826a
Screen Shot 2016-11-14 at 4.26.08 PM.png
255 KB View Download

Comment 4 by shrike@chromium.org, Nov 17 2016

Cc: erikc...@chromium.org
Going back a year, there's a spike around 10/1, and macOS 10.11 shipped on 9/30. It therefore seems there's a spike in PerformanceMonitor.HighCPU.BrowserProcess when Apple releases a new OS. I would expect this to be a spike but at least looking from last year the numbers never return to normal.

erikchen@ - any thoughts on how to figure out what's going on?

Screen Shot 2016-11-17 at 11.30.55 AM.png
269 KB View Download

Comment 6 by shrike@chromium.org, Mar 16 2017

Cc: mattm@chromium.org
Owner: shrike@chromium.org
Summary: Regression - PerformanceMonitor.HighCPU.BrowserProcess on macOS 10.12 (was: Regression - PerformanceMonitor.HighCPU.BrowserProcess in Chrome Mac Beta on 10.12)
There's the issue of normalizing by user count (not sure how to do that in the Timeline dashboard), and you can see how 10.11 events start falling as 10.12 events pick up, but if everything was OK I would expect the 10.12 events to level off at not more than 1.6M. We've gone way past that, and are still climbing.

https://uma.googleplex.com/timeline_v2?sid=70f3c80de07e8ebbd0c4094d9c2eb363

Screen Shot 2017-03-16 at 10.31.17 AM.png
342 KB View Download

Comment 7 by shrike@chromium.org, May 15 2017

This metric seems to be on its way to recovering: https://uma.googleplex.com/timeline_v2?sid=8504fcb18f1c61359fe204d2b10931a7.

I'm guessing 10.12.5, which was released on March 27, fixed a long-standing bug in 10.12.

Screen Shot 2017-05-15 at 3.50.06 PM.png
318 KB View Download

Comment 8 by shrike@chromium.org, Jul 24 2017

Cc: shrike@chromium.org
Owner: lgrey@chromium.org
This metric recovered some but not fully. It would be great, I think, to try to kick in the sampling profiler when we record a high value for this UMA metric so that we can see just what Chrome is doing during these times.

https://uma.googleplex.com/timeline_v2?sid=0819ddeb9c38721be73019fb85b517e6

Screen Shot 2017-07-24 at 3.46.46 PM.png
276 KB View Download

Comment 9 by lgrey@chromium.org, Jul 24 2017

Cc: wittman@chromium.org
wittman@ are we ready to do something like this yet or do we want to get further along re: stability?

Also, if we were to do this, would we need to build a new dashboard like the startup one, or is there some other way to get at the data?
If we're lucky we may be able to investigate using the startup data. I'll take a look at what we've collected and see if we can get any useful insights.

We probably should have a firmer idea about stability before extending to additional use cases like this. We would be able to reuse the dashboard -- the metrics team has someone working this quarter on adding the capabilities to slice the data by the additional dimensions we'd need for this.
Cc: isherman@chromium.org

Comment 12 by lgrey@chromium.org, Sep 12 2017

shrike@ do you know if there's a way to normalize that chart by users? Assuming we have more 10.12 users than 10.11/10.10 it's hard to judge exactly how much worse it is by absolute # of occurrences.
lgrey@ - not sure. I don't see a way to do it from the dashboard. I'm guessing it might take a custom dremel query. Can you ask the UMA folk?

Re: judging how bad it is, I agree that normalized metrics should help, but if you go back in time to before the 10.12 release when most everyone was running 10.11, the number of occurrences was never so high. For example, see the graph in c#4.

Comment 14 by lgrey@chromium.org, Sep 12 2017

Will do.

Agree about the chart at release, but I want to be able to judge the delta between then and now, and also judge improvement. Going back 500 days, the current numbers for 10.12 are about 30% higher than 10.11's peak (compare to the 10.12 peak in January which is 500% of the 10.11 peak), so there might not be anything there besides user growth/OS updating.
If you can get normalized data that would be great. It still seems likely that the extra 30% is regression given what the graph in c#4 shows. A 30% YoY increase in the number of Chrome users running the latest macOS seems unlikely.
Here's the graph from c#1 normalized by reported UMA record:

https://uma.googleplex.com/timeline_v2?sid=131e7f299356c3ab5e6c8c857aee5696

This would seem to support the notion that the OS update mentioned in c#7 did address the issue.
Screenshot 2017-09-13 at 12.22.31 PM.png
389 KB View Download
Status: Fixed (was: Assigned)
Just looked at the latest version of the graph wittman@ posted. I think this is fixed, will reopen if we see data to the contrary.

Sign in to add a comment