All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=818697

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=53f9c9b26f3490b8060e6af3bdc7a588afb6158c493eb982e93e32a93b5db5e9


Bot(s) for this bug's original alert(s):

android-nexus5
android-nexus6

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14e072e2440000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/14e072e2440000

Enable CodeCacheAfterExecute by default. by mythria@chromium.org
https://chromium.googlesource.com/chromium/src/+/4043418267791fb7e8f89150f3dcfbadb5eeef16

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

It looks like the CodeCacheAfterExecute change regressed the PWA benchmark for loading FlipKart Lite by about 100 ms (5%).

This change also landed one day before the M66 branch cut, which seems risky regardless. Would you consider reverting it for M66 and retargeting for M67, if we don't understand the cause of the regression?

(cc: kinuko, kouhei, ksakamoto for loading/metrics expertise)

I spent some more time looking at graphs[1]. Some findings:
- The metric that regressed is "CPU time to first meaningful paint (FMP)" for "hot" FlipKart (this means a page load after service worker has been installed and code compiled) on Nexus 5 and 6.
- I don't see any other loading.mobile metrics changing in either a good or bad way. Notably of the 4 PWAs, only FlipKart showed a regression. Also similar metrics like "wall time to FMP" or "wall time first contentful paint (FCP)" didn't change (we don't seem to have CPU time to FCP).

I believe FMP might be finicky and we seem to trust FCP over FMP lately?

Since only "CPU time to FMP" for one of four PWAs changed, and wall time for FMP or FCP didn't change, I guess a case could be made that this isn't blocking, especially if the Finch experiment showed good results. It would be more reassuring to see the FMP/FCP metrics get better though rather than being unchanged. Are there benchmarks other than loading.mobile that might have improved with this change (or subtests other than the PWA ones)? And did Finch show any good results on loading metrics?

[1] https://chromeperf.appspot.com/report?sid=16902b615f081bf5b0d54ee52a032005ebbabb59109588ead842cc46e31d796e&start_rev=537560&end_rev=540990

From loading PoV, this isn't blocking since wall time for FMP is unchanged.

CPUTimeToFMP "regression" means that CPU usage until FMP has increaced, which is not necessarily a bad thing.
It seems mythria@'s change generates more code cache. Does it explain the CPU usage increase? If so, we can WontFix this.

Thanks ksakamoto, that would make sense to me. Is there ever a time we would care when CPU time to FMP regressed and wall time didn't? Could we remove it as a metric to track or get alerts for?

We don't really care CPUTimeToFMP regression if wall time FMP did not change.

I agree to what Sakamoto says in #8!

Great :) Can we remove the metric or otherwise halt further alerts about it? How would one go about doing that?

+Tim: FYI about the problem of false alerts caused by CPUTimeToFMP.

+Simon can help with halting the alert for this metric

Thanks for looking into it. I also think this can be marked won't fix. On the hot runs we deserialize more code and hence may be a slight increase in cpuTime is expected during the initial compile. 

From the finch experiment, we saw a slight reduction in the NavigationToDomContentLoadedEventFired and NavigationToFirstInteractive events. So, I was checking for TTI metric on these pages. It appears that we don't have them since two weeks [1]. May be related to another  crbug.com/809833 . 
 
[1] https://chromeperf.appspot.com/report?sid=a0be20094f09de0930981c8e8d3f85696381b5037ec29aa2dba2ea79e59dcb41&start_rev=535832&end_rev=541050

Disabling alerts on CPU time to FMP sounds reasonable to me.
The hope was that it would be less noisy than wall clock time FMP, and could be useful as a diagnostic.

It doesn't feel like it's providing much value though.
Deep, any thoughts?

+1 to disabling alerts on CPU time. 

Just to confirm, you would like to remove:

ChromiumPerf/*/loading.desktop/cpuTimeToFirstMeaningfulPaint_avg/*/*

and should I also remove:

ChromiumPerf/*/loading.mobile/cpuTimeToFirstMeaningfulPaint_avg/*/*

Yup, remove them both. Thanks!

Ok done.

tdresser, dproy, simonhatch: thank you!