All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=900663

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=0d919647df734c34203db7f0ada1e882ba8bb813f629cb988aa73d1c29ae4dce


Bot(s) for this bug's original alert(s):

Android Nexus5 Perf

rendering.mobile - Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/106fc726e40000

📍 Found significant differences after each of 3 commits.
https://pinpoint-dot-chromeperf.appspot.com/job/106fc726e40000

Remove --limit-fps flag. by kylechar@chromium.org
https://chromium.googlesource.com/chromium/src/+/7c62990ade547e536439b34a278c6369ef1c8582
thread_total_all_cpu_time_per_frame: 15.25 → 15.45 (+0.2022)

Reland "Enable Perfetto by default for all telemetry tests" by oysteine@chromium.org
https://chromium.googlesource.com/chromium/src/+/3bb24369bef996b43b2d2711f0033ba472f779ea
thread_total_all_cpu_time_per_frame: 15.39 → 16.67 (+1.278)

[chrome:omnibox] Encapsulate custom elements and update orchestration of inputs by manukh@chromium.org
https://chromium.googlesource.com/chromium/src/+/1cbd30f237d8502f9f46549a0f992b3a22be7fa5
thread_total_all_cpu_time_per_frame: 16.67 → 16.97 (+0.2999)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/12efa888140000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/15a4a790140000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1261517be40000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/15a4a790140000

Reland "Enable Perfetto by default for all telemetry tests" by oysteine@chromium.org
https://chromium.googlesource.com/chromium/src/+/3bb24369bef996b43b2d2711f0033ba472f779ea
thread_total_all_cpu_time_per_frame: 15.17 → 16.61 (+1.435)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/1261517be40000

[chrome:omnibox] Encapsulate custom elements and update orchestration of inputs by commit-bot@chromium.org
https://chromium-review.googlesource.com/c/chromium/src/+/1299534/10
thread_total_all_cpu_time_per_frame: 15.59 → 16.79 (+1.2)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1389c588140000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/1389c588140000

[chrome:omnibox] Encapsulate custom elements and update orchestration of inputs by commit-bot@chromium.org
https://chromium-review.googlesource.com/c/chromium/src/+/1299534/10
thread_total_all_cpu_time_per_frame: 14.8 → 16.62 (+1.817)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/16ce6330140000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/15af4bec140000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/16ce6330140000

[chrome:omnibox] Encapsulate custom elements and update orchestration of inputs by commit-bot@chromium.org
https://chromium-review.googlesource.com/c/chromium/src/+/1299534/10
thread_total_all_cpu_time_per_frame: 14.67 → 16.64 (+1.978)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/16dbe5ebe40000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/107c458fe40000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/107c458fe40000

Reland "Enable Perfetto by default for all telemetry tests" by commit-bot@chromium.org
https://chromium-review.googlesource.com/c/chromium/src/+/1307793/3
thread_total_all_cpu_time_per_frame: 14.76 → 16.65 (+1.886)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/16dbe5ebe40000

Remove --limit-fps flag. by commit-bot@chromium.org
https://chromium-review.googlesource.com/c/chromium/src/+/1305244/2
thread_total_all_cpu_time_per_frame: 14.77 → 15.3 (+0.5314)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

This is a little strange -- in the last pinpoint looks like both traces are using perfetto (services/tracing/public/cpp/perfetto/task_runner.cc shows up), but the older one is missing some detail. For example IO thread tasks are labeled as "OnLibevent" while in the new one they're "MessagePumpLibevent::OnLibeventNotification".

Oystein renamed the trace event at some point:
https://cs.chromium.org/chromium/src/base/message_loop/message_pump_libevent.cc?type=cs&q=OnLibevent&sq=package:chromium&g=0&l=309

(https://chromium-review.googlesource.com/c/chromium/src/+/1330067)

Not sure if this is really related to this regression though? I could imagine that missing worker thread events are causing this, too, see  issue 907067 .

I don't have a complete grasp on our metrics framework, but as far as I can tell, this metric story `css.opacity.plus.n.layers.99` is related to `css_opacity_plus_n_layers.html` which is measuring performance of changing css opacity on a div. My CL is changing an internal debug page (chrome:omnibox), and it would seem strange if it could impact the performance of applying css styling on a different page.

oysteine@ do you think this may be related to `Reland "Enable Perfetto by default for all telemetry tests"`?

This is most likely related to the perfetto change, yes.

Recovered with http://crrev.com/612360