All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=826829

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=1b30b70ed36228672d85d2f7f71f97037660670cdf506ac52dd3a930ecb5dd4e


Bot(s) for this bug's original alert(s):

android-nexus5
android-nexus5X
android-nexus6
android-nexus7v2

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14b02cbb440000

📍 Found significant differences after each of 2 commits.
https://pinpoint-dot-chromeperf.appspot.com/job/14b02cbb440000

Roll src/third_party/catapult/ 7b53f088f..f73167a68 (1 commit) by catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com
https://chromium.googlesource.com/chromium/src/+/320ac37b57c63feeac20bd169d80efdaa4c0a298

Get navigation info without using FrameLoader snapshots by dproy@chromium.org
https://chromium.googlesource.com/catapult/+/5d35a2c7122c1ac4a9a37e7743a19f628f25a38b

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14ae2778c40000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/14ae2778c40000

AddStackTrace by dproy@chromium.org
https://chromium-review.googlesource.com/c/chromium/src/+/998059/1

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

dproy: Do you understand this regression?

It looks like page load time regressed across the board on system_health.common_mobile and loading.mobile.

I do - working on it. 

 Issue 827183  has been merged into this issue.

 Issue 826825  has been merged into this issue.

 Issue 827148  has been merged into this issue.

 Issue 825005  has been merged into this issue.

 Issue 828008  has been merged into this issue.

 Issue 831089  has been merged into this issue.

It looks likes this bug was uncovered by my CL, but the root cause is an edge case in the metric computation logic, and reverting my CL will fix the regression but the metric will still be wrong. Sending out a CL with a proper fix soon.

Actually, after digging in more, I believe the metric was wrong before, and is now more correct. Here is a history of what happened:

There are two relevant CLs:

CL 1 [1]: We start ignoring navigation start events with empty URLs.
CL 2 [2]: We start extracting the URL of a navigation start event directly from the navigationStart instant event. Before, we were computing the URL of a navigation start event by looking at the last FrameLoader object snapshot before it, and sometimes we had no snapshot or an outdated snapshot before and therefore got no URL or an outdated URL (See  Issue 824761  for more context.)

All the regressions that are still merged into this issue has the following pattern in pinpoint for FMP (See [3] for example):

Low FMP value -> Few commits in the middle with no value -> High FMP value.

The change from phase 1 to phase 2 happened when CL 1 landed. The change from phase 2 to phase 3 happened when CL 2 landed.

The traces that exhibit this pattern usually look like this (see [4] for example):

-NavStart 1 ----   FrameLoader snapshot 1 ----- NavStart 2 ------> (time)
URL: abc.com       URL: ""                      URL:""


If we use FrameLoader snapshots to compute the URL, then
- Computed URL for NavStart1 is empty (because there is no FrameLoader snapshot before it), and
- Computed URL for NavStart2 is also empty (because the FrameLoader snapshot before it has "" for URL.)

But if we extract the URL directly from the navigation start trace event, we have
- "abc.com" for NavStart 1, and
- "" for NavStart 2.

Now:
- Before CL 1, FMP used NavStart2 as starting point, so time to FMP was low.
- After CL 1, we excluded both NavStart1 and NavStart2 as starting points, since they both have empty computed URLs, and so we had no FMP values at all.
- After CL 2, We exclude only NavStart 2, and time to FMP is computed with NavStart 1 as the starting point, so the value is higher than before.

I checked the trace, and NavStart 1 always aligns with the NavigationToCommit slice in the browser while NavStart 2 doesn't, so I'm convinced it is the right navigation start event to use.

Now, why we sometimes have that NavStart2 event in the same frame as NavStart1 with empty documentLoaderURL is a mystery to me, but this bug has a lot of people in the cc list, so I'll reach out to loading folks separately to understand this.

Marking this as WontFix.


[1] https://chromium-review.googlesource.com/c/catapult/+/937701
[2] https://chromium-review.googlesource.com/c/catapult/+/968942
[3] https://pinpoint-dot-chromeperf.appspot.com/job/14b02cbb440000
[4] https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/VoiceMemos_2018-03-29_02-54-16_37950.html

 Issue 828336  has been merged into this issue.