Hmm, in the metadata for the telemetry trace I see that os-arch is "armv8l" - are testing 64-bit Chrome builds?

I checked recent logs on the bot, the builds are 32bit.

recent log I took: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf.fyi%2FAndroid_Nexus_5X_Perf_FYI%2F1511%2F%2B%2Frecipes%2Fsteps%2Fextract_build%2F0%2Fstdout

so perhaps os-arch corresponds to the OS on the device?

on recent benchmark runs I see the gap to be more like 60-65ms

Yup, I thought os-arch comes from JNI (and 32-bit apps see 32-bit ABI), but it turns out it comes from uname.

OK, so I investigated this a bit. First, the same thing happens on sprout (1GiB Android One device) running M.

Next, I hacked ActivityThread and replaced Instrumentation instance with one that uses TraceEvents: https://chromium-review.googlesource.com/c/chromium/src/+/832819/6/base/android/java/src/org/chromium/base/TracingInstrumentation.java

Turned out that Instrumentation.newActivity(ClassLoader,...) is what is consuming all time. I investigated a bit deeper and found that (n5x-newActivity.png):

1. ClassLoader.loadClass("org.chromium.chrome.browser.ChromeTabbedActivity") took 6ms.

2. Instantiating that class took 76ms.
  2.1. 74ms passed until AsyncInitializationActivity <cinit> was run (i.e. class was loaded).

3. After loading and instantiating a class, TracingInstrumentation.newActivity() called the default implementation, which did the same (loading / instantiating a class) in just 0.1ms.

Based on this I suspect that the time is taken by loading base classes of AsyncInitializationActivity. But not android.app.Activity itself - traces from experimental.startup.android.coldish go through ChromeLauncherActivity, which loads android.app.Activity before routing to CTA, and still shows 100ms gap.

Deleted: n5x-newActivity.png 28.8 KB

	n5x-newActivity.png 28.8 KB View Download

Deleted: sprout-newActivity.png 37.2 KB

	sprout-newActivity.png 37.2 KB View Download

Deleted: n5x-newActivity.html.gz 1.1 MB

n5x-newActivity.html.gz 1.1 MB Download

nice! so if instantiating AsyncInitializationActivity takes so long, then 'DEX ordering' has large potential! (+mattcary, +lizeb)

dskiba: did you experiment with newly installed APK each time? Does it manifest the same behavior on consequent warm runs?

This happens on consecutive warm runs on both Nexus 5X (running N) and sprout (running M). But the gap is ~30ms on gobo running O, so it looks they optimized that (I need to test on some other phone running O though).

+agrieve in case anything jumps out related to dexloading

Sounds like a dex locality thing. From this year's IO, it was pointed out that O optimizes start-up classes better:
https://youtu.be/Yi9-BqUxsno?list=PLOU2XLYxmsIInFRc3M44HUTQc3b_YJ4-Y&t=915

Could also be that O has a faster dex verify step. If this is true, then having it reference fewer other classes would speed it up.

Available, but no owner or component? Please find a component, as no one will ever find this without one.