Previous work:

 * https://docs.google.com/document/d/1wmKPgA0BFHvzNISgrXCK8-61Q7UIGmyr0k1m20jwR1U/edit
 * https://docs.google.com/document/d/1dtSXDGBZTAWy2a4NKbKvQu4250KnviliT7JrKNlXf0U/edit
 * https://docs.google.com/document/d/1EF4nRC1N3sUN7GiFf6_L7KD9qa8gfU2sboAlJHodbAk/edit
 * https://docs.google.com/document/d/1sUGV0qZWg0xh4uNB8KMDwfQQvmzz4aoFObtI0Y_R90o/edit

From Hello Chrome we found that removing following services had most impact on startup:

 * Various initializations[1] that delay startup
 * History service
 * Sync
 * Net predictor

Any idea how many of these would be affected by async KeyedService?

[1] https://team.git.corp.google.com/hello-chrome/HelloChrome2/+/20107a91d17811fc3df3df46e2f6fe6f45925a13

Any traces? (couldn't find them in the doc)

CC caitkp who was involved in the aforementioned 2013 attempts, FYI

Here is a trace from a 512MB device (warning: 180MB :)

https://drive.google.com/open?id=1IWL1Szjncf6L7zCKaeSE5IHT7a3MeZ39

This one should be taken with a grain of salt since there is a lot of descheduling happening during native initialization. This one looks a little more realistic (although still very descheduled):

https://drive.google.com/open?id=1gmb-p64UofCJOtw-zllPYpIcammeHAVi

(Working on a doc with more details.)

Interesting, how do you blame sync from these traces? And history? (Chrome_History thread name? It's about to be coalesced in TaskScheduler workers FWIW)

Any idea why TaskScheduler workers aren't labeled on these traces?

Guessing you only know there are a lot of descheds for blocking I/O but we don't if implicit (paging) or explicit (synchronous reads) I/O?

Excited to read your doc/findings :)

Ah, the way we narrowed this down to the list of services above was by removing them one by one. The numbers for that are here:

https://docs.google.com/spreadsheets/d/1Ch0KLhGUyDStuMcj7gfkWTM4gzB52Emmamm-hxLx-Ww/edit#gid=1870269173

The traces above were taken with a build from a ~month ago so maybe some labeling wasn't there yet?

And yes, the blocking I/O is suspicious and being investigated in bug 849316. I'm fairly certain it's implicit paging, because if we're doing that much reading on the UI thread something is pretty badly wrong :)

Re. "if we're doing that much reading on the UI thread something is pretty badly wrong :)"

Yeah... well I wouldn't be surprised if something is pretty badly wrong either. We do do a lot of I/O on the UI thread early. There's a policy to allow I/O for initialization until the message loop is started. Some of this I/O is indeed blocking initialization (e.g. reading Local State/field trials), other might be parallelizable but never was.

Ah, you're right, I remember seeing an exception for early startup. I think we're going to do a full I/O activity analysis of startup anyway -- that should highlight problems from explicit reads/writes like that too.