tl;dr: WAI, unfortunate consequence of the current prerendering implementation.

This happens because when GSA triggers prerendering, the renderer is not the same one as the spare renderer we use to speed up normal page loads.
1. When a client calls warmup(), we create a spare renderer. (sandboxed_process0)
2. When we decide to prerender (in a separate API call), we destroy this spare renderer (sandboxed_process0) as it cannot be used for prerendering, and we don't want 2 background renderers, only one of which can be used later. We create a renderer for prerender (sandboxed_process1)
3. If the prerender is a miss, destroy the prerendering renderer (sandboxed_process1 cannot be reused), create a new one (sandboxed_process2).

To confirm that:
1. Log the Custom Tabs requests
$ adb shell "echo '_ --custom-tabs-log-service-requests' > /data/local/tmp/chrome-command-line"

2. Kill Chrome and GSA, and start a search:
$ adb shell am force-stop com.google.android.apps.chrome
$ adb shell am force-stop com.google.android.googlequicksearchbox:search

3. Log CustomTabs and ActivityManager
$ adb logcat | egrep 'cr_|ActivityManager'

4. Do a search ("hello"), open a link. Part of logcat:

---- SEARCH WITHOUT PRERENDERING ----

[GSA Starts]
05-12 10:38:14.874   871  3269 I ActivityManager: START u0 {act=android.intent.action.MAIN dat=agsa:/session?id=1183176887307 flg=0x1400a000 cmp=com.google.android.googlequicksearchbox/com.google.android.apps.gsa.legacyui.VelvetActivity (has extras)} from uid 10032 on display 0
05-12 10:38:15.027   871   986 I ActivityManager: Displayed com.google.android.googlequicksearchbox/com.google.android.apps.gsa.legacyui.VelvetActivity: +137ms

[GSA Binds to Chrome]
05-12 10:38:16.107   871  5686 I ActivityManager: Start proc 13425:com.google.android.apps.chrome/u0a265 for service com.google.android.apps.chrome/org.chromium.chrome.browser.customtabs.CustomTabsConnectionService
05-12 10:38:16.391 13425 13425 W cr_ChromeConnection: Service#onBind() = true, Calling UID = 10265

[CustomTabs warmup(), newSession(). Initializes the browser process, creates a renderer and the GPU process]
05-12 10:38:16.399 13425 13438 W cr_ChromeConnection: warmup() = true, Calling UID = 10032
05-12 10:38:16.401 13425 13437 W cr_ChromeConnection: newSession() = true, Calling UID = 10032
[Browser Process]
05-12 10:38:16.429 13425 13425 I cr_LibraryLoader: Using linker: org.chromium.base.library_loader.ModernLinker
05-12 10:38:16.456 13425 13425 I cr_LibraryLoader: Loading chrome
05-12 10:38:16.562 13425 13425 I cr_LibraryLoader: Time to load native libraries: 133 ms (timestamps 284-417)
05-12 10:38:16.563 13425 13425 I cr_LibraryLoader: Expected native library version number "60.0.3097.0", actual native library version number "60.0.3097.0"
05-12 10:38:16.563 13425 13425 I cr_BrowserStartup: Initializing chromium process, singleProcess=false
05-12 10:38:16.930 13425 13466 W cr_ChildProcLauncher: Create a new ChildConnectionAllocator with package name = com.google.android.apps.chrome, inSandbox = true

[Spare renderer]
05-12 10:38:16.946   871  9392 I ActivityManager: Start proc 13495:com.google.android.apps.chrome:sandboxed_process0/u0i20 for service com.google.android.apps.chrome/org.chromium.content.app.SandboxedProcessService0
05-12 10:38:17.785 13495 13495 I cr_ChildProcessService: Creating new ChildProcessService pid=13495
05-12 10:38:17.799 13495 13508 I cr_LibraryLoader: Using linker: org.chromium.base.library_loader.ModernLinker
05-12 10:38:17.826 13495 13518 I cr_LibraryLoader: Loading chrome
05-12 10:38:17.864 13495 13518 I cr_LibraryLoader: Time to load native libraries: 56 ms (timestamps 1663-1719)
05-12 10:38:17.866 13495 13518 I cr_LibraryLoader: Expected native library version number "60.0.3097.0", actual native library version number "60.0.3097.0"

[GPU process]
05-12 10:38:17.984   871  3575 I ActivityManager: Start proc 13529:com.google.android.apps.chrome:privileged_process0/u0a265 for service com.google.android.apps.chrome/org.chromium.content.app.PrivilegedProcessService0
05-12 10:38:18.288 13529 13529 I cr_ChildProcessService: Creating new ChildProcessService pid=13529
05-12 10:38:18.320 13529 13542 I cr_LibraryLoader: Using linker: org.chromium.base.library_loader.ModernLinker
05-12 10:38:18.353 13529 13546 I cr_LibraryLoader: Loading chrome
05-12 10:38:18.395 13529 13546 I cr_LibraryLoader: Time to load native libraries: 73 ms (timestamps 2176-2249)
05-12 10:38:18.396 13529 13546 I cr_LibraryLoader: Expected native library version number "60.0.3097.0", actual native library version number "60.0.3097.0"

[VIEW Intent in GSA]
05-12 10:38:22.207   871  4081 I ActivityManager: START u0 {act=android.intent.action.VIEW cat=[android.intent.category.BROWSABLE] dat=http://www.hellomagazine.com/... flg=0x18081000 pkg=com.google.android.googlequicksearchbox cmp=com.google.android.googlequicksearchbox/com.google.android.apps.gsa.cctshim.ChromeplateShim (has extras)} from uid 10032 on display 0

[VIEW Intent in Chrome]
05-12 10:38:22.265   871  5685 I ActivityManager: START u0 {act=android.intent.action.VIEW cat=[android.intent.category.BROWSABLE] dat=http://www.hellomagazine.com/... pkg=com.google.android.googlequicksearchbox cmp=com.google.android.apps.chrome/.Main (has extras)} from uid 10032 on display 0
05-12 10:38:22.350   871  5686 I ActivityManager: START u0 {act=android.intent.action.VIEW cat=[android.intent.category.BROWSABLE] dat=http://www.hellomagazine.com/... flg=0x800000 pkg=com.google.android.googlequicksearchbox cmp=com.google.android.apps.chrome/org.chromium.chrome.browser.customtabs.CustomTabActivity (has extras)} from uid 10265 on display 0
05-12 10:38:22.478   871   986 I ActivityManager: Displayed com.google.android.apps.chrome/org.chromium.chrome.browser.customtabs.CustomTabActivity: +116ms (total +258ms)


And here are the chrome processes:
$ adb shell dumpsys meminfo | grep com.google.android.apps.chrome
    107,153K: com.google.android.apps.chrome (pid 13425 / activities)
     95,141K: com.google.android.apps.chrome:sandboxed_process0 (pid 13495)
     53,463K: com.google.android.apps.chrome:privileged_process0 (pid 13529)


--- SEARCH WITH PRERENDERING ---
- Search for "Trump" (need to trigger an AMP result)
- logcat (more heavily edited)

[GSA]

05-12 10:46:19.205   871   986 I ActivityManager: Displayed com.google.android.googlequicksearchbox/com.google.android.apps.gsa.legacyui.VelvetActivity: +128ms

[Bind, warmup(), newSession()]
05-12 10:46:20.218   871  3606 I ActivityManager: Start proc 14597:com.google.android.apps.chrome/u0a265 for service com.google.android.apps.chrome/org.chromium.chrome.browser.customtabs.CustomTabsConnectionService
05-12 10:46:20.545 14597 14597 W cr_ChromeConnection: Service#onBind() = true, Calling UID = 10265
05-12 10:46:20.550 14597 14612 W cr_ChromeConnection: warmup() = true, Calling UID = 10032
05-12 10:46:20.553 14597 14612 W cr_ChromeConnection: newSession() = true, Calling UID = 10032
05-12 10:46:21.087   871  5654 I ActivityManager: Start proc 14658:com.google.android.apps.chrome:sandboxed_process0/u0i24 for service com.google.android.apps.chrome/org.chromium.content.app.SandboxedProcessService0

[mayLaunchUrl()]
05-12 10:46:21.872 14597 14612 W cr_ChromeConnection: mayLaunchUrl() = true, Calling UID = 10032
[New process for prerender]
05-12 10:46:21.932   871  3606 I ActivityManager: Start proc 14691:com.google.android.apps.chrome:sandboxed_process1/u0i25 for service com.google.android.apps.chrome/org.chromium.content.app.SandboxedProcessService1
[When prerendering, kill the spare renderer, as we don't want to use too many resources]
05-12 10:46:22.587   871  5685 I ActivityManager: Process com.google.android.apps.chrome:sandboxed_process0 (pid 14658) has died
[GPU process]
05-12 10:46:23.406   871  5685 I ActivityManager: Start proc 14737:com.google.android.apps.chrome:privileged_process0/u0a265 for service com.google.android.apps.chrome/org.chromium.content.app.PrivilegedProcessService0

[Start on a URL that hasn't been prerendered]
05-12 10:46:45.097   871  3606 I ActivityManager: START u0 {act=android.intent.action.VIEW cat=[android.intent.category.BROWSABLE] dat=https://www.theguardian.com/... flg=0x18081000 pkg=com.google.android.googlequicksearchbox cmp=com.google.android.googlequicksearchbox/com.google.android.apps.gsa.cctshim.ChromeplateShim (has extras)} from uid 10032 on display 0
05-12 10:46:45.151   871  4081 I ActivityManager: START u0 {act=android.intent.action.VIEW cat=[android.intent.category.BROWSABLE] dat=https://www.theguardian.com/... pkg=com.google.android.googlequicksearchbox cmp=com.google.android.apps.chrome/.Main (has extras)} from uid 10032 on display 0

05-12 10:46:45.227   871  3170 I ActivityManager: START u0 {act=android.intent.action.VIEW cat=[android.intent.category.BROWSABLE] dat=https://www.theguardian.com/... flg=0x800000 pkg=com.google.android.googlequicksearchbox cmp=com.google.android.apps.chrome/org.chromium.chrome.browser.customtabs.CustomTabActivity (has extras)} from uid 10265 on display 0

[Prerender miss -> create a new renderer, destroy the prerendering one]
05-12 10:46:45.280   871  5686 I ActivityManager: Start proc 14859:com.google.android.apps.chrome:sandboxed_process2/u0i26 for service com.google.android.apps.chrome/org.chromium.content.app.SandboxedProcessService2
05-12 10:46:45.361   871  9392 I ActivityManager: Process com.google.android.apps.chrome:sandboxed_process1 (pid 14691) has died

05-12 10:46:45.431   871   986 I ActivityManager: Displayed com.google.android.apps.chrome/org.chromium.chrome.browser.customtabs.CustomTabActivity: +193ms (total +325ms)

$ adb shell dumpsys meminfo | grep com.google.android.apps.chrome
    101,769K: com.google.android.apps.chrome (pid 14597 / activities)
     66,494K: com.google.android.apps.chrome:sandboxed_process2 (pid 14859)
     55,554K: com.google.android.apps.chrome:privileged_process0 (pid 14737)

Thanks for the investigation. Why we can't use spare renderer for prerenderer?

The spare renderer is created with a special flag in the (java) WebContentsFactory. It is then given at Tab initialization.

On the other hand, PrerenderManager (C++) creates a (C++) WebContents itself, with a bunch of special cases, as it has to (potentially) insert itself into an existing navigation flow.

It is all solvable, for instance:
- Create a spare RenderFrameHost instead of a WebContents: there was a patch doing that, regressed the Custom Tabs use case, although it made the spare renderer more general (especially with PlzNavigate).
- Change the PrerenderManager implementation

The second option was not explored in details as prerender is going away. We may decide to land the first option, as the patch is not really invasive.

However, the current situation is a bit uncomfortable performance-wise, as we are basically giving up on the performance benefits of the spare renderer when we have a prerendering miss (which is the majority of prerenders for Custom Tabs).

This is not easily solvable unless we accept to have two background renderers: (1) a prerendering one, and (2) a spare one.

In case of a prerendering miss, we cannot reuse the renderer that has just been used for prerendering, as it will most likely have navigated to a different origin than the URL we want to navigate to.

The simplest alternative is then, at least on high-end devices:
1. Create a spare renderer
2. When a prerender request comes in, don't kill the spare renderer
3. When the intent arrives:
   - Prerender hit: kill the spare renderer, use the prerender
   - Prerender miss: kill the prerender, use the spare renderer

Extra memory cost compared to prerendering is the cost of a spare renderer, that is a few MB of private dirty. Performance gains are ~200ms on a Nexus 5-like device (haven't measured this exact configuration). The majority of prerenders are never matched, see this UMA (CANCELLED vs USED):

https://uma.googleplex.com/p/chrome/histograms/?endDate=20170514&dayCount=7&histograms=CustomTabs.PredictionStatus%2CPrerender.externalrequestforced_FinalStatus&fixupData=true&showMax=true&filters=platform%2Ceq%2CA%2Cchannel%2Ceq%2C4%2Cisofficial%2Ceq%2CTrue&implicitFilters=isofficial

What do you think?

With PlzNavigate, there should be not much need for the spare renderer as the network request can be issued much earlier.

Is there any UMA showing how much spare renderer save the loading time for incoming intent?