Cc-ing benchmark owners. Will disable this story on Windows 10.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14f5030bc40000

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4f5bfb44ed260127602cfc4994865a80fd5d0eb4

commit 4f5bfb44ed260127602cfc4994865a80fd5d0eb4
Author: Annie Sullivan <sullivan@chromium.org>
Date: Thu May 03 17:25:03 2018

Disable v8.browsing_desktop-future/browse:social:twitter_infinite_scroll on Win.

It is failing flakily.

NOTRY=true
TBR=nednguyen@google.com

Bug: 839470
Change-Id: Ia5657133812ffa541dca0ff0c72db819ee7163ea
Reviewed-on: https://chromium-review.googlesource.com/1042271
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Annie Sullivan <sullivan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555783}
[modify] https://crrev.com/4f5bfb44ed260127602cfc4994865a80fd5d0eb4/tools/perf/expectations.config

This is also failing on Win7 Nvidia, and on v8.browsing_desktop (non-future version)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a5f2f51e43e86c7863595be6e0198b43f550b354

commit a5f2f51e43e86c7863595be6e0198b43f550b354
Author: Annie Sullivan <sullivan@chromium.org>
Date: Thu May 03 17:44:57 2018

Disable v8.browsing_desktop/browse:social:twitter_infinite_scroll on Win.

TBR=nednguyen@google.com
NOTRY=true

Bug: 839470
Change-Id: I7010d53bfdc84b9ac540735df8be060c9e5c2735
Reviewed-on: https://chromium-review.googlesource.com/1042279
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Annie Sullivan <sullivan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555798}
[modify] https://crrev.com/a5f2f51e43e86c7863595be6e0198b43f550b354/tools/perf/expectations.config

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/14f5030bc40000

Make --site-per-process the default on ToT via fieldtrial_testing_config by lukasza@chromium.org
https://chromium.googlesource.com/chromium/src/+/fb1ccf02ee8ca79e1404abfd3a3a7d540b7d2dbd

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

dtu: I wanted to pull the logs on the failures to see why the success rate seems to go from 90% to 60% at that CL, but it actually looks like all the failures are swarming task failures and not failures caused by the CL? Can you take a look?

In this case, the Swarming task failures are test failures. So yes, the failures are caused by this CL.

Thanks, Dave! Assigning back to lukasza, here's more information on what's going on:

After your CL, the test got flaky on Windows 10. You can see that on flakiness dashboard: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=v8.browsing_desktop-future&builder=chromium.perf%3AWin%2010%20Perf

So we used pinpoint to bisect, and it narrows down flakiness to your CL (the chart is the success rate of the test): https://pinpoint-dot-chromeperf.appspot.com/job/14f5030bc40000

A very similar thing is happening on  bug 839411 , but the bisect hasn't completed yet.

This might be tricky to debug, because it doesn't seem to happen on all hardware; I think all the machines it's reproducing on are Dell PowerEdge r220s. If you need to reproduce and it's not failing locally we can try to pull one of the machines out of our swarming pool to rdesktop into.

The command line for the test is:
tools/perf/run_benchmark v8.browsing_desktop-future --story-filter browse.social.twitter.infinite.scroll

This is a performance integration test, it tries to open the browser, load a recorded version of twitter, scroll, and collect performance metrics. 

I posted a screenshot showing how to access the logs from pinpoint, but they're not super helpful. They say the test timed out, but this can happen when the browser crashes.

See go/reproduce-telemetry for more documentation.

Deleted: pinpoint.png 176 KB

	pinpoint.png 176 KB View Download

Thanks Annie for the detailed steps, I just had a quick look at the logs. It is timing out waiting for a javascript condition when scrolling[1] in all the traces. So, it could be also that some resource is not being loaded with site-isolation? That still doesn't rule out browser crash though.

[1]https://cs.chromium.org/chromium/src/tools/perf/page_sets/system_health/browsing_stories.py?q=system_health/b&sq=package:chromium&l=855

RE: it could be also that some resource is not being loaded with site-isolation

Is it possible to check DevTools console for messages similar to:
Blocked current origin from receiving cross-site document at <URL> with MIME type <text/html, text/json, etc.>.

Alternatively - is it possible to look at the UMA data for SiteIsolation.XSD.Browser.Action and/or SiteIsolation.XSD.Browser.Blocked?

If the message or the UMA are present, it might indicate that Cross-Origin Read Blocking [1] blocks one of the responses.

[1] https://chromium.googlesource.com/chromium/src/+/master/services/network/cross_origin_read_blocking_explainer.md

 Issue 839411  has been merged into this issue.

Also - is it possible to check for browser crashes (and possibly correlate them to specific Crash reports)?  (#c12 suggests that a browser crash might be one explanation for the observed timeouts of the benchmark)

I looked at the devtools console output (on linux, though the bug is related to windows. I thought the platform shouldn't matter much for checking this). There are no messages related to cross-site document being blocked. This bug is for an internal benchmark. These benchmarks use a recorded web page. I am not sure UMA data would help here?

We get browser crash logs in the pinpoint jobs (#c12 explains how to look for them), but in this case there is nothing in the log. I am not sure if there are other ways of looking for crash logs.

Hmmm... is it possible that Site Isolation affects the timing of various things (e.g. by allowing cross-site frames to load in parallel in their separate renderers) and this confuses the "driver" of the test (e.g. so that it simulates clicks, reloads or other interactions at not quite the right time)?  This theory seems compatible with the observed flakiness (e.g. sometimes even synchronizing via sleep statements will luckily get the right timing).  OTOH, I don't know how one could verify this theory... :-(

1/4 failed test tasks I looked at in https://pinpoint-dot-chromeperf.appspot.com/job/14f5030bc40000 contains a message about a minidump (https://chromium-swarm.appspot.com/task?id=3d40ab456f1ffb10&refresh=10&show_raw=1).  Looking at the minidump points at content::protocol::SystemInfoHandlerGpuObserver::ObserverWatchdogCallback (so possible a dupe of  issue 842782 , similarily to what we've together found in  https://crbug.com/842180#c10  -  https://crbug.com/842180#c17 ).

I am not sure if this is good news (we have found the root cause) or bad news (3/4 of the failed test runs didn't say anything about a minidump, so maybe there are more unknown failure causes hiding here)...  FWIW, the 3 failed tasks I looked at that didn't contain minidumps are the ones below:
- https://chromium-swarm.appspot.com/task?id=3d40ab41a73c0910&refresh=10&show_raw=1
- https://chromium-swarm.appspot.com/task?id=3d40ab42f00f6d10&refresh=10&show_raw=1
- https://chromium-swarm.appspot.com/task?id=3d40ab448f081510&refresh=10&show_raw=1