I can't get a good signal locally. I tried reproducing with the range above, but I have a few dozen tests timing out or having failures, no matter what.

I did:
ninja -j1000 -C out/release blink_tests
python third_party/blink/tools/run_web_tests.py -t release --exit-after-n-failures 100 --exit-after-n-crashes-or-timeouts 10

I tested both ends of the range mentioned in the description and also tried with different V8 versions.

@John: Are layout tests supposed to run well locally? Do you know who could help if I have some sort of setup problem?

My suspicion is that the V8 downstream builders are red because they don't use swarming. That's the only way our testing actually deviates from Chromium. In this case, some test ordering problem might be the cause.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/48e24ff68bc44b1b403db6a47c05e316f1c3d01f

commit 48e24ff68bc44b1b403db6a47c05e316f1c3d01f
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu Jan 10 14:30:00 2019

[V8] Temporarily double the timeout on blink downstream bots

Bug: 920545
Change-Id: Ia30a5029330fd32bb9b61dd7ae3b0885de49f8af
Reviewed-on: https://chromium-review.googlesource.com/c/1404620
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/blink_web_tests_unexpected_error.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.py
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/blink_web_tests_interrupted.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/minimal_pass_continues.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/too_many_failures_for_retcode.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_pass.json

I believe they should, but I also think that the blink downstream bots are the only bots still doing so. If you've got further questions about the layout tests, there's an internal list you can ask.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/afd71a485be80a116eb2d6142015189c1658d826

commit afd71a485be80a116eb2d6142015189c1658d826
Author: Michael Achenbach <machenbach@chromium.org>
Date: Fri Jan 11 10:36:34 2019

[V8] Temporarily remove blink builder from lkgr

TBR=sergiyb@chromium.org

Bug: 920545
Change-Id: I42a85f1b10f7fefb93df1f06055e1779534bb0af
Reviewed-on: https://chromium-review.googlesource.com/c/1406690
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19927}
[modify] https://crrev.com/afd71a485be80a116eb2d6142015189c1658d826/infra/services/lkgr_finder/config/v8_cfg.pyl

Assign to tmrts for bisicting the original culprit. Meanwhile, I will work on the blocking bug to migrate this to swarming.

Chromium revision 3ffc0d0376652a1e196513a959ebb8a8b9160dd4(https://chromium-review.googlesource.com/c/chromium/src/+/1401449) is failing the V8 layout tests. CC @ qyearsley@chromium.org, robertma@chromium.org

Bisected the failing revisions.

Here's the last passing revision: https://ci.chromium.org/swarming/task/426f595978737a10?server=chromium-swarm.appspot.com

Here's the first failing revision:
https://ci.chromium.org/swarming/task/426f5a5bab974a10?server=chromium-swarm.appspot.com

By V8 not using isolated script tests, but BlinkTests (https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?l=2208), something might have gotten incompatible after the change mentioned in comment 9.

@Quinten, Robert: Any idea what that might be? My suspicion is that isolated script tests do some more setup that we miss, or pass a flag, or it's because it's running in a different pool.

In the mean time I will continue to migrate the builders to swarming ( issue 828847 ). Once that's done, this analysis here might be obsolete and V8 will stop using BlinkTests.

Re 10: Another suspicion is some test order dependency that doesn't show up on swarming, but when running all tests sequentially.

For reference, here's the patch I've used to run these old builds:
---
 scripts/slave/recipes/blink_downstream.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/slave/recipes/blink_downstream.py b/scripts/slave/recipes/blink_downstream.py
index b30e919afb..6e3d74a0bf 100644
--- a/scripts/slave/recipes/blink_downstream.py
+++ b/scripts/slave/recipes/blink_downstream.py
@@ -165,9 +165,10 @@ def RunSteps(api):
   api.gclient.c.revisions[bot_config['component']['path']] = (
       bot_config['component']['revision'] % component_revision)
 
+  api.gclient.c.revisions['src'] = 'f8119b9be8440b231a19c0b4019310cfa8065433'
   # Ensure we remember the chromium revision.
   api.gclient.c.got_revision_reverse_mapping['got_cr_revision'] = 'src'
-  api.gclient.c.got_revision_mapping.pop('src', None)
+  #api.gclient.c.got_revision_mapping.pop('src', None)
 
   # Run all steps in the checkout dir (consistent with chromium_tests).
   with api.context(cwd=api.chromium_checkout.get_checkout_dir(bot_config)):
@@ -185,8 +186,8 @@ def RunSteps(api):
 
     def component_pinned_fn(_failing_steps):
       bot_update_json = step_result.json.output
-      api.gclient.c.revisions['src'] = str(
-          bot_update_json['properties']['got_cr_revision'])
+      # api.gclient.c.revisions['src'] = str(
+      #     bot_update_json['properties']['got_cr_revision'])
       # Reset component revision to the pinned revision from chromium's DEPS
       # for comparison.
       del api.gclient.c.revisions[bot_config['component']['path']]
-- 
2.20.1.97.g81188d93c3-goog

There are tons of "Error: unrecognized flag --experimental_extras". Yet a code search shows no hit of "experimental_extras". Where's this from?

https://crrev.com/c/1401449 only changed wptserve, which is only used in external/wpt, so it doesn't explain the massive failures outside of the directory. And the log messages related to servers look normal.

This error seems to occur only on our non-swarming bots, but it occurs equally often in the green build. So I guess it's benign.

We fixed  issue 828847  in the meantime and run layout tests on swarming now. Also the error described here is gone.

I nobody wants to dig deeper here, I suggest we close this.

Re 13: "it doesn't explain the massive failures outside of the directory". The testing in comment 9 was done by running once before the commit and once with the commit, keeping all other things stable. This shows that the behavior we've seen starts with that commit. But I've too little expertise with layout test infra to do any further reasoning.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d4986b69f6ab184b6d922bcaefea2682f61c4465

commit d4986b69f6ab184b6d922bcaefea2682f61c4465
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu Jan 17 14:01:58 2019

Revert "[V8] Temporarily remove blink builder from lkgr"

This reverts commit afd71a485be80a116eb2d6142015189c1658d826.

Reason for revert: Builders are fixed and back to normal.

Original change's description:
> [V8] Temporarily remove blink builder from lkgr
> 
> TBR=sergiyb@chromium.org
> 
> Bug: 920545
> Change-Id: I42a85f1b10f7fefb93df1f06055e1779534bb0af
> Reviewed-on: https://chromium-review.googlesource.com/c/1406690
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#19927}

TBR=machenbach@chromium.org,sergiyb@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 920545
Change-Id: I36ba458bdc843ae96ff5215e304c6fa5404cd6ed
Reviewed-on: https://chromium-review.googlesource.com/c/1417650
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20051}
[modify] https://crrev.com/d4986b69f6ab184b6d922bcaefea2682f61c4465/infra/services/lkgr_finder/config/v8_cfg.pyl