this blocks rolling

Interesting line from the logs: "Please use third_party/blink/tools/run_web_tests.*.  This command will be removed.". Looks like we need to update the recipe.

This repros when adding the --no-pixel-tests flag:
third_party/blink/tools/run_web_tests.py -t Release css1/* --no-pixel-tests

The downstream bots were using this flag. I'll remove it as a quick-fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64

commit ff1ca53a9b3ef2314a24bed44c2a0b970417ba64
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu Apr 26 11:17:11 2018

[V8] Fix broken blink_downstream builders

Passing the --no-pixel-tests makes things fail. The downstream builders seem
to be the only ones using this flag.

TBR=sergiyb@chromium.org

Bug: 837160
Change-Id: Ie0aee8b0348be2dfcf264399c9d491c523026f9a
Reviewed-on: https://chromium-review.googlesource.com/1029868
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>

[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.py
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/webkit_tests_interrupted.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__pass.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_pass.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/README.recipes.md
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_fail.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/minimal_pass_continues.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/webkit_tests_unexpected_error.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/too_many_failures_for_retcode.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_pass.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_pass.json
[modify] https://crrev.com/ff1ca53a9b3ef2314a24bed44c2a0b970417ba64/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_pass.json

After stopping using --no-pixel-tests there are still lots of timeout errors left. Now I investigate if they might have been introduced by a V8 change in the mean time...

The remaining problems don't even repro when logged onto the bots remotely :(

Pushing the lkgr manually in the meantime to keep rolling.

The bots auto-magically greenified.

@robertma: Was this maybe related to recent blink test runner changes?

The revision range when this broke:
https://chromium.googlesource.com/chromium/src/+log/280a408413bcff..1d797a099c8c7

The revision range when it got green again:
https://chromium.googlesource.com/chromium/src/+log/f964cedb43a0f8682f2a..1565765370b4193f2

In both ranges are blinkpy changes. Was the remaining problem related to test order? When I tried to repro on the bot, I always only ran a few tests, not all, which obviously led to different orders.

Are there more such changes ongoing? Can we do anything to make our bots more resilient? Should we switch them to swarming?

Re #10: it was probably related to my change https://crrev.com/805c043e94a3b6dcc47e52530575c53f025632aa , which also broke Chromium Windows waterfall bots. Sorry.

The ranges you quoted are slightly off.

The last actual failure is https://build.chromium.org/p/client.v8.fyi/builders/V8-Blink%20Linux%2064/builds/23187 (f4215859085f9e6bcb8de36b7fdae103ac9defd3).
The next build is almost green (the few failures were probably flakes): https://build.chromium.org/p/client.v8.fyi/builders/V8-Blink%20Linux%2064/builds/23188 (f964cedb43a0f8682f2ab1facdf35c03f26affa2)

And in that range I reverted my culprit CL: https://chromium.googlesource.com/chromium/src/+log/f4215859085f9e6bcb8de36b7fdae103ac9defd3..f964cedb43a0f8682f2ab1facdf35c03f26affa2


-------------------


I believe the original failure was caused by the renaming of run-webkit-tests, which is part of the recent Blink mv ( issue 829697 ).

The next failure seems related to "--no-pixel-tests". This flag might be indeed broken. I don't recall anywhere else using this flag, and the related code hasn't been touched for a while. It might be accidentally broken by the renaming, or there might be accidental interdependencies between tests that are exposed by the flag. I can reproduce the problem locally, but I still need to investigate more.

And the last failure was caused by me...

In terms of making the bots more resilient, the key suggestion for you is to use the same command (including the path and flags) as in Chromium. I don't think Swarming would make a difference. However, I imagine V8 is in a different repo and you can't directly reference the recipes in Chromium, so I don't know how. 

On our side (Blink Infra), now that I know V8 also uses these scripts, I'll double check when I work on or review CLs that change the interface (previously we mostly only search in Chromium).  Besides, is there an equivalent try bot of V8-Blink in Chromium that we can trigger to make sure we don't break yours?

This is actually a Chromium bot. It should run layout tests just like any other layout test bot (despite not being on swarming).

The difference on this bot is:
It works similar to blink trybots. It first uses Chromium with V8 at ToT and runs all tests. On failures, it uses V8 at the pinned DEPS revision and repeats the failures. Like that we know when V8 commits are going to break layout tests, while hiding downstream noise (mostly).

After removing the pixel test flag, the only remaining difference is an extra expectation file, but it's nearly empty. Due to this and no swarming, I believe the bot runs tests in a very different order.

There is also a small chance that the bots lacks some dependencies maybe? I.e. things set up by puppet...

-----

My original reason or adding --no-pixel-tests was to reduce noise of the bots (since V8 is very unlikely to break that). I thought it's for omitting the pixel-by-pixel comparison with expectation pictures. But it seems to do other things then instead that are broken.