New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Closed: Jan 25
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 828847
issue 920357



Sign in to add a comment
link

Issue 920545: Blink downstream bots are red

Reported by machenbach@google.com, Jan 10 Project Member

Issue description

Red on all platforms. Started at different commits on each platform, e.g.:
https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8-Blink%20Linux%2064/29096

The smallest common range of chromium changes is:
https://chromium.googlesource.com/chromium/src/+log/f8119b9be8440b231a19c0b40..3b58524dfeb57
 

Comment 1 by machenbach@google.com, Jan 10

Owner: machenb...@chromium.org
Status: Started (was: Untriaged)

Comment 2 by machenbach@google.com, Jan 10

Blockedon: 920357

Comment 3 by machenbach@google.com, Jan 10

Cc: jbudorick@google.com yangguo@chromium.org
I can't get a good signal locally. I tried reproducing with the range above, but I have a few dozen tests timing out or having failures, no matter what.

I did:
ninja -j1000 -C out/release blink_tests
python third_party/blink/tools/run_web_tests.py -t release --exit-after-n-failures 100 --exit-after-n-crashes-or-timeouts 10

I tested both ends of the range mentioned in the description and also tried with different V8 versions.

@John: Are layout tests supposed to run well locally? Do you know who could help if I have some sort of setup problem?

My suspicion is that the V8 downstream builders are red because they don't use swarming. That's the only way our testing actually deviates from Chromium. In this case, some test ordering problem might be the cause.

Comment 4 by bugdroid1@chromium.org, Jan 10

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/48e24ff68bc44b1b403db6a47c05e316f1c3d01f

commit 48e24ff68bc44b1b403db6a47c05e316f1c3d01f
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu Jan 10 14:30:00 2019

[V8] Temporarily double the timeout on blink downstream bots

Bug:  920545 
Change-Id: Ia30a5029330fd32bb9b61dd7ae3b0885de49f8af
Reviewed-on: https://chromium-review.googlesource.com/c/1404620
Reviewed-by: Sergiy Belozorov <sergiyb@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/blink_web_tests_unexpected_error.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.py
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/blink_web_tests_interrupted.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/minimal_pass_continues.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/too_many_failures_for_retcode.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_pass.json
[modify] https://crrev.com/48e24ff68bc44b1b403db6a47c05e316f1c3d01f/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_pass.json

Comment 5 by jbudorick@chromium.org, Jan 10

Cc: -jbudorick@google.com jbudorick@chromium.org
I believe they should, but I also think that the blink downstream bots are the only bots still doing so. If you've got further questions about the layout tests, there's an internal list you can ask.

Comment 6 by machenbach@google.com, Jan 11

Blockedon: 828847

Comment 7 by bugdroid1@chromium.org, Jan 11

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/afd71a485be80a116eb2d6142015189c1658d826

commit afd71a485be80a116eb2d6142015189c1658d826
Author: Michael Achenbach <machenbach@chromium.org>
Date: Fri Jan 11 10:36:34 2019

[V8] Temporarily remove blink builder from lkgr

TBR=sergiyb@chromium.org

Bug:  920545 
Change-Id: I42a85f1b10f7fefb93df1f06055e1779534bb0af
Reviewed-on: https://chromium-review.googlesource.com/c/1406690
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19927}
[modify] https://crrev.com/afd71a485be80a116eb2d6142015189c1658d826/infra/services/lkgr_finder/config/v8_cfg.pyl

Comment 8 by machenbach@google.com, Jan 16

Owner: tm...@chromium.org
Assign to tmrts for bisicting the original culprit. Meanwhile, I will work on the blocking bug to migrate this to swarming.

Comment 10 by machenbach@google.com, Jan 16

By V8 not using isolated script tests, but BlinkTests (https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?l=2208), something might have gotten incompatible after the change mentioned in comment 9.

@Quinten, Robert: Any idea what that might be? My suspicion is that isolated script tests do some more setup that we miss, or pass a flag, or it's because it's running in a different pool.

In the mean time I will continue to migrate the builders to swarming ( issue 828847 ). Once that's done, this analysis here might be obsolete and V8 will stop using BlinkTests.

Comment 11 by machenbach@google.com, Jan 16

Re 10: Another suspicion is some test order dependency that doesn't show up on swarming, but when running all tests sequentially.

Comment 12 by tmrts@google.com, Jan 16

For reference, here's the patch I've used to run these old builds:
---
 scripts/slave/recipes/blink_downstream.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/slave/recipes/blink_downstream.py b/scripts/slave/recipes/blink_downstream.py
index b30e919afb..6e3d74a0bf 100644
--- a/scripts/slave/recipes/blink_downstream.py
+++ b/scripts/slave/recipes/blink_downstream.py
@@ -165,9 +165,10 @@ def RunSteps(api):
   api.gclient.c.revisions[bot_config['component']['path']] = (
       bot_config['component']['revision'] % component_revision)
 
+  api.gclient.c.revisions['src'] = 'f8119b9be8440b231a19c0b4019310cfa8065433'
   # Ensure we remember the chromium revision.
   api.gclient.c.got_revision_reverse_mapping['got_cr_revision'] = 'src'
-  api.gclient.c.got_revision_mapping.pop('src', None)
+  #api.gclient.c.got_revision_mapping.pop('src', None)
 
   # Run all steps in the checkout dir (consistent with chromium_tests).
   with api.context(cwd=api.chromium_checkout.get_checkout_dir(bot_config)):
@@ -185,8 +186,8 @@ def RunSteps(api):
 
     def component_pinned_fn(_failing_steps):
       bot_update_json = step_result.json.output
-      api.gclient.c.revisions['src'] = str(
-          bot_update_json['properties']['got_cr_revision'])
+      # api.gclient.c.revisions['src'] = str(
+      #     bot_update_json['properties']['got_cr_revision'])
       # Reset component revision to the pinned revision from chromium's DEPS
       # for comparison.
       del api.gclient.c.revisions[bot_config['component']['path']]
-- 
2.20.1.97.g81188d93c3-goog

Comment 13 by robertma@chromium.org, Jan 16

There are tons of "Error: unrecognized flag --experimental_extras". Yet a code search shows no hit of "experimental_extras". Where's this from?

https://crrev.com/c/1401449 only changed wptserve, which is only used in external/wpt, so it doesn't explain the massive failures outside of the directory. And the log messages related to servers look normal.

Comment 14 by machenbach@google.com, Jan 17

This error seems to occur only on our non-swarming bots, but it occurs equally often in the green build. So I guess it's benign.

Comment 15 by machenbach@google.com, Jan 17

Labels: -Pri-1 Pri-2
We fixed  issue 828847  in the meantime and run layout tests on swarming now. Also the error described here is gone.

I nobody wants to dig deeper here, I suggest we close this.

Re 13: "it doesn't explain the massive failures outside of the directory". The testing in comment 9 was done by running once before the commit and once with the commit, keeping all other things stable. This shows that the behavior we've seen starts with that commit. But I've too little expertise with layout test infra to do any further reasoning.

Comment 16 by bugdroid1@chromium.org, Jan 17

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d4986b69f6ab184b6d922bcaefea2682f61c4465

commit d4986b69f6ab184b6d922bcaefea2682f61c4465
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu Jan 17 14:01:58 2019

Revert "[V8] Temporarily remove blink builder from lkgr"

This reverts commit afd71a485be80a116eb2d6142015189c1658d826.

Reason for revert: Builders are fixed and back to normal.

Original change's description:
> [V8] Temporarily remove blink builder from lkgr
> 
> TBR=sergiyb@chromium.org
> 
> Bug:  920545 
> Change-Id: I42a85f1b10f7fefb93df1f06055e1779534bb0af
> Reviewed-on: https://chromium-review.googlesource.com/c/1406690
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#19927}

TBR=machenbach@chromium.org,sergiyb@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  920545 
Change-Id: I36ba458bdc843ae96ff5215e304c6fa5404cd6ed
Reviewed-on: https://chromium-review.googlesource.com/c/1417650
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20051}
[modify] https://crrev.com/d4986b69f6ab184b6d922bcaefea2682f61c4465/infra/services/lkgr_finder/config/v8_cfg.pyl

Comment 17 by machenb...@chromium.org, Jan 25

Status: WontFix (was: Started)
Lets close this. After switching to swarming, analyzing this further is obsolete for us.

Sign in to add a comment