New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 721700 link

Starred by 2 users
Status: Duplicate
Merged: issue 524758
Owner: ----
Closed: May 2017
Cc:
█ tansell@chromium.org
qyears...@chromium.org
dpranke@chromium.org
mar...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 720511



Sign in to add a comment

Layout tests should run on swarming everywhere or nowhere

Project Member Reported by machenb...@chromium.org, May 12 2017 
Right now only chromium CQ seems to run layout tests on swarming. The main waterfall and the blink trybots don't run it on swarming.

According to findings in  issue 720511  the test results are different on CQ and CI making it very hard to track down where exactly regressions started. Some of the comments also suggest a test order problem that might be different on swarming shards compared to CI.

Due to  issue 720623 , v8 and skia don't run the webkit tests from chromium_linux_rel but use linux_trusty_blink_rel explicitly. The latter doesn't use swarming and seems to give a different signal than the former, making it possible for skia and v8 to break the chromium CQ bot without notice.

Comment 1 by qyears...@chromium.org, May 25 2017

Status: Available (was: Untriaged)
Ideally, of course, we want to run on swarming everywhere ( issue 524758 ), and the current state is a transitional state.

As a short-term next step, making linux_trusty_blink_rel use swarming would make it consistent with linux_chromium_rel_ng, but then both would be inconsistent with the waterfall bot "WebKit Linux Trusty".

So maybe maybe some of the next builders to roll out to should be "WebKit Linux Trusty" and "linux_trusty_blink_rel"?

Tim, do you think it's useful to keep this bug open or does it make sense to to merge it into  bug 524758  or another bug?

Comment 2 by tansell@chromium.org, May 26 2017 

The layout tests should run identically when run on swarming and when run locally.

We should try and track down any cause for differences between the two.

Comment 3 by machenb...@chromium.org, May 26 2017 

Why not always switch both? The CI and the corresponding CQ bot? And if something doesn't work out, switch both back.

The issue marked as blocked shows how many dev-hours can get wasted if there's a difference between the two.

If a transition is desired, I suggest adding separate blink-swarming trybots as experimental trybots side-by-side...

Comment 4 by dpranke@chromium.org, May 31 2017

Mergedinto: 524758
Status: Duplicate (was: Available)
Landing these changes have proven to be hard no matter how we do it, and there are downsides to every approach.

I am fine with the approach we've taken during the transition so far, though I agree that if we can get the chromium CQ trybot change for a given platform to stick, we should flip over the other bots that run the layout tests on the same platform as soon as possible.

Once we get linux_chromium_rel_ng flipped, I'd also be fine with revisiting this for the other platforms so that we do flip the webkit/blink bots at the same time as well. At that point, we'll have more experience with things, and it should be easier to land a broader change.

I'm going to close this as, apart from asking about the migration strategy, this bug is no different than  bug 524758 .
►

Sign in to add a comment