New issue
Advanced search Search tips

Issue 843116 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug



Sign in to add a comment

win7_chromium_rel_ng false positive success

Project Member Reported by fs...@chromium.org, May 15 2018

Issue description

Look at this:
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/163821

The build says success, while webkit_layout_tests failed.

Not sure what's happening.
 

Comment 1 by fs...@chromium.org, May 15 2018

Cc: fs...@chromium.org
Components: -Infra>Platform>Buildbot -Infra>Platform>CQ Infra>Client>Chrome
Labels: OS-Windows
Could it be because the failing test is marked experimental?

webkit_layout_tests (with patch, experimental)
Owner: jbudorick@chromium.org
Status: Assigned (was: Untriaged)
Indeed, this is the point of experimental tests

fserb, you're not the only person to get confused by this -- anything in particular that you think could help make this clearer? Adding a brief explanation that gets linked from the build page? Something else?

Comment 4 by fs...@chromium.org, May 15 2018

hmmmm. But the problem is those tests WERE broken on linux,mac and they should have been broken on win7 too. So I was confused by win7 being green, not by the test being red.

I had noticed the "experimental" on the title, but I assumed that, since it was replacing the normal one, that were the real deal. 

Is it possible to run both the experimental and the non-experimental one?


It's the same suite; it's not replacing the normal one, the builder is simply treating it differently. We marked it as experimental a month or so ago because the suite was problematically flaky (or flakier) on windows.

Comment 6 by fs...@chromium.org, May 15 2018

Ahn. Can you call it something else that makes is explicit that is being ignored, like "experimental - results ignored"

And wouldn't that mean that if I have a LayoutTest that only breaks on win7, that I'll be able to submit my change?
Yes, "experimental" means that if tests fail, your changes is not blocked from landing. As jbudorick@ says, in a sense this isn't worse than not running them at all.

(In another sense, it might be worse, because you might not understand what "experimental" means but you understand what it means for the test to not be there).

Sign in to add a comment