This also causes trouble for wpt import sometimes, in the case where a virtual version of a wpt test passes but the non-virtual version fails; see  bug 777453 .

This might be related to  issue 767356 . Something is likely wrong with our rebaselining algorithm. I'm investigating and will mark this issue as a duplicate when I'm certain.

Thanks for investigating Robert!

In general when rebaselining, optimization (deduplication) is done before removing the all-PASS testharness baselines. But sometimes all-PASS baselines are actually needed to prevent fall-back other baselines.

If there's a generic baseline with FAIL lines and a more specific baseline that is all-PASS, they won't be deduplicated. But then, after that, that all-PASS baseline shouldn't be removed; it needs to be kept to prevent falling back to the other baseline.

So: Even assuming the optimization code is all correct, I think that removing-of-all-PASS baselines is not correct, unless that test passes on all platforms and all virtual tests based on the test.

Related bug:  bug 570044 

Relevant code link:
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py?l=420

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/524aad83ac488247e9dcdaa59fdffb6ba1b99795

commit 524aad83ac488247e9dcdaa59fdffb6ba1b99795
Author: Robert Ma <robertma@chromium.org>
Date: Thu Nov 16 17:59:18 2017

[rebaseline] Keep non-generic all-PASS baselines

Only generic all-PASS testharness.js baselines can be safely removed.
Removing platform-specific all-PASS baselines can lead to falling back
to a different (and wrong) baseline.

Bug:  768525 
Change-Id: Ia4f2f37068e21cf85535864dbf6e196392fc1533
Reviewed-on: https://chromium-review.googlesource.com/773738
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517120}
[modify] https://crrev.com/524aad83ac488247e9dcdaa59fdffb6ba1b99795/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
[modify] https://crrev.com/524aad83ac488247e9dcdaa59fdffb6ba1b99795/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b

commit 89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b
Author: Robert Ma <robertma@chromium.org>
Date: Sat Dec 02 01:34:47 2017

Move all-PASS baseline optimization to BaselineOptimizer

which is a more logical and intuitive place than Rebaseline.

Besides, we add support for removing redundant all-PASS testharness.js
baselines that are not at the root. This may happen when a platfrom
result becomes all-PASS, when its fallback platforms are already
all-PASS, in which case we only download a new all-PASS baseline to the
platform. The previous optimization only looked at the root.

Bug:  768525 
Change-Id: Icf4fa98850f5a72541a20d54217a64daef8cb389
Reviewed-on: https://chromium-review.googlesource.com/803795
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521173}
[modify] https://crrev.com/89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b/docs/testing/layout_test_baseline_fallback.md
[modify] https://crrev.com/89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b/third_party/WebKit/Tools/Scripts/webkitpy/common/checkout/baseline_optimizer.py
[modify] https://crrev.com/89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b/third_party/WebKit/Tools/Scripts/webkitpy/common/checkout/baseline_optimizer_unittest.py
[modify] https://crrev.com/89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
[modify] https://crrev.com/89eeaa5ad4a121168286c9e1ebfbe4e63ff67f8b/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py