Trying to work around it on the current in-flight import review:
https://chromium-review.googlesource.com/c/chromium/src/+/1147931

robertma@, if that's a bad idea, please let me know. Now that we have import notifications, I'm not inclined to do manual imports, but a 3-step process of updating TestExpectations, auto-importing and reverting also isn't much fun.

Nope, that didn't work, the importer uploaded another patch set to revert my changes and then failed :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b74ba743463a9e3d517ce6d38664797a329cc6a2

commit b74ba743463a9e3d517ce6d38664797a329cc6a2
Author: Philip Jägenstedt <foolip@chromium.org>
Date: Tue Jul 24 10:33:42 2018

Skip a virtual/ test to unblock the WPT importer

It will be enabled again after the next import.

Bug:  866802 
Change-Id: Id2f4198b7699be1bdf185758401fdb74413a1f21
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/1148208
Reviewed-by: Jonathon Kereliuk <kereliuk@chromium.org>
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577478}
[modify] https://crrev.com/b74ba743463a9e3d517ce6d38664797a329cc6a2/third_party/WebKit/LayoutTests/TestExpectations

This is a bug in optimize-baselines (called by rebaseline-cl). And it's not really related to virtual suites.

The symptom is that all-PASS baselines kept in order to prevent fallback cannot be updated. e.g. a test fails on most platforms but mac; the generic baseline contains failures and we have to put an all-PASS baseline at mac to prevent it from falling back to the generic one. In such scenario, if a new subtest is added to the test, rebaseline-cl cannot automatically update the baseline at mac to include the new test.

The root cause is in the BaselineOptimizer where all-PASS baselines are treated specially and all all-PASS baselines are considered equal, without comparing their content (or rather, hash). It was done in order to simplify the handling of implicit all-PASS results, but I didn't realize difference between two all-PASS results could matter in some case. e.g. In the aforementioned scenario, we'd need to promote the new mac-{version} baselines up to mac, but because all of them are considered equal, BaselineOptimizer simply removes the mac-{version} ones without overwriting the mac one.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/82d1bd2babc290ca236aad2933cc0b4254d1cd01

commit 82d1bd2babc290ca236aad2933cc0b4254d1cd01
Author: Robert Ma <robertma@chromium.org>
Date: Wed Jul 25 21:53:44 2018

Rebaseline a virtual PiP test and unskip it

TBR=foolip

Bug:  866802 
Change-Id: I633708217593c5b132beb69eeb27f7e6caeeedc0
Reviewed-on: https://chromium-review.googlesource.com/1150632
Reviewed-by: Robert Ma <robertma@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578084}
[modify] https://crrev.com/82d1bd2babc290ca236aad2933cc0b4254d1cd01/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/82d1bd2babc290ca236aad2933cc0b4254d1cd01/third_party/WebKit/LayoutTests/virtual/video-surface-layer/external/wpt/picture-in-picture/idlharness.window-expected.txt

This importer is broken again because of this starting from https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/21746.

It's the exact same test too: virtual/video-surface-layer/external/wpt/picture-in-picture/idlharness.window.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8c8a7234e3d9df8fc6580ecd477ce14a53c828e5

commit 8c8a7234e3d9df8fc6580ecd477ce14a53c828e5
Author: Philip Jägenstedt <foolip@chromium.org>
Date: Thu Jul 26 09:59:28 2018

Revert "Rebaseline a virtual PiP test and unskip it"

This reverts commit 82d1bd2babc290ca236aad2933cc0b4254d1cd01.

Reason for revert: wpt import is blocked again on the exact test.

Original change's description:
> Rebaseline a virtual PiP test and unskip it
> 
> TBR=foolip
> 
> Bug:  866802 
> Change-Id: I633708217593c5b132beb69eeb27f7e6caeeedc0
> Reviewed-on: https://chromium-review.googlesource.com/1150632
> Reviewed-by: Robert Ma <robertma@chromium.org>
> Commit-Queue: Robert Ma <robertma@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#578084}

TBR=foolip@chromium.org,robertma@chromium.org

Change-Id: I0090e75e1b219fbc3b0117c887b4e348ca628473
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  866802 
Reviewed-on: https://chromium-review.googlesource.com/1151149
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578253}
[modify] https://crrev.com/8c8a7234e3d9df8fc6580ecd477ce14a53c828e5/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/8c8a7234e3d9df8fc6580ecd477ce14a53c828e5/third_party/WebKit/LayoutTests/virtual/video-surface-layer/external/wpt/picture-in-picture/idlharness.window-expected.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c5c6952d686a5df745289840595d8b0316a3aa2

commit 1c5c6952d686a5df745289840595d8b0316a3aa2
Author: Robert Ma <robertma@chromium.org>
Date: Thu Jul 26 17:56:32 2018

[blinkpy] Allow rebaseline-cl to update all-PASS baselines

Previously, optimize-baselines assumes all all-PASS testharness.js
results are the same, partly to avoid special-casing the implicit
all-PASS results. However, in scenarios demonstrated in
 https://crbug.com/866802#c4 , we do need to differentiate different
all-PASS results to allow us to update all-PASS baselines (that are kept
to prevent fallback).

This CL implements a ResultDigest class with special equality
comparitors which treat implicit all-PASS results specially (equal to
any all-PASS results). Thanks to the abstraction, the call sites are
still kept simple.

Bug:  866802 
Change-Id: Ic6a361caa86f12598f29ad7693711d527f63006f
Reviewed-on: https://chromium-review.googlesource.com/1150629
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578361}
[modify] https://crrev.com/1c5c6952d686a5df745289840595d8b0316a3aa2/third_party/blink/tools/blinkpy/common/checkout/baseline_optimizer.py
[modify] https://crrev.com/1c5c6952d686a5df745289840595d8b0316a3aa2/third_party/blink/tools/blinkpy/common/checkout/baseline_optimizer_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f82bb8de940b4afffe0f4fe16ac7c7ac116c6a0a

commit f82bb8de940b4afffe0f4fe16ac7c7ac116c6a0a
Author: Robert Ma <robertma@chromium.org>
Date: Thu Jul 26 18:41:53 2018

Reland "Rebaseline a virtual PiP test and unskip it"

This is a reland of 82d1bd2babc290ca236aad2933cc0b4254d1cd01

Original change's description:
> Rebaseline a virtual PiP test and unskip it
> 
> TBR=foolip
> 
> Bug:  866802 
> Change-Id: I633708217593c5b132beb69eeb27f7e6caeeedc0
> Reviewed-on: https://chromium-review.googlesource.com/1150632
> Reviewed-by: Robert Ma <robertma@chromium.org>
> Commit-Queue: Robert Ma <robertma@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#578084}

Bug:  866802 
Change-Id: Ib6b87fa99899ec0d974c417cb6e2362ddeabcb1b
Reviewed-on: https://chromium-review.googlesource.com/1151467
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578382}
[modify] https://crrev.com/f82bb8de940b4afffe0f4fe16ac7c7ac116c6a0a/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/f82bb8de940b4afffe0f4fe16ac7c7ac116c6a0a/third_party/WebKit/LayoutTests/virtual/video-surface-layer/external/wpt/picture-in-picture/idlharness.window-expected.txt