The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6976fe02ac409ac1a1f0bd51fbb299e106cfc7ce

commit 6976fe02ac409ac1a1f0bd51fbb299e106cfc7ce
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Mon Oct 23 15:51:50 2017

Manually add MediaStream-default-feature-policy.https.html to NeverFixTests

This should hopefully unblock WPT imports, as the test was renamed upstream
but our infrastructure is removing the corresponding NeverFixTests entry
instead of updating it.

TBR=qyearsley@chromium.org,robertma@chromium.org,raymes@chromium.org

Bug:  777453 
Change-Id: I923a692dd742785a6b246c1732ded593e86706f5
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/733322
Reviewed-by: Raphael Kubo da Costa (rakuco) <raphael.kubo.da.costa@intel.com>
Commit-Queue: Raphael Kubo da Costa (rakuco) <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#510804}
[modify] https://crrev.com/6976fe02ac409ac1a1f0bd51fbb299e106cfc7ce/third_party/WebKit/LayoutTests/NeverFixTests

Thanks for reporting Raphael.

So I looked at the importer code again, and it looks like it will change NeverFixTests lines as long as the rename is a 100% rename, but not it didn't in this case because it was a rename with 90% similarity (some changes). The reason why I decided it to make it require 100% similarity was because there was a case a while ago where a file was changed and renamed at the same time, and so the expectation line didn't apply anymore to the renamed test, and the updated expectation line was incorrect.

In this particular case, that test (MediaStream-default-feature-policy.https.html) seems to have just failed with an baseline mismatch. This looks it might be another case where the test should have been rebaselined, but the virtual version of the test (virtual/feature-policy-permissions/...) is all-PASS but the non-virtual version has some failures... so upon rebaseline, the virtual version is removed (all-PASS baselines are auto-removed), then upon running the tests, the non-virtual baseline is used, thus the mismatch.

So in short, I think the real problem here is  bug 768525 .

Does that sound right? Or maybe there is some other reason why that test is disabled in NeverFixTests? Is there anything else that should be fixed?

------------
Notes:

The updating of expectations files for renamed tests is done in TestImporter.update_all_test_expectations_files() This uses port.all_expectations_dict() to read expectations lines from different files, which is supposed to include NeverFixTests.

Code search links:
  https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py?l=500
  https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py?l=1372

The test rename was 90% similarity:
https://chromium-review.googlesource.com/c/chromium/src/+/732067/2/third_party/WebKit/LayoutTests/external/wpt/mediacapture-streams/MediaStream-default-feature-policy.https.html

The failure was a text baseline mismatch:
https://storage.googleapis.com/chromium-layout-test-archives/linux_chromium_rel_ng/571005/layout-test-results/test-expectations.html

 Issue 777502  has been merged into this issue.

Thanks for the report and analysis, Raphael and Quinten.

Looks like there are two aspects to this issue:

1. Importer uses a strict 100% rename detection threshold, which leads to this incident. Importer actually worked as expected here. And I'm also in favor of 100% threshold: for one, I'd argue that renaming and modifying at the same time is not a good practice in git; also so far there's been few occasions manual intervention was needed.

2. The expectation line was added in https://chromium-review.googlesource.com/c/chromium/src/+/654021 . It makes sense to me to mark a test as never fixed if the test only runs with certain flags and is otherwise a waste of resource, which seems like the original intention here. However, if the test were not skipped, rebaseline should have worked correctly.  Issue 768525  is definitely a real bug.

Thanks for the investigation, Quinten. I think you're right that this is an instance of  bug 768525 .