Issue 662929  has been merged into this issue.

After some investigation, I realized that this incorrect behavior is due to https://codereview.chromium.org/2389873002.

What happened in https://codereview.chromium.org/2590903002 was actually that some JS resource files were modified but not the HTML tests themselves, and update-w3c-test-expectations has an extra check (the method get_modified_existing_tests) which makes it so that only tests that are directly modified will be rebaselined.

Now that rebaseline-cl uses the retry summary etc., I think that update-w3c-test-expectations should be updated to no longer check which tests exist and are modified - that is, get_modified_existing_tests should be removed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b341f101e1123f2eb9c67c074d8c1a5036fc46b2

commit b341f101e1123f2eb9c67c074d8c1a5036fc46b2
Author: qyearsley <qyearsley@chromium.org>
Date: Tue Jan 03 19:51:39 2017

Add example cases in unit tests for tests with "Slow" expectations.

This CL just expands the unit tests for:
  layout_test_results
  rebaseline_cl
  update_w3c_test_expectations:

The purpose of this was to investigate  http://crbug.com/676196  and see whether slow tests were not rebaselined as expected due to being treated differently than results with expected: "Pass" and actual: "Text".

The result was that it appear that slow results should be handled as expected.

This CL doesn't change behavior -- it just adds more test methods to unit tests.

BUG= 676196 

Review-Url: https://codereview.chromium.org/2605103002
Cr-Commit-Position: refs/heads/master@{#441191}

[modify] https://crrev.com/b341f101e1123f2eb9c67c074d8c1a5036fc46b2/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layout_test_results_unittest.py
[modify] https://crrev.com/b341f101e1123f2eb9c67c074d8c1a5036fc46b2/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
[modify] https://crrev.com/b341f101e1123f2eb9c67c074d8c1a5036fc46b2/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py

Uploaded CL: https://codereview.chromium.org/2604213002 which would basically undo http://crrev.com/2389873002 ("only rebaseline existing tests", reviewed by jsbell).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/456acc90587baec5eac79f0273d46145f6712cf7

commit 456acc90587baec5eac79f0273d46145f6712cf7
Author: qyearsley <qyearsley@chromium.org>
Date: Fri Jan 13 00:45:22 2017

When auto-importing, don't limit rebaselining to directly modified tests.

BUG= 676196 

Review-Url: https://codereview.chromium.org/2604213002
Cr-Commit-Position: refs/heads/master@{#443417}

[modify] https://crrev.com/456acc90587baec5eac79f0273d46145f6712cf7/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations.py
[modify] https://crrev.com/456acc90587baec5eac79f0273d46145f6712cf7/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py

Opening again since I think that this is probably related to some dubious deletions of TestExpectation lines that shouldn't have been modified; see https://codereview.chromium.org/2645553002/.

Update: I was looking at update_w3c_test_expectations.py to find how those lines might have been deleted, but didn't find anything obvious; there's no special logic to remove lines in update-w3c-test-expectations, or to do anything other than insert some new lines for new unexpected results (crashes/timeouts).

But, perhaps the process of reading in the expectations, parsing them all, and writing them out again somehow modifies the file. Related code: https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations.py?l=299

Filed  bug 687442  for the removing of TestExpectations lines.

This bug can be focused on whether expectations should be added for files that aren't directly modified (In theory, sometimes they should be, but only if they depend on resources that were modified).

So, I think that in general, expectations should be added for tests that aren't directly modified, and this is a fairly common case, so this is "working as intended". The issue of removing lines was  bug 687442 .

The only situation where this is a problem is for flaky tests, which is bug 701234.