New issue
Advanced search Search tips

Issue 797743 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 800570
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

WPT Import: Tests already in TestExpectations can end up being duplicated

Project Member Reported by raphael....@intel.com, Dec 27 2017

Issue description

This has been causing WPT imports to fail for the past few days. From https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/8733, for example:

2017-12-27 01:44:12,290 - All jobs finished.
2017-12-27 01:44:12,291 - Adding test expectations lines to LayoutTests/TestExpectations.
2017-12-27 01:44:23,554 - Tests to rebaseline:
2017-12-27 01:44:23,555 -   external/wpt/websockets/opening-handshake/005.html?wss
2017-12-27 01:44:42,301 - Lines to write to TestExpectations:
2017-12-27 01:44:42,302 -   crbug.com/626703 external/wpt/websockets/keeping-connection-open/001.html?wss [ Timeout ]
2017-12-27 01:44:42,302 -   crbug.com/626703 external/wpt/websockets/constructor/014.html?wss [ Timeout ]
2017-12-27 01:44:43,269 - Generating MANIFEST.json
2017-12-27 01:44:48,940 - Committing changes.
Traceback (most recent call last):
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/wpt-import", line 24, in <module>
    main()
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/wpt-import", line 17, in main
    host.exit(importer.main())
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py", line 162, in main
    if not self.update_expectations_for_cl():
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py", line 208, in update_expectations_for_cl
    self._upload_patchset(message)
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py", line 435, in _upload_patchset
    self.git_cl.run(['upload', '-f', '-t', message, '--gerrit'])
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl.py", line 60, in run
    return self._host.executive.run_command(command, cwd=self._cwd)
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/system/executive.py", line 337, in run_command
    (error_handler or self.default_error_handler)(script_error)
  File "/mnt/data/b/rr/tmp9ekTxk/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/system/executive.py", line 244, in default_error_handler
    raise error
webkitpy.common.system.executive.ScriptError: Failed to run "['git', 'cl', 'upload', '-f', '-t', 'Update test expectations and baselines.', '--gerrit']" exit_code: 1
output: Last 500 characters of output:
ml?wss
third_party/WebKit/LayoutTests/TestExpectations:2782 Duplicate or ambiguous entry lines third_party/WebKit/LayoutTests/TestExpectations:1931 and third_party/WebKit/LayoutTests/TestExpectations:2782. external/wpt/websockets/keeping-connection-open/001.html?wss
Lint failed.

external/wpt/websockets/keeping-connection-open/001.html?wss has an entry in TestExpectations that current looks like this:

    crbug.com/709227 external/wpt/websockets/keeping-connection-open/001.html?wss [ Failure ]

whereas wpt_expectations_updater.py is trying to add a new entry that looks like this:

    crbug.com/626703 external/wpt/websockets/keeping-connection-open/001.html?wss [ Timeout ]

instead of updating the existing TestExpecations entry.
 
I'll manually edit TestExpectations to unblock imports, but a proper solution needs more discussion.
Project Member

Comment 2 by bugdroid1@chromium.org, Dec 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8259896f3b644a5ce426e8367138c55a05bf9aec

commit 8259896f3b644a5ce426e8367138c55a05bf9aec
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Wed Dec 27 12:01:00 2017

Manually mark a few external/wpt/websockets tests as timing out.

Recent WPT imports (post commit 3975ca81, "Fix some websockets tests") are
failing because the WPT code is attempting to add new TestExpectation
entries for some tests that are already there.

For now, unblock imports by manually expecting
external/wpt/websockets/constructor/014.html?wss and crbug.com/709227
external/wpt/websockets/keeping-connection-open/001.html?wss to both fail
and time out.

TBR=robertma@chromium.org

Bug:  797743 
Change-Id: I5c6b381a30eea4428472c77253cf5291c65d71dc
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/844321
Reviewed-by: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#526216}
[modify] https://crrev.com/8259896f3b644a5ce426e8367138c55a05bf9aec/third_party/WebKit/LayoutTests/TestExpectations

Project Member

Comment 3 by bugdroid1@chromium.org, Dec 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a9fd9eb2ade18dbd028703a7f6bcb862923c444a

commit a9fd9eb2ade18dbd028703a7f6bcb862923c444a
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Wed Dec 27 18:42:57 2017

Accept failures in external/wpt/websockets/opening-handshake/005.html?wss

Recent WPT imports (post commit 3975ca81, "Fix some websockets tests") are
failing because the WPT code is attempting to mark the test above as timing
out only on Mac10.3 even though the test is actually failing on
Linux (Mac10.3 is not even being tested in the trybots).

For now, unblock imports by expecting failures in the test above so that the
WPT code (hopefully) does not try to update its corresponding
TestExpectations line.

TBR=robertma@chromium.org

Bug:  797743 
Change-Id: I2bbf5a24b13219e9d2880a7ce1afe36f042348e7
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/844816
Reviewed-by: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#526233}
[modify] https://crrev.com/a9fd9eb2ade18dbd028703a7f6bcb862923c444a/third_party/WebKit/LayoutTests/TestExpectations

Regarding comment #3: I'm not totally sure why a failure in the linux_trusty_blink_rel trybot would cause the [Mac10.3] specifier to be added to TestExpectations.
Cc: -robertma@chromium.org
Labels: -Pri-3 Pri-2
Owner: robertma@chromium.org
Status: Started (was: Available)
An example of the weird [Mac10.13] specifier can be found at https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/8736

Note that the log says:
2017-12-27 10:25:19,749 - Adding test expectations lines to LayoutTests/TestExpectations.
2017-12-27 10:25:26,749 - No results for build Build(builder_name=u'android_blink_rel', build_number=5051)
2017-12-27 10:25:28,191 - Tests to rebaseline:
2017-12-27 10:25:28,192 -   external/wpt/websockets/opening-handshake/005.html?wss
2017-12-27 10:25:47,843 - No lines to write to TestExpectations.

which means the expectation line was modified by rebaseline-cl, not added by wpt_expectations_updater. The latter case happens when a test cannot be rebaselined.

I wasn't actually aware that rebaseline{-cl} might also modify TestExpectations. There might be a bug. Nevertheless, the changes should be logged.

I also have a feeling that this is related to  issue 799076 .
Mergedinto: 800570
Status: Duplicate (was: Started)
The lingering [ Mac10.13 ] specifier was caused by  issue 800557 .

The original issue in the top description, however, is a serious problem I didn't know before. Filed a new bug and merging this issue there.

Sign in to add a comment