[WPT import] when some bots have no results, wpt-update-expectations should "fill in" platforms. |
|||||
Issue descriptionBackground: When wpt-importer is doing an import, the "wpt-update-expectations" script is invoked to both update baselines and add lines to TestExpectations. It does this based on results of try jobs triggered for all supported platforms. Sometimes, some of the platforms fail; the rebaseline-cl tool will fill in results for missing platforms when rebaselining, but the expectation-line part of the tool doesn't take missing platforms into account. For example, in: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.infra.cron%2Fwpt-importer%2F88%2F%2B%2Frecipes%2Fsteps%2Fupdate_wpt%2F0%2Fstdout Mac10.10 and Mac10.11 results were missing, and the tool suggested adding expectations like this: Lines to write to TestExpectations: crbug.com/626703 [ Android Linux Mac10.12 Mac10.9 Retina Win ] external/wpt/battery-status/battery-charging-manual.https.html [ Skip ] crbug.com/626703 [ Android Linux Mac10.12 Mac10.9 Retina Win ] external/wpt/battery-status/battery-plugging-in-manual.https.html [ Skip ] crbug.com/626703 [ Android Linux Mac10.12 Mac10.9 Retina Win ] external/wpt/battery-status/battery-unplugging-manual.https.html [ Skip ] It probably would have been more helpful for it to suggest adding cross-platform expectations: crbug.com/626703 external/wpt/battery-status/battery-charging-manual.https.html [ Skip ] crbug.com/626703 external/wpt/battery-status/battery-plugging-in-manual.https.html [ Skip ] crbug.com/626703 external/wpt/battery-status/battery-unplugging-manual.https.html [ Skip ]
,
May 26 2017
Will now start trying to fix this.
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e243e269e9eb7ab38a06d6592902f4b78f96722d commit e243e269e9eb7ab38a06d6592902f4b78f96722d Author: Quinten Yearsley <qyearsley@google.com> Date: Thu Jun 01 20:35:12 2017 wpt-update-expectations: "Fill in" results for ports with no results. This makes it so that if some bot fails to run for some reason, new expectation lines include the port for the builder that didn't have results. For example, suppose mac10.9_blink_rel fails compile and there are cross-platform test failures that are observed on all other bots. The newly-added lines would be cross-platform. (This is what happened last week.) Bug: 722887 Change-Id: I209267ba9e63ec912e62b08b6469f47db7bba6df Reviewed-on: https://chromium-review.googlesource.com/520582 Commit-Queue: Quinten Yearsley <qyearsley@chromium.org> Reviewed-by: Jeff Carpenter <jeffcarp@chromium.org> Cr-Commit-Position: refs/heads/master@{#476421} [modify] https://crrev.com/e243e269e9eb7ab38a06d6592902f4b78f96722d/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/e243e269e9eb7ab38a06d6592902f4b78f96722d/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Jun 1 2017
In theory, this should be fixed now; will re-open if I see this again.
,
Jul 3 2017
,
Jul 3 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by qyears...@chromium.org
, May 26 2017