New issue
Advanced search Search tips

Issue 629275 link

Starred by 2 users

Issue metadata

Status: Duplicate
Merged: issue 621599
Owner:
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocked on:
issue 634200
issue 635090
issue 637348

Blocking:
issue 621599



Sign in to add a comment

Make a w3c test "auto roll bot" to update W3C tests add expectations and baselines.

Project Member Reported by dcampb@google.com, Jul 18 2016

Issue description

Currently, deps updater's --auto-update flag, updates w3c deps and commits if none of the try-job results failed. 

As an add-on, we would like this script to update TestExpectations with expectations lines for the failing tests from the try-job results and still commit. 


related to bug:  625254 
 
Project Member

Comment 1 by sheriffbot@chromium.org, Jul 19 2016

Labels: Hotlist-Google
Project Member

Comment 3 by bugdroid1@chromium.org, Jul 27 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/80e6a17957c49b648085c652168db83dea2c727f

commit 80e6a17957c49b648085c652168db83dea2c727f
Author: dcampb <dcampb@google.com>
Date: Wed Jul 27 18:36:26 2016

Removed line used for testing in deps_updater.py

BUG= 629275 ,

Review-Url: https://codereview.chromium.org/2185823004
Cr-Commit-Position: refs/heads/master@{#408197}

[modify] https://crrev.com/80e6a17957c49b648085c652168db83dea2c727f/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Comment 4 by dcampb@google.com, Aug 3 2016

Status: Started (was: Assigned)

Comment 5 by dcampb@google.com, Aug 4 2016

Blockedon: 634200

Comment 6 by raikiri@google.com, Aug 5 2016

Blockedon: 635090
Blockedon: 637348
Cc: -qyears...@chromium.org dcampb@google.com
Labels: -OS-Linux -OS-Windows -Hotlist-Google -OS-Mac
Owner: qyears...@chromium.org
This is mostly done, but we haven't quite yet got successful runs; will mark as done then. Current blocker is  issue 637348 .
Project Member

Comment 8 by sheriffbot@chromium.org, Aug 13 2016

Labels: Hotlist-Google
Project Member

Comment 9 by bugdroid1@chromium.org, Aug 17 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6c68eeb5dd4ccf9fa0e7bf97c5999d31d4fd9cc

commit c6c68eeb5dd4ccf9fa0e7bf97c5999d31d4fd9cc
Author: qyearsley <qyearsley@chromium.org>
Date: Wed Aug 17 18:18:00 2016

In deps_updater.py: After updating, set commit and close on CQ failure.

Currently, w3c-test-autoroller:
 1. Uploads a CL.
 2. Triggers try jobs on Blink try bots.
 3. Waits for try job results (which are failing).
 4. Modifies TestExpectations and commits.
 5. Uploads another patch.
 6. Calls `git cl land -f --auth-refresh-token-json ...`.

I think that when landing CLs, w3c-test-autoroller commits
should go through the CQ, and if the CQ fails even after
updating TestExpectations etc., then the CL should be closed.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2248913003
Cr-Commit-Position: refs/heads/master@{#412589}

[modify] https://crrev.com/c6c68eeb5dd4ccf9fa0e7bf97c5999d31d4fd9cc/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Project Member

Comment 10 by bugdroid1@chromium.org, Aug 17 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f4d25a5140a064d50a5c348acf7697219b2be3da

commit f4d25a5140a064d50a5c348acf7697219b2be3da
Author: qyearsley <qyearsley@chromium.org>
Date: Wed Aug 17 20:00:24 2016

In deps_updater.py, only use Rietveld refresh token if given.

In general, when `update-w3c-deps --auto-import` is run, it should
only use the Rietveld authentication when given.

This change makes it so that if the script is run on a user's
workstation, then that user's credentials may be used.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2249633005
Cr-Commit-Position: refs/heads/master@{#412619}

[modify] https://crrev.com/f4d25a5140a064d50a5c348acf7697219b2be3da/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Project Member

Comment 11 by bugdroid1@chromium.org, Aug 24 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c0ec8559a02e430a346d991b8e75d35eeb7088fa

commit c0ec8559a02e430a346d991b8e75d35eeb7088fa
Author: qyearsley <qyearsley@chromium.org>
Date: Wed Aug 24 17:24:38 2016

Fix git cl upload invocation in deps_updater.py.

This is a follow-up fix to http://crrev.com/2265603003.

In this CL:
 - Add missing comma in arg list for git cl upload.
   This is causing recent builds to fail because
   "--rietveld" and "-m" are concatenated together, e.g.
   https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-autoroller/builds/6540
 - Remove now-unused argument auth_refresh_token_json of do_auto_update.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2268183004
Cr-Commit-Position: refs/heads/master@{#414097}

[modify] https://crrev.com/c0ec8559a02e430a346d991b8e75d35eeb7088fa/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Project Member

Comment 12 by bugdroid1@chromium.org, Aug 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0f77323cdf30b116f8055b11ef890d6a835b2044

commit 0f77323cdf30b116f8055b11ef890d6a835b2044
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Aug 25 18:58:45 2016

In update-w3c-test-expectations, handle failure to fetch results.

Specifically, when no layout tests are fetched for a particular build,
log a warning and continue instead of raising an exception and aborting.

I noticed this error when running the script on my workstation.

This CL also adds several TODO notes.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2277303003
Cr-Commit-Position: refs/heads/master@{#414497}

[modify] https://crrev.com/0f77323cdf30b116f8055b11ef890d6a835b2044/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot_mock.py
[modify] https://crrev.com/0f77323cdf30b116f8055b11ef890d6a835b2044/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py
[modify] https://crrev.com/0f77323cdf30b116f8055b11ef890d6a835b2044/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations.py
[modify] https://crrev.com/0f77323cdf30b116f8055b11ef890d6a835b2044/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations_unittest.py

Project Member

Comment 13 by bugdroid1@chromium.org, Aug 30 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b6cc4b6ed72adabac176e3acc5a41c048de65e8a

commit b6cc4b6ed72adabac176e3acc5a41c048de65e8a
Author: qyearsley <qyearsley@chromium.org>
Date: Tue Aug 30 03:45:40 2016

In webkitpy/common/net/git_cl.py, print rather than log "waiting" message.

Reason: In the recent w3c-test-autoroller jobs (e.g.
https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-autoroller/builds/6930),
the update wpt step is timing out after 40 minutes with no output after
print "Triggering try jobs".

I believe that this started happening after http://crrev.com/2265603003,
which made it so that when waiting for try job results it uses log.info
rather than print.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2285343002
Cr-Commit-Position: refs/heads/master@{#415055}

[modify] https://crrev.com/b6cc4b6ed72adabac176e3acc5a41c048de65e8a/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl.py

Project Member

Comment 14 by bugdroid1@chromium.org, Aug 31 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9a34bc3cf231cec8694e70ad61ec57a1f4986a21

commit 9a34bc3cf231cec8694e70ad61ec57a1f4986a21
Author: qyearsley <qyearsley@chromium.org>
Date: Wed Aug 31 22:35:41 2016

In update-w3c-test-expectations, only rebaseline newly-changed tests.

Reason: When downloading expectations for newly imported/updated w3c
tests, we only want to download baselines for tests that are modified
in the CL, because w3c test update CLs only modify tests (not Blink
source code).

This should prevent downloading baselines for other tests that failed
on the try bots on some platform.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2291153003
Cr-Commit-Position: refs/heads/master@{#415787}

[modify] https://crrev.com/9a34bc3cf231cec8694e70ad61ec57a1f4986a21/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations.py

Project Member

Comment 15 by bugdroid1@chromium.org, Sep 1 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/051c4fd1f1b9b3fa5e1dc8387b2c0e69d84726a4

commit 051c4fd1f1b9b3fa5e1dc8387b2c0e69d84726a4
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Sep 01 22:23:08 2016

Update list of resource files to copy from web-platform-tests.

Context: There is a list of files from web-platform-tests that are
put in LayoutTests/resources. Because these are copied from
web-platform-tests, the copy in LayoutTests/resources is expected
to be identical to the one in the imported wpt directory, and this
is enforced by a PRESUBMIT.

This CL updates both the list in the PRESUBMIT, and the list of files
that are automatically copied over during the import process.

Associating this with  bug 629275  since the w3c test autoroller
is currently getting stuck trying to import when idlharness.js
is updated but it's not copied over so it's failing the presubmit.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2302653002
Cr-Commit-Position: refs/heads/master@{#416083}

[modify] https://crrev.com/051c4fd1f1b9b3fa5e1dc8387b2c0e69d84726a4/third_party/WebKit/LayoutTests/PRESUBMIT.py
[modify] https://crrev.com/051c4fd1f1b9b3fa5e1dc8387b2c0e69d84726a4/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Cc: -raikiri@google.com -dcampb@google.com
Labels: -Hotlist-Google
Summary: Make a w3c test "auto roll bot" to update W3C tests add expectations and baselines. (was: Make a w3c test "auto roll bot" which tries to update W3C tests and adds lines to TestExpectations if needed.)
Project Member

Comment 17 by sheriffbot@chromium.org, Sep 2 2016

Labels: Hotlist-Google
Project Member

Comment 18 by bugdroid1@chromium.org, Sep 2 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/905922aa31995068d026594cd86a7c8aa95578d7

commit 905922aa31995068d026594cd86a7c8aa95578d7
Author: qyearsley <qyearsley@chromium.org>
Date: Fri Sep 02 17:16:06 2016

Fix invocation of git_cl.run() in DepsUpdater.write_test_expectations.

This isn't caught by the unit test because it's not tested; it's
questionable whether it's worth adding a unit test for that function as
it is now, since that method just invokes some external commands.

This issue was caught by looking at the latest failures on
w3c-test-autoroller.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2304053002
Cr-Commit-Position: refs/heads/master@{#416288}

[modify] https://crrev.com/905922aa31995068d026594cd86a7c8aa95578d7/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Project Member

Comment 19 by bugdroid1@chromium.org, Sep 8 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7d9d7a102b1eb03414e366e540b27c680a480ca3

commit 7d9d7a102b1eb03414e366e540b27c680a480ca3
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Sep 08 23:43:24 2016

Fix "git cl set-commit" line in deps_updater.py.

After running update-w3c-test-expectations, w3c-test-autoroller is now failing to run `git cl set-commit`.

For example: https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-autoroller/builds/7496/steps/update%20wpt/logs/stdio

After this, I'm expecting that it should update expectations/baselines, then try to commit, and then likely fail to pass the commit queue and abort.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2319763002
Cr-Commit-Position: refs/heads/master@{#417437}

[modify] https://crrev.com/7d9d7a102b1eb03414e366e540b27c680a480ca3/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Project Member

Comment 20 by bugdroid1@chromium.org, Sep 8 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7d9d7a102b1eb03414e366e540b27c680a480ca3

commit 7d9d7a102b1eb03414e366e540b27c680a480ca3
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Sep 08 23:43:24 2016

Fix "git cl set-commit" line in deps_updater.py.

After running update-w3c-test-expectations, w3c-test-autoroller is now failing to run `git cl set-commit`.

For example: https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-autoroller/builds/7496/steps/update%20wpt/logs/stdio

After this, I'm expecting that it should update expectations/baselines, then try to commit, and then likely fail to pass the commit queue and abort.

BUG= 629275 

Review-Url: https://codereview.chromium.org/2319763002
Cr-Commit-Position: refs/heads/master@{#417437}

[modify] https://crrev.com/7d9d7a102b1eb03414e366e540b27c680a480ca3/third_party/WebKit/Tools/Scripts/webkitpy/w3c/deps_updater.py

Status: dupe (was: Started)
Mergedinto: 621599
Status: Duplicate (was: dupe)

Sign in to add a comment