WPT import: Searching for exportable commits is failing |
||
Issue descriptionImports are currently blocked: https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/10254 2018-01-17 03:21:32,469 - Importing wpt@d5566dc52e0fc93255e373be97be2af6a27672e2 to Chromium 4bd9c8e6e51dfe55296d4c30ea965fc6c3c7f7b5 Traceback (most recent call last): File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/wpt-import", line 24, in <module> main() File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/wpt-import", line 17, in main host.exit(importer.main()) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py", line 117, in main commits = self.apply_exportable_commits_locally(local_wpt) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py", line 311, in apply_exportable_commits_locally commits = self.exportable_but_not_exported_commits(local_wpt) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py", line 342, in exportable_but_not_exported_commits self.host, local_wpt, self.wpt_github, require_clean=False, verify_merged_pr=True) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/chromium_exportable_commits.py", line 47, in exportable_commits_over_last_n_commits return _exportable_commits_since(start_commit, host, local_wpt, wpt_github, require_clean, verify_merged_pr) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/chromium_exportable_commits.py", line 73, in _exportable_commits_since state, error = get_commit_export_state(commit, local_wpt, wpt_github, verify_merged_pr) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/chromium_exportable_commits.py", line 110, in get_commit_export_state if _is_commit_exported(chromium_commit, local_wpt, wpt_github, verify_merged_pr): File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/chromium_exportable_commits.py", line 118, in _is_commit_exported pull_request = wpt_github.pr_for_chromium_commit(chromium_commit) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github.py", line 316, in pr_for_chromium_commit pull_request = self.pr_with_change_id(chromium_commit.change_id()) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github.py", line 328, in pr_with_change_id for pull_request in self.all_pull_requests(): File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/memoized.py", line 45, in __call__ result = self._function(*args) File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github.py", line 213, in all_pull_requests response = self.request(path, method='GET') File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github.py", line 67, in request headers=headers File "/mnt/data/b/rr/tmpe1A38l/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/web.py", line 49, in request return opener.open(request) File "/mnt/data/b/cipd_path_tools/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/mnt/data/b/cipd_path_tools/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/mnt/data/b/cipd_path_tools/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(*args) File "/mnt/data/b/cipd_path_tools/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/mnt/data/b/cipd_path_tools/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 422: Unprocessable Entity With some debugging, I can see that the problem is when the code looking for exportable commits is querying GitHub. Specifically, https://api.github.com/search/issues?q=repo%3Aw3c%2Fweb-platform-tests+type%3Apr+label%3Achromium-export&page=11&per_page=100 causes GitHub to return the following with an HTTP 422 response: {"message":"Only the first 1000 search results are available","documentation_url":"https://developer.github.com/v3/search/"}
,
Jan 17 2018
The weird part is that this search currently has 1083 results, and only a handful of PRs were created less than a day ago, so that limit should've been reached at least a few days ago. In any case, we've recently reached the 1k exported commits mark :-)
,
Jan 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/34f3f3e296efac666a641da6c09285166802cdc4 commit 34f3f3e296efac666a641da6c09285166802cdc4 Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com> Date: Wed Jan 17 14:47:58 2018 WPT import: Lower the default PR history window in WPTGitHub to 1000. Imports are currently blocked because the pagination code querying GitHub for all pull requests with the "chromium-export" label is going further than the 1000 results GitHub's API documentation says its API provides, and we fail because the API query (from page 11 onwards) returns an HTTP 422 error. Bug: 803033 Change-Id: I1af80448355b6b8e90338459ed3be4572828578b Reviewed-on: https://chromium-review.googlesource.com/870392 Reviewed-by: Philip Jägenstedt <foolip@chromium.org> Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com> Cr-Commit-Position: refs/heads/master@{#529739} [modify] https://crrev.com/34f3f3e296efac666a641da6c09285166802cdc4/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github.py [modify] https://crrev.com/34f3f3e296efac666a641da6c09285166802cdc4/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github_unittest.py
,
Jan 17 2018
I actually knew this limitation (https://crbug.com/752214#c2), but the actual behaviour is slightly different from what I expected. I thought GitHub would simply tell us there are 1000 results. In particular, I extract the link to the next page from HTTP response headers, instead of constructing the URL myself. I'd expect GitHub not telling me there is a page 11. Anyhow, thank you Raphael for putting off the fire! The immediate issue should now be fixed, and a window size of 1000 should be more than enough, so there is no urgency now. For the long-term goal of using a more sensible window size, see issue 752214.
,
Jan 19 2018
> In any case, we've recently reached the 1k exported commits mark :-) This is really pretty awesome, yay! \o/ |
||
►
Sign in to add a comment |
||
Comment 1 by raphael....@intel.com
, Jan 17 2018