[WPT Export] Fetching GitHub PRs is unreliable and flaky |
||||
Issue descriptionThere was a Chromium change on Jan 31 that was reverted: https://chromium-review.googlesource.com/882002 https://chromium-review.googlesource.com/893536 An export PR for the first was created: https://github.com/w3c/web-platform-tests/pull/9239 None was created for the second because it wouldn't apply to wpt master before the first PR was merged. Before the PR was merged automatically, robertma@ edited the description and closed it. However, this was apparently not enough, because later a PR was re-created: https://github.com/w3c/web-platform-tests/pull/9523 Why it took ~15 days before this happened I can't tell, that's one odd thing to understand. Then, through some manual efforts, I got #9523 merged yesterday, not knowing that it shouldn't be exported. As a result we're currently trying to import those changes back in, which have already been reverted. I will also manually export the revert to prevent this from happening. Beyond understanding why the export commit was never exported, we should also document what to do if we do have a commit+revert and for some reason don't want to export them. How do we reliably prevent both commits from being exported?
,
Feb 22 2018
The original description was: ``` Port another set of sticky tests to JS rather than reftests Change-Id: Idf6dd882d2d50b1ec349f245d76717553ee266b3 Reviewed-on: https://chromium-review.googlesource.com/882002 Reviewed-by: Robert Flack <flackr@chromium.org> Commit-Queue: Stephen McGruer <smcgruer@chromium.org> Cr-Commit-Position: refs/heads/master@{#532966} Revert "Port another set of sticky tests to JS rather than reftests" Change-Id: I7a773131066157e0d4216d7fa7c228bd8bff2021 Reviewed-on: https://chromium-review.googlesource.com/893536 Reviewed-by: Bret Sepulveda <bsep@chromium.org> Commit-Queue: Bret Sepulveda <bsep@chromium.org> Cr-Commit-Position: refs/heads/master@{#533000} ```
,
Feb 22 2018
Hmm, I don't know why, but https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/67787 was run after I edited the description and still the last line was "Searching for exportable Chromium commits" with no commit found to export apparently. Maybe we don't look that far back in history? I'll have to manually export the revert.
,
Feb 22 2018
Once we've figured out how to handle cases like this we should document it in the rotation doc: https://docs.google.com/document/d/10och6DoxEe6AhQX7MiQR8AhtjDheKIlf-GpF0impeJQ/edit?disco=AAAABpK7U08 In that comment Raphael describes a somewhat similar mishap.
,
Feb 22 2018
Revert landed in https://github.com/w3c/web-platform-tests/pull/9625, let's see how the next import fares.
,
Feb 22 2018
Phew, https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/12673 looks good, just the expected change being imported.
,
Feb 22 2018
It doesn't help that logs from when the original CLs were written don't show up, e.g.: * https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/64259 * https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/64113 It's interesting that the wpt-exporter run that re-exported the first CL also processed the CL reverting the former, and it (as expected) failed to create a PR for it: * https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/66528 However, the next run just mentions https://github.com/w3c/web-platform-tests/pull/9523 and doesn't report any failures or attempts to create other PRs: * https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/66529 I wonder if these are all symptoms that the code looking for exportable commits can be flaky at times.
,
Feb 23 2018
Wow, thanks for that extra digging, Raphael, very interesting that the revert was once considered exportable. Then I suspect it fell out of the window where we look for exportable commits.
,
Feb 23 2018
I considered that hypothesis, but if I understood the code correctly we look at the latest 5000 commits touching third_party/WebKit/LayoutTests/external/wpt. The CLs should be in that window at all times, we didn't have that many exportable changes between the end of January and mid-February.
,
Feb 26 2018
(I don't understand why the logs for wpt-exporter builds 64259 & 64113 disappear. Isn't LogDog supposed to keep these logs for a relatively long period?) As for the problematic export 66528, the concerning message is this line: 2018-02-14 12:08:52,789 - Fetched 100 PRs from GitHub. Only one page of PRs was fetched from GitHub, which was the root cause of the problem. So fetching the list of PRs is indeed unreliable sometimes. An insufficient list of PRs could be more damaging than what happened in this incident. We were lucky this time that only one PR was mistakenly re-created. There's a few possible failures: * The search result is incomplete, but the "incomplete_results" field isn't set. (https://developer.github.com/v3/search/#timeouts-and-incomplete-results) * The response doesn't set the Link header correctly, so the code can't find the next page. Basically, we need more sanity checks than checking for "incomplete_results". I'm thinking of checking both the "total_count" field and the actual total number of PRs fetched.
,
May 3 2018
is there an update on this issue @robertma ?
,
Jun 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2fa089d993adead8e7e5f97accc1081804ee0607 commit 2fa089d993adead8e7e5f97accc1081804ee0607 Author: Robert Ma <robertma@chromium.org> Date: Mon Jun 04 21:40:39 2018 [WPT Export] Check GitHub API returns enough PRs We've seen cases where GitHub API returns a suspiciously small amount of PRs without setting incomplete_results to false. This CL guards against this edge case to prevent clobbering unexported CLs or exporting the same CL twice. Bug: 814617 Change-Id: I1958799a1f3a0c548ec500eb1709eb6c429c4a19 Reviewed-on: https://chromium-review.googlesource.com/1085847 Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Commit-Queue: Robert Ma <robertma@chromium.org> Cr-Commit-Position: refs/heads/master@{#564248} [modify] https://crrev.com/2fa089d993adead8e7e5f97accc1081804ee0607/third_party/blink/tools/blinkpy/w3c/wpt_github.py [modify] https://crrev.com/2fa089d993adead8e7e5f97accc1081804ee0607/third_party/blink/tools/blinkpy/w3c/wpt_github_unittest.py
,
Jun 4 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by foolip@chromium.org
, Feb 22 2018