New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 814617 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

[WPT Export] Fetching GitHub PRs is unreliable and flaky

Project Member Reported by foolip@chromium.org, Feb 22 2018

Issue description

There was a Chromium change on Jan 31 that was reverted:
https://chromium-review.googlesource.com/882002
https://chromium-review.googlesource.com/893536

An export PR for the first was created:
https://github.com/w3c/web-platform-tests/pull/9239

None was created for the second because it wouldn't apply to wpt master before the first PR was merged.

Before the PR was merged automatically, robertma@ edited the description and closed it. However, this was apparently not enough, because later a PR was re-created:
https://github.com/w3c/web-platform-tests/pull/9523

Why it took ~15 days before this happened I can't tell, that's one odd thing to understand.

Then, through some manual efforts, I got #9523 merged yesterday, not knowing that it shouldn't be exported. As a result we're currently trying to import those changes back in, which have already been reverted. I will also manually export the revert to prevent this from happening.

Beyond understanding why the export commit was never exported, we should also document what to do if we do have a commit+revert and for some reason don't want to export them. How do we reliably prevent both commits from being exported?
 

Comment 1 by foolip@chromium.org, Feb 22 2018

Right now "Port another" doesn't appear in wpt-exporter logs, so the exporter isn't trying to export it but failing for some reason. Presumably the description of https://github.com/w3c/web-platform-tests/pull/9239 is what does it. I'll try changing that description to see what happens.

Comment 2 by foolip@chromium.org, Feb 22 2018

The original description was:
```
Port another set of sticky tests to JS rather than reftests
Change-Id: Idf6dd882d2d50b1ec349f245d76717553ee266b3
Reviewed-on: https://chromium-review.googlesource.com/882002
Reviewed-by: Robert Flack <flackr@chromium.org>
Commit-Queue: Stephen McGruer <smcgruer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532966}

Revert "Port another set of sticky tests to JS rather than reftests"
Change-Id: I7a773131066157e0d4216d7fa7c228bd8bff2021
Reviewed-on: https://chromium-review.googlesource.com/893536
Reviewed-by: Bret Sepulveda <bsep@chromium.org>
Commit-Queue: Bret Sepulveda <bsep@chromium.org>
Cr-Commit-Position: refs/heads/master@{#533000}
```

Comment 3 by foolip@chromium.org, Feb 22 2018

Hmm, I don't know why, but https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/67787 was run after I edited the description and still the last line was "Searching for exportable Chromium commits" with no commit found to export apparently. Maybe we don't look that far back in history?

I'll have to manually export the revert.

Comment 4 by foolip@chromium.org, Feb 22 2018

Cc: raphael....@intel.com
Once we've figured out how to handle cases like this we should document it in the rotation doc:
https://docs.google.com/document/d/10och6DoxEe6AhQX7MiQR8AhtjDheKIlf-GpF0impeJQ/edit?disco=AAAABpK7U08

In that comment Raphael describes a somewhat similar mishap.

Comment 5 by foolip@chromium.org, Feb 22 2018

Revert landed in https://github.com/w3c/web-platform-tests/pull/9625, let's see how the next import fares.

Comment 6 by foolip@chromium.org, Feb 22 2018

Phew, https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/12673 looks good, just the expected change being imported.
It doesn't help that logs from when the original CLs were written don't show up, e.g.:
* https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/64259
* https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/64113

It's interesting that the wpt-exporter run that re-exported the first CL also processed the CL reverting the former, and it (as expected) failed to create a PR for it:
* https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/66528

However, the next run just mentions https://github.com/w3c/web-platform-tests/pull/9523 and doesn't report any failures or attempts to create other PRs:
* https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/66529

I wonder if these are all symptoms that the code looking for exportable commits can be flaky at times.

Comment 8 by foolip@chromium.org, Feb 23 2018

Wow, thanks for that extra digging, Raphael, very interesting that the revert was once considered exportable. Then I suspect it fell out of the window where we look for exportable commits.
I considered that hypothesis, but if I understood the code correctly we look at the latest 5000 commits touching third_party/WebKit/LayoutTests/external/wpt. The CLs should be in that window at all times, we didn't have that many exportable changes between the end of January and mid-February.
Labels: -Pri-3 Pri-2
Status: Available (was: Untriaged)
Summary: [WPT Export] Fetching GitHub PRs is unreliable and flaky (was: Edited and aborted "Port another set of sticky tests to JS rather than reftests" export still re-exported later)
(I don't understand why the logs for wpt-exporter builds 64259 & 64113 disappear. Isn't LogDog supposed to keep these logs for a relatively long period?)

As for the problematic export 66528, the concerning message is this line:
2018-02-14 12:08:52,789 - Fetched 100 PRs from GitHub.

Only one page of PRs was fetched from GitHub, which was the root cause of the problem. So fetching the list of PRs is indeed unreliable sometimes. An insufficient list of PRs could be more damaging than what happened in this incident. We were lucky this time that only one PR was mistakenly re-created.

There's a few possible failures:
* The search result is incomplete, but the "incomplete_results" field isn't set. (https://developer.github.com/v3/search/#timeouts-and-incomplete-results)
* The response doesn't set the Link header correctly, so the code can't find the next page.

Basically, we need more sanity checks than checking for "incomplete_results". I'm thinking of checking both the "total_count" field and the actual total number of PRs fetched.
is there an update on this issue @robertma ?
Project Member

Comment 12 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2fa089d993adead8e7e5f97accc1081804ee0607

commit 2fa089d993adead8e7e5f97accc1081804ee0607
Author: Robert Ma <robertma@chromium.org>
Date: Mon Jun 04 21:40:39 2018

[WPT Export] Check GitHub API returns enough PRs

We've seen cases where GitHub API returns a suspiciously small amount of
PRs without setting incomplete_results to false. This CL guards against
this edge case to prevent clobbering unexported CLs or exporting the
same CL twice.

Bug:  814617 
Change-Id: I1958799a1f3a0c548ec500eb1709eb6c429c4a19
Reviewed-on: https://chromium-review.googlesource.com/1085847
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564248}
[modify] https://crrev.com/2fa089d993adead8e7e5f97accc1081804ee0607/third_party/blink/tools/blinkpy/w3c/wpt_github.py
[modify] https://crrev.com/2fa089d993adead8e7e5f97accc1081804ee0607/third_party/blink/tools/blinkpy/w3c/wpt_github_unittest.py

Status: Fixed (was: Available)

Sign in to add a comment