wpt-exporter is failing due to network timeout |
||||||
Issue descriptionThis began happening on Oct 12: https://build.chromium.org/p/chromium.infra.cron/builders/wpt-exporter/builds/43880 Since then a total of 6 builds have been green, but all on Oct 14 are red, the latest being: https://build.chromium.org/p/chromium.infra.cron/builders/wpt-exporter/builds/44019 End of log: 2017-10-14 06:59:46,478 - Cloning GitHub w3c/web-platform-tests into /tmp/wpt 2017-10-14 07:00:12,875 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 10 seconds... 2017-10-14 07:00:22,906 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 15.0 seconds... 2017-10-14 07:00:37,948 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 22.5 seconds... 2017-10-14 07:01:00,504 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 33.75 seconds... 2017-10-14 07:01:34,323 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 50.625 seconds... 2017-10-14 07:02:25,019 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 75.9375 seconds... 2017-10-14 07:03:41,034 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 113.90625 seconds... 2017-10-14 07:05:35,100 - Received HTTP status 400 loading "https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200". Retrying in 170.859375 seconds... Traceback (most recent call last): File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/wpt-export", line 26, in <module> main() File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/wpt-export", line 18, in main success = exporter.main() File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_exporter.py", line 56, in main open_gerrit_cls = self.gerrit.query_exportable_open_cls() File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/gerrit.py", line 52, in query_exportable_open_cls open_cls_data = self.get(path) File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/gerrit.py", line 30, in get raw_data = self.host.web.get_binary(url) File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/web.py", line 37, in get_binary return NetworkTransaction(return_none_on_404=return_none_on_404).run(lambda: urllib2.urlopen(url).read()) File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/network_transaction.py", line 61, in run self._check_for_timeout() File "/b/rr/tmp7ua35p/w/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/network_transaction.py", line 68, in _check_for_timeout raise NetworkTimeout() webkitpy.common.net.network_transaction.NetworkTimeout: NetworkTimeout step returned non-zero exit code: 1
,
Oct 14 2017
https://chromium-review.googlesource.com/changes/?q=project:"chromium/src"+status:open&o=CURRENT_FILES&o=CURRENT_REVISION&o=COMMIT_FOOTERS&o=DETAILED_ACCOUNTS&o=DETAILED_LABELS&n=200 works for me in the browser, but was a bit slow, and the first time it looked like a network error.
,
Oct 14 2017
The API we're using is https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html
,
Oct 14 2017
Also cc jeffcarp@ who wrote wpt-exporter, in case this has happened before or if it's obvious what the problem is.
,
Oct 14 2017
,
Oct 14 2017
Filed a Gerrit issue which is blocking this. I'll see about manually exporting anything that's currently blocked.
,
Oct 14 2017
https://build.chromium.org/p/chromium.infra.cron/builders/wpt-exporter/builds/43937 was the last green build, so any changes that have landed after that most likely haven't been exported, and possibly don't even have a PR yet.
,
Oct 14 2017
I have looked at all changes in LayoutTests/external/wpt/ since when the problem began on Oct 12, and manually handled these: https://github.com/w3c/web-platform-tests/pull/7767 (merged) https://github.com/w3c/web-platform-tests/pull/7677 (closed) https://github.com/w3c/web-platform-tests/pull/7297 (merged) https://github.com/w3c/web-platform-tests/pull/7293 (merged) If more manual exporting is needed, start as commit c5bbb48108ece3e569aaaa9dabd813f722c2f6cb, the master at the time when I did this.
,
Oct 16 2017
Damn, sorry I missed this on my rotation :-(.
,
Oct 16 2017
The issue that this is blocked on is now marked fixed, and wpt-exporter has been successfully exporting changes today!
,
Oct 16 2017
Sorry just triaging this - looks like things are back to normal. I had not seen this type of error before.
,
Oct 23 2017
Although the immediate problem was fixed from the Gerrit side, I think there is still something we should do in the exporter: given that we have experienced two Gerrit outages/bugs in the past month, we should catch the network exceptions and continue onto the second half of the workflow, i.e. exporting landed commits, which does not depend on Gerrit. The change is small and can benefit us by providing at least partial service of the exporter when Gerrit API is down. Note: we'd still want to make the wpt-export builder red if the Gerrit part throws.
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e21758bf780cf8c25927525be22dac8f54132fd8 commit e21758bf780cf8c25927525be22dac8f54132fd8 Author: Robert Ma <robertma@chromium.org> Date: Wed Nov 15 00:37:22 2017 [WPT Export] Catch Gerrit search API errors Continue to export landed commits in case Gerrit search API fails (which is not rare event). The script will still return false and turn on the red light so that the person on rotation can notice. Bug: 774781 Change-Id: I8b8f92128953a9616b78833cfa5af5cb8e71decb Reviewed-on: https://chromium-review.googlesource.com/769830 Commit-Queue: Robert Ma <robertma@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Cr-Commit-Position: refs/heads/master@{#516522} [modify] https://crrev.com/e21758bf780cf8c25927525be22dac8f54132fd8/third_party/WebKit/Tools/Scripts/webkitpy/w3c/gerrit.py [modify] https://crrev.com/e21758bf780cf8c25927525be22dac8f54132fd8/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_exporter.py [modify] https://crrev.com/e21758bf780cf8c25927525be22dac8f54132fd8/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_exporter_unittest.py [modify] https://crrev.com/e21758bf780cf8c25927525be22dac8f54132fd8/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_github.py
,
Nov 15 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by foolip@chromium.org
, Oct 14 2017Status: Assigned (was: Untriaged)