WPT Export M2 ☔ Issue |
||||||||||||||||||||||||||||||||||||||
Issue descriptionNow that the basic functionality for WPT Export is complete, milestone 2 is focused on these initiatives: 1. Make the exporter easier to manage as a piece of Chrome Infra - bug 693211 Add better alerting - bug 705561 Document how to put the exporter in --dry-run mode with a CL if misbehaving - bug 693821 Document how to unblock the export process 2. The export process is subject to head-of-line blocking ( bug 686471 ) - bug 702849 Make the export process asynchronous / make it impossible for one PR to block it - bug 700092 Create a GitHub PR immediately after an exportable CL is uploaded to Gerrit 3. There is poor visibility into the export process - bug 701147 All in-flight CLs and PRs should be visible in one place - bug 701176 The exporter should comment status updates on the CL 4. The export can fail due to tests found to be flaky in wpt infrastructure - This has been a problem at least twice: - https://github.com/w3c/web-platform-tests/pull/4832 - https://github.com/w3c/web-platform-tests/pull/5184 - We should (somehow) run the same tests as part of our CQ so that the problem can be fixed before landing. - Creating provisional PRs from CLs ( bug 700092 ) effectively fixes this problem. Once all CLs go through Gerrit, this will be less of an issue since we can resolve upstream flakiness before the downstream CL lands. 5. MANIFEST.json makes adding new WPT tests frustrating - bug 666957 MANIFEST.json is now updated in the background Auxiliary Goals: - Make the export process more visible to Blink developers and set up the expectation that they may need to modify their CLs if the Travis checks fail. - WPT Export should continue to not rely on a database for state management. ⛆ |
|
|
,
Apr 13 2017
,
May 12 2017
,
May 15 2017
,
May 15 2017
,
May 16 2017
,
May 16 2017
,
May 18 2017
,
May 22 2017
,
May 22 2017
,
May 26 2017
,
May 30 2017
,
Jun 7 2017
,
Jun 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7fddd24078bb7ec99884d17f0bad2893e7019508 commit 7fddd24078bb7ec99884d17f0bad2893e7019508 Author: Jeff Carpenter <jeffcarp@chromium.org> Date: Thu Jun 08 18:29:48 2017 Strip extra whitespace from service-worker tests Creating this change to test the provisional Gerrit CL -> WPT PR process all the way through. Bug: 707006 Change-Id: I650caed9e40844c05c6b6f6576f209e952aa8522 Reviewed-on: https://chromium-review.googlesource.com/511283 Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Reviewed-by: Mike West <mkwst@google.com> Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org> Cr-Commit-Position: refs/heads/master@{#478033} [modify] https://crrev.com/7fddd24078bb7ec99884d17f0bad2893e7019508/third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/redirected-response.https.html [modify] https://crrev.com/7fddd24078bb7ec99884d17f0bad2893e7019508/third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/resources/fetch-access-control-login.html [modify] https://crrev.com/7fddd24078bb7ec99884d17f0bad2893e7019508/third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/resources/fetch-cors-xhr-iframe.html [modify] https://crrev.com/7fddd24078bb7ec99884d17f0bad2893e7019508/third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/resources/invalid-blobtype-iframe.https.html [modify] https://crrev.com/7fddd24078bb7ec99884d17f0bad2893e7019508/third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/resources/invalid-header-iframe.https.html [modify] https://crrev.com/7fddd24078bb7ec99884d17f0bad2893e7019508/third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/update-after-oneday.https.html
,
Jun 26 2017
Grading Q2 OKRs, just wanted to share this data somewhere it'll be saved: # WPT Export latency from datetime on the commit in Chromium to datetime the PR was merged on GitHub # All numbers are in minutes # NumSamples = 115; Min = 16.80; Max = 1421.90 # Mean = 271.438116; Variance = 167643.745458; SD = 409.443214; Median 38.866667 # each ∎ represents a count of 1 16.8000 - 157.3100 [ 78]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 157.3100 - 297.8200 [ 5]: ∎∎∎∎∎ 297.8200 - 438.3300 [ 10]: ∎∎∎∎∎∎∎∎∎∎ 438.3300 - 578.8400 [ 0]: 578.8400 - 719.3500 [ 3]: ∎∎∎ 719.3500 - 859.8600 [ 1]: ∎ 859.8600 - 1000.3700 [ 4]: ∎∎∎∎ 1000.3700 - 1140.8800 [ 6]: ∎∎∎∎∎∎ 1140.8800 - 1281.3900 [ 3]: ∎∎∎ 1281.3900 - 1421.9000 [ 5]: ∎∎∎∎∎
,
Jun 27 2017
Thanks Jeff! It looks like the slowest export was 23.7 hours, is that correct? I recall a few that seem to have lingered for longer.
,
Jun 27 2017
Totally, I thought it was very suspicious that any PRs that might have fell out of our OKR goal were mysteriously missing 🙊🙈 Turns out it was due to me not having a complete understanding of the Python timedelta API. Here are the revised numbers: Average PR creation to merge latency: 566.4117283950616 minutes ( 9.440195473251027 hours) 11 / 116 PRs out of 24H SLA - 9.48% 44 / 116 PRs inside 35m SLA - 37.93% Skipped 8 # NumSamples = 108; Min = 16.80; Max = 8946.63 # Mean = 566.411728; Variance = 1833087.434142; SD = 1353.915593; Median 39.016667 # each ∎ represents a count of 1 16.8000 - 909.7833 [ 87]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 909.7833 - 1802.7667 [ 14]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 1802.7667 - 2695.7500 [ 0]: 2695.7500 - 3588.7333 [ 2]: ∎∎ 3588.7333 - 4481.7167 [ 2]: ∎∎ 4481.7167 - 5374.7000 [ 1]: ∎ 5374.7000 - 6267.6833 [ 0]: 6267.6833 - 7160.6667 [ 1]: ∎ 7160.6667 - 8053.6500 [ 0]: 8053.6500 - 8946.6333 [ 1]: ∎
,
Jun 27 2017
Minor amendment: the script skips PRs with negative timedeltas (PRs that were closed or merged before the commit landed) but those were still counted toward the total. The revised OKR numbers are: 11 / 108 PRs out of 24H SLA - 10.18% (score = 0.90) 44 / 108 PRs inside 35m SLA - 40.74% (score = 0.81)
,
Jun 27 2017
Jeff, can you determine from the data what number of minutes for the "50% of exports are under 35 minutes" goal would have resulted in a score of 0.5? I would like to repeat the goal, and am thinking that as long as we're expecting improvements (because of Gerrit), then re-calibrating so that 0.5 equals the last quarter is reasonable.
,
Jun 27 2017
From the data, the 50th percentile is 39.01 minutes and the 90th percentile is 1432.64 minutes (or 23.87 hours).
,
Jun 29 2017
,
Jun 30 2017
,
Jun 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/536bc68d1b98b0e4093b330df2fa850dd24f6da3 commit 536bc68d1b98b0e4093b330df2fa850dd24f6da3 Author: Jeff Carpenter <jeffcarp@chromium.org> Date: Fri Jun 30 22:49:42 2017 [WPT Export] Add more details to provisional export CL message This is to encourage people to not land CLs that aren't green upstream. Bug: 707006 Change-Id: Ied47f0846adc33ab7b884a4141d12cc97e415618 Reviewed-on: https://chromium-review.googlesource.com/553744 Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Cr-Commit-Position: refs/heads/master@{#483850} [modify] https://crrev.com/536bc68d1b98b0e4093b330df2fa850dd24f6da3/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_exporter.py
,
Jul 3 2017
,
Jul 3 2017
,
Jul 5 2017
,
Jul 6 2017
,
Jul 7 2017
,
Jul 13 2017
,
Jul 14 2017
,
Jul 14 2017
,
Jul 18 2017
,
Jul 19 2017
,
Jul 19 2017
,
Jul 20 2017
,
Jul 25 2017
,
Jul 26 2017
,
Jul 31 2017
,
Aug 1 2017
,
Aug 3 2017
,
Aug 3 2017
For context: this issue mainly tracked Exporter work in Q2 2017. I'm trying to wind this issue down and close it after resolving the open dependent issues. I'm not sure if it's worth having a new umbrella issue for Q3 - a search for Blink>Infra>Ecosystem and "[WPT Export]" is enough to see all open Exporter bugs. https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component:Blink%3EInfra%3EEcosystem%20%22[WPT%20Export]%22&sort=pri&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified
,
Aug 3 2017
,
Sep 15 2017
|
|||||||||||||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||||||||||||||||
Comment 1 by jeffcarp@chromium.org
, Mar 30 2017