One could also look at all wpt-exporter logs, but better I think to just compare the histories of the two repos, that way the methodology is different and more likely to catch mistakes.

In light of 754643 I'm expanding the scope of this a bit. In addition to checking that the modified files match, we also need to see if all of the changes in the file were included, which isn't necessarily the case.

I think this is how I'd verify that each export is complete, where $CRCOMMIT and $WPTCOMMIT are the hopefully-matching commits in each repo:

In chromium:
git format-patch -1 $CRCOMMIT -- [list of files in LayoutTests excluding -expected.txt]

In wpt:
git checkout $WPTCOMMIT^ (commit on which we did the export)
git am -p6 [patch from above]
git diff $WPTCOMMIT

There should be no diff. Use git diff -w to ignore whitespace changes, perhaps.

jeffcarp@ already did it once two weeks ago: https://bugs.chromium.org/p/chromium/issues/detail?id=737898#c12

Perhaps he can weigh in to provide some details of the validation process and maybe we can write a script to automate it -- even in the ideal scenario importer/exporter run correctly, it'd be still great to have some different tools/instructions to conduct audit.

jeffarp@, can you share the scripts you used for that audit? I could run that once a month while updating stats so that we notice if new bugs are introduced.

It'd be great to have a script to do this. However, the way I did the audit was by doing a one-off modification of the exporter by:

- Increasing the Chromium commit window size to include all commits since the beginning
- Fetch 100% of GitHub PRs (this is now done via  bug 740175 , thanks robertma!)
- Adding logging of any commits that generate a non-empty patch

Then going one-by-one over the patches that failed and checking the current state of Chromium against WPT. I don't think I still have that exact patch but I can redo the modifications if needed.

Practically I think it would be possible and not super expensive to increase the default Chromium commit window for the Exporter to include all commits since the introduction of the exporter and add some extra verification to confirm the changes are the same between WPT and Chromium. That way we wouldn't need an extra out-of-band process for auditing.

More data loss in  issue 756428 .

In https://bugs.chromium.org/p/chromium/issues/detail?id=756428#c6 there's a tricky situation that an audit will have to deal with. When we find exports that don't match, it's not enough to just apply the patch on the tip of web-platform-tests and assume it's good if there are no conflicts. Rather, the first export has to be reverted, and then we need to do the export again. Otherwise we will be leaving some changes that were never intended to land.

> Practically I think it would be possible and not super expensive to increase the default Chromium commit window for the Exporter to include all commits since the introduction of the exporter and add some extra verification to confirm the changes are the same between WPT and Chromium. That way we wouldn't need an extra out-of-band process for auditing.

If one kept trying to apply patches to ToT, then eventually older Chromium commits would begin conflicting. It might happen rather quickly even in some cases. I suppose, however, that one could have as part of the script a commit up to which we've already verified that everything is good, and we just increase that once in a while if needed to silence problems?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9f6322feaaa70f698737ef53715f3d9160e8f828

commit 9f6322feaaa70f698737ef53715f3d9160e8f828
Author: Austin James Ahlstrom <aahlstrom@google.com>
Date: Fri Sep 01 21:20:32 2017

Reland "Port XHR access-control LayoutTests to Web Platform Tests"

This is a reland of a03f32888a033a56d15b552f888bad33f5a71ae5.
In the original CL, wpt-exporter merged the provisional PR on GitHub
prematurely at patchset 3, even though the CL had not landed.
The final checked-in patchset 12 was therefore not exported, and the
change between PS3 and PS12 was eventually clobbered by the importer:
https://chromium.googlesource.com/chromium/src/+log/master/third_party/WebKit/LayoutTests/external/wpt/XMLHttpRequest/allow-lists-starting-with-comma.htm

Original change's description:
> Port XHR access-control LayoutTests to Web Platform Tests
> 
> Porting access-control-allow-lists-starting-with-comma.html
> from LayoutTests to Web Platform Tests
> 
> Bug:  745385 
> Change-Id: I035fe6b26463ff9e2bcf70af2be80464849766d4
> Reviewed-on: https://chromium-review.googlesource.com/571249
> Commit-Queue: Austin James Ahlstrom <aahlstrom@google.com>
> Reviewed-by: Takeshi Yoshino <tyoshino@chromium.org>
> Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#489850}

Bug: 754619,  745385 
Change-Id: I5dfdb8c9ccb4fe4fcae94c237eaee3196103e7f8
Reviewed-on: https://chromium-review.googlesource.com/648046
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#499302}
[modify] https://crrev.com/9f6322feaaa70f698737ef53715f3d9160e8f828/third_party/WebKit/LayoutTests/external/wpt/XMLHttpRequest/allow-lists-starting-with-comma.htm
[modify] https://crrev.com/9f6322feaaa70f698737ef53715f3d9160e8f828/third_party/WebKit/LayoutTests/external/wpt/XMLHttpRequest/resources/access-control-allow-lists.py

Ecosystem infra bug triage ping: is this still on your roadmap Robert? Is Pri-2 still the right priority?

Yup, this is on my Q4 plan, and P2 sounds right.

This is listed under "Resolve long-standing issues that affect the reliability of WPT in Blink" in Ecosystem Infra Q1 OKRs:
https://docs.google.com/document/d/1ojlLDiRruI4e95Z7JQ10JQPRMiQHd_40rBkYCL6IJqw/edit?usp=sharing

(Came up in triage of P2 issues older than 60 days.)

@robertma what is the current state of this?

Pinging again for an update since this is P2.

YAP (yet another ping) robertma@chromium.org

Ping!