I'm not sure what our process is in general for adding wheels. Is it sort of self-serve or is it the responsibility of Foundation team? Either way, do we have some kind of doc about adding wheels? (AFAIK current wheels were basically added first by Dan before he left, and then Robbie?)

Foundtation team responsibility. Quinten, did you want to try to add these to get familiar with it? I can show you how.

SGTM, I'll take a look and probably ask questions later.

I've now started looking at this.

AFAIK, the docs are at https://chromium.googlesource.com/infra/infra/+/HEAD/infra/tools/dockerbuild/README.md, and the process for adding new wheels is to edit the list at https://cs.chromium.org/chromium/infra/infra/tools/dockerbuild/wheel.py?l=393, and then run the dockerbuild utility which can (1) fetch wheels from PyPI (2) build CIPD packages (3) upload.

After starting, I've run into a possible issue, which is that not all of the versions listed above are necessarily available from PyPI:

 - mozinfo v1.0.0 is available, but not v0.10
 - mozdebug v0.1.1 is avilable, which I assume will work for v0.1?

Note, I took a look at the urllib3 v1.22 wheel that's already available, and in the METADATA file for the wheel it includes "Provides-Extra: secure", so it looks like that should contain those dependencies.

It looks like mozinfo .10 is available (have to click history tho. pypi only likes showing the latest):
  * https://pypi.org/project/mozinfo/#history
  * https://pypi.org/project/mozinfo/0.10/#files

Ah, I see -- Looks like this means I have to use "UniversalSource" rather than Universal for that one? And in order for that to work, I need to install docker? I'm trying this out now, and it seems to be doing a lot of magic.

yes, it would need to be universal source, and yes, you'd need docker installed (since the universalsource things get built under the linux docker image)

Note @lpz, the CIPD packages for these wheels are now uploaded and available for use by vpython.

I'll hold off on closing this CL until https:/crrev.com/c/1370547 is committed though.

Great, thank you very much!

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2d4a1ce1c601af68605d0a8472f77cb0cd1b8514

commit 2d4a1ce1c601af68605d0a8472f77cb0cd1b8514
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Wed Dec 12 20:28:34 2018

Add wheels required by WPT

Bug:  912269 
Change-Id: I977a15f360f92d580adb5a160341f8624610c533
Reviewed-on: https://chromium-review.googlesource.com/c/1370547
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19520}
[modify] https://crrev.com/2d4a1ce1c601af68605d0a8472f77cb0cd1b8514/infra/tools/dockerbuild/wheel.py
[modify] https://crrev.com/2d4a1ce1c601af68605d0a8472f77cb0cd1b8514/infra/tools/dockerbuild/wheels.md

From Luke:

> ...each of the new wheels gave an error such as:
> Collecting mozterm (from mozlog==3.8->-r /tmp/vpython_bootstrap841590524/requirements.txt (line 8))
>  Could not find a version that satisfies the requirement mozterm (from mozlog==3.8->-r > /tmp/vpython_bootstrap841590524/requirements.txt (line 8)) (from versions: )
> No matching distribution found for mozterm (from mozlog==3.8->-r /tmp/vpython_bootstrap841590524/requirements.txt (line 8))
>The error sounds like mozlog (which was added in the above CL) has its own dependency (mozterm) that isn't available...

So, I guess we probably want to always list all transitive dependencies explicitly in all vpython files, and we want to provide wheels for all packages that are transitively depended on.

(Robbie, is that right?

Using pip install to check the dependencies it seems like they are:

html5lib v1.0.1: six-1.12.0, webencodings-0.5.1
mozinfo v0.10: six-1.12.0, mozfile-2.0.0
mozlog v3.8: six-1.12.0, blessings-1.7, mozterm-1.0.0
mozdebug v0.1: mozfile-2.0.0 mozinfo-1.0.0 six-1.12.0
mozprocess v0.26: mozfile-2.0.0, mozinfo-1.0.0, six-1.12.0

Deduplicated, the additional packages are:

blessings-1.7
mozfile-2.0.0
mozterm-1.0.0
mozinfo-1.0.0
six-1.12.0 
webencodings-0.5.1

Universal wheels are available for all of these ... so if my understanding is correct, we just need to upload wheels for those (https://crrev.com/c/1388004) and all of those need to be listed explicitly in the vpython file.

Wait a minute, when installing the other mozbase packages, mozinfo 1.0.0 is fetched rather than mozinfo 0.10; but actually looking at the metadata for the packages, the version of mozinfo isn't specified and presumably mozinfo-0.10 is sufficient for all -- furthermore six 1.10 seems to be sufficient... so the set of extra packages to install could be:

  blessings-1.7
  mozfile-2.0.0
  mozterm-1.0.0
  webencodings-0.5.1

and the full list of dependencies to list in the vpython file for wpt could perhaps be:

  # Direct dependencies
  html5lib-1.0.1
  mozinfo-0.10
  mozlog-3.8
  mozdebug-0.1
  mozprocess-0.26
  # Transitive dependencies
  blessings-1.7
  mozfile-2.0.0
  mozterm-1.0.0
  six-1.10.0 
  webencodings-0.5.1

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1bb19ce2e6ddd0ecee05ae9e00aca6f1c95d7e67

commit 1bb19ce2e6ddd0ecee05ae9e00aca6f1c95d7e67
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Fri Dec 21 00:29:15 2018

Add additional wheel dependencies for WPT

These are transitive dependencies of the packages
previously added for  https://crbug.com/912269 .

Bug:  912269 
Change-Id: I37421a33dae883645162ca22c1470ac706830cd6
Reviewed-on: https://chromium-review.googlesource.com/c/1388004
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19696}
[modify] https://crrev.com/1bb19ce2e6ddd0ecee05ae9e00aca6f1c95d7e67/infra/tools/dockerbuild/wheel.py
[modify] https://crrev.com/1bb19ce2e6ddd0ecee05ae9e00aca6f1c95d7e67/infra/tools/dockerbuild/wheels.md