An update about this: Last night we apparently had a case where vpython/pip ate the available harddisk space (22GB) in less than an hour.

I have now added an *hourly* cleanup cronjob

I'm not sure if we consider vpython a foundation thing or a client thing, at this point.

@agable / @estaab, can one of you find an owner for this?

An update on this. Recently I have been having this problem on at least one of our Windows testers.

What I see happening is that tests that use vpython (particularly browser_tests)

1) trigger an update
2) that update takes longer than either the timeout for the test, or the timeout for the vpython process to start.
3) The test and the python process are killed
4) No cleanup is done for the pip and vpython temp directories

-> As the tester have limited disk space: Full disk, which breaks even more tests.

My suggestion is the following: if a test executable (like browser_tests) use vpython for something, it will

A) Run the vpython install/update process as it starts, and wait until it is complete, no timeout
B) All other use of vpython in the tests disables the install/update functionality

Are you running the test within a chromium checkout? There's a gclient hook in chromium that tries to get all the vpython initialization out of the way before you start running tests:
https://codesearch.chromium.org/chromium/src/DEPS?rcl=9bd27cf64264b645d6793b3380be18921c87e217&l=2762

The tests are run run within a vivaldi checkout, with chromium as a submodule, but using the package build zip, extract_build script from the old buildbot system. In this environment Hooks are not run.

On Mac and Linux, the extract script (our version of it, at least) do run the vpython initialization from the shell script. (our version of the Hooks file also only run on mac and linux)

I have been able to run the vpython shell script in the new autobuilder that will likely be deployed in a few weeks, which is possible since the new system runs inside a Git Bash instance, not cmd (I would have preferred that the entire vpython update system was a Python script that could be called directly).

I am not sure about the bat file working properly, and since many of us are running inside Git Bash, the bat files have problems working.

Also, earlier today I _did_ run the vpython bat file manually (and I had previously tried with the bash script), but it seems that that tester instance still ran into trouble, and 52 tests that normally passes failed, and the next run deleted a number of pip and vpython temp directories.

So, it seems like it sometimes get in a state that triggers the online update in multiple tests.

As I said earlier: The test exe should call the updater itself, and actively disable updates when running it as part of tests.

Just FYI, in the current run (which remember, is on a machine where I had run vpython.bat manually a few hours ago) I just deleted 6GB of vpython/pip files from the temp dir (which had been basically empty of such files when I wrote the comment above), and while I was deleting those files, another 500+ MB of vpython/pip tempfiles were added.

Another update: While it seems I have now managed to the get right vpython location initialized (and updated), I am still seeing failing tests for 50+ browser_tests on that machine (no problem with the others).

In the case of ChromeNavigationBrowserTest.BlockLegacySubresources it timed out twice, then returned the following error message, because the disk was full of pip/vpython temp folders that had not been cleaned up:

----------------
[E2019-01-17T14:05:56.090700Z 5348 0 venv.go:942] Command (cwd=): [C:\Users\vivaldi\.vpython-root\8922c\Scripts\python -B -E -s -m pip install --isolated --compile --no-index --find-links C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\packages --requirement C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt --no-binary=:none: --only-binary=:all: --use-wheel]Process output:
Collecting boto==2.48.0 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 1))
Collecting coverage==4.3.4 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 2))
Collecting funcsigs==1.0.2 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 3))
Collecting google_compute_engine==2.6.2 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 4))
Collecting httplib2==0.10.3 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 5))
Collecting mock==2.0.0 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 6))
Collecting numpy==1.11.3 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 7))
Collecting opencv_python==2.4.13.2 (from -r C:\Users\vivaldi\AppData\Local\Temp\vpython_bootstrap236918013\requirements.txt (line 8))
Exception:
Traceback (most recent call last):
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\download.py", line 715, in unpack_file_url
    unpack_file(from_path, location, content_type, link)
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\utils\__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "C:\Users\vivaldi\.vpython-root\8922c\lib\site-packages\pip\utils\__init__.py", line 501, in unzip_file
    fp.write(data)
IOError: [Errno 28] No space left on device
----------------

It is possible that I just found the cause of the problem: I disabled the Python 2.7.15 installation on that machine, then all tests passed.

It would therefore seem that the vpython system from (an up-to-date) depot_tools have some kind of issue with Python 2.7.15

However, both the two other shards in this cluster, which still have 2.7.15 in the path, are passing all tests, so I am a little at a loss at understanding why shard 1 is behaving differently. Additionally, when this machine was running the new buildbot worker, using only Python 2.7.15 and the third_party depot_tools module, I did not see such problems (There are other new failures, though, but those do not appear to be python related)