isolateserver.py buffer whole objects into memory instead of streaming |
|||
Issue descriptionFollowing instructions here on Windows: https://www.chromium.org/developers/testing/isolated-testing/for-swes#TOC-Download-binaries-from-a-try-job And hit an error: PS C:\src\chrome\src\tools\swarming_client> python isolateserver.py download -I https://isol ateserver.appspot.com --namespace default-gzip -s cbc757f280a514780e0759988bd2f63ca7451952 - -target bar 93188 2018-03-21 22:25:27.799 E: Symlink support is not enabled 442 files remaining... Traceback (most recent call last): File "isolateserver.py", line 2072, in <module> sys.exit(main(sys.argv[1:])) File "isolateserver.py", line 2064, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "C:\src\chrome\src\tools\swarming_client\third_party\depot_tools\subcommand.py", line 251, in execute return command(parser, args[1:]) File "isolateserver.py", line 1943, in CMDdownload use_symlinks=options.use_symlinks) File "isolateserver.py", line 1688, in fetch_isolated with cache.getfileobj(digest) as srcfileobj: File "isolateserver.py", line 1056, in getfileobj return io.BytesIO(d) MemoryError Sending the crash report ... done. Report URL: https://isolateserver.appspot.com/restricted/ereporter2/errors/5730927862874112 Process exited due to exception
,
May 20 2018
This popped up to me today when trying to download a Chrome-based isolate. I think this affects the version of python being shipped with depot_tools. I changed my paths to put my non-gclient 64-bit version ahead in the path order and the download worked. Probably we should update the Windows depot tools python to the 64-bit version. python isolateserver.py download -I https://isolateserver.appspot.com --namespace default-gzip -s 0ddf20d30befba5c3ac0b7bf5a1dea39d8977718 --target foo
,
Jun 16 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/64990b3adb9a8a531878ae29ac1d5c885984d49f commit 64990b3adb9a8a531878ae29ac1d5c885984d49f Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Sat Jun 16 13:11:02 2018 [client] Fix 3 issues for the cost of one CL - swarming.py reproduce uses a cache by default to fix usage on 32 bits. - swarming.py reproduce cleans up 'work' directory by default. - isolateserver.py uses a cache by default to fix usage on 32 bits. Sorry for lack of tests, I burned all my energy on the last CLs so there was none left. In practice, people who use these report bugs when they fail, so yay? Bug: 824832 , 801046 Change-Id: I097aa353cc8d0cdecd05902ad359c210c115f46c Reviewed-on: https://chromium-review.googlesource.com/1103118 Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> [modify] https://crrev.com/64990b3adb9a8a531878ae29ac1d5c885984d49f/client/isolateserver.py [modify] https://crrev.com/64990b3adb9a8a531878ae29ac1d5c885984d49f/client/swarming.py
,
Jun 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f4ed0f75179a60aabbc6c6dfab3ea7159de05fa1 commit f4ed0f75179a60aabbc6c6dfab3ea7159de05fa1 Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Sun Jun 17 02:36:52 2018 Roll src/tools/swarming_client/ 281c39019..9a518d097 (8 commits) https://chromium.googlesource.com/infra/luci/client-py.git/+log/281c390193ec..9a518d097dca $ git log 281c39019..9a518d097 --date=short --no-merges --format='%ad %ae %s' 2018-06-16 maruel [swarming] Implement NamedCache.cleanup() 2018-06-16 maruel [client] Fix 3 issues for the cost of one CL 2018-06-15 maruel [client] Fix cache LRU coherency 2018-06-15 maruel [client] Fix Memory CAC to be LRU, internal cleanups 2018-06-14 maruel [client] cache the named cache size; stop keeping empty caches 2018-06-14 maruel [client] Update lru.py and test 2018-06-13 maruel [client] Refactor the caches to stop using context manager 2018-06-13 iannucci [swarming] Actually apply pool based TaskTemplates. Created with: roll-dep src/tools/swarming_client TBR=qyearsley@chromium.org Bug: 801046 , 824832 Change-Id: Id2dd2bc2b0295d96726eb200f6e9a0cc8aea1eb7 Reviewed-on: https://chromium-review.googlesource.com/1103534 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> Cr-Commit-Position: refs/heads/master@{#567906} [modify] https://crrev.com/f4ed0f75179a60aabbc6c6dfab3ea7159de05fa1/DEPS
,
Jun 17 2018
,
Jun 21 2018
Issue 735170 has been merged into this issue. |
|||
►
Sign in to add a comment |
|||
Comment 1 by mar...@chromium.org
, Mar 22 2018Summary: isolateserver.py buffer whole objects into memory instead of streaming (was: Download swarming binaries from a try job fails on Windows)