Issue metadata
Sign in to add a comment
|
fuchsia_x64 tryserver failing gclient runhooks, blocking CQ |
||||||||||||||||||||||
Issue descriptionFailing almost all recent builds: https://ci.chromium.org/buildbot/tryserver.chromium.linux/fuchsia_x64/?limit=200 Example: https://ci.chromium.org/buildbot/tryserver.chromium.linux/fuchsia_x64/36239 Log: ________ running 'vpython src/build/fuchsia/update_sdk.py 1a7e46683da9638f5bff44930f307043927c665f' in '/b/c/b/linux' Copying gs://fuchsia/sdk/linux-amd64/1a7e46683da9638f5bff44930f307043927c665f... / [0 files][ 0.0 B/126.9 MiB] - - [0 files][126.3 MiB/126.9 MiB] - [1 files][126.9 MiB/126.9 MiB] \ Operation completed over 1 objects/126.9 MiB. Downloading SDK 1a7e46683da9638f5bff44930f307043927c665f... Creating directory /b/c/b/linux/src/third_party/fuchsia-sdk Traceback (most recent call last): File "src/build/fuchsia/update_sdk.py", line 65, in <module> sys.exit(main()) File "src/build/fuchsia/update_sdk.py", line 56, in main tarfile.open(mode='r:gz', fileobj=f).extractall(path=output_dir) File "/usr/lib/python2.7/tarfile.py", line 1678, in open return func(name, filemode, fileobj, **kwargs) File "/usr/lib/python2.7/tarfile.py", line 1727, in gzopen **kwargs) File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen return cls(name, mode, fileobj, **kwargs) File "/usr/lib/python2.7/tarfile.py", line 1574, in __init__ self.firstmember = self.next() File "/usr/lib/python2.7/tarfile.py", line 2338, in next raise ReadError("empty file") tarfile.ReadError: empty file Error: Command 'vpython src/build/fuchsia/update_sdk.py 1a7e46683da9638f5bff44930f307043927c665f' returned non-zero exit status 1 in /b/c/b/linux step returned non-zero exit code: 2 Troopers or sheriffs, could you please diagnose what's going on? Is it pointing to a bad hash, did something get corrupted in cloud storage, etc.?
,
Dec 15 2017
Looking. It is reproducible locally.
,
Dec 15 2017
The gs file sha1 matches its name, so it hasn't been changed. I wonder how this DEPS roll passed CQ...
,
Dec 15 2017
The file looks like valid tar.gz tarball and I can extract it successfully via python, just not via update_sdk.py script for some reason...
,
Dec 15 2017
I think there were changes in gsutil.py that caused this. I have a fix, will upload it shortly
,
Dec 15 2017
Yep. The fix: https://chromium-review.googlesource.com/c/chromium/src/+/830257. I believe it was ultimately caused by https://chromium.googlesource.com/chromium/tools/depot_tools/+/2e8d8348b8574f06c26dbf3ef959b5df11ba5148 which appears to have changed the way gsutil does 'cp'. I suspect it does 'mv' at some point now, but update_sdk.py keeps holding reference to old inode, and discovers that it is empty.
,
Dec 15 2017
,
Dec 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/48fdf63300aae84f8159c22831df8cff24b5d984 commit 48fdf63300aae84f8159c22831df8cff24b5d984 Author: Vadim Shtayura <vadimsh@chromium.org> Date: Fri Dec 15 19:45:35 2017 Potential fix for the failing Fuchsia SDK update hook. It appears using external process to write to a file we hold an open handle for isn't 100% reliable (I suspect we need an fsync somewhere or maybe gsutil is doing 'mv' somewhere and writes to another inode, or something like that). Reopening the file before reading from it fixes the bug, at least locally for me. R=scottmg@chromium.org, kbr@chromium.org BUG= 795330 Change-Id: I498540ec62439c02243b84a5577c4b225ba5e1f3 Reviewed-on: https://chromium-review.googlesource.com/830257 Commit-Queue: Sergey Ulanov <sergeyu@chromium.org> Reviewed-by: Sergey Ulanov <sergeyu@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#524439} [modify] https://crrev.com/48fdf63300aae84f8159c22831df8cff24b5d984/build/fuchsia/update_sdk.py
,
Dec 15 2017
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by kbr@chromium.org
, Dec 15 2017