New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 795146 link

Starred by 2 users
Status: Fixed
Owner:
█ robert...@chromium.org
Last visit > 30 days ago
Closed: Jun 2018
Cc:
jchin...@chromium.org
█ iannucci@chromium.org
tandrii@chromium.org
brat...@opera.com
most...@vewd.com
st...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

gclient runhooks fails in windows tryjobs on swarming

Project Member Reported by robert...@chromium.org, Dec 15 2017 
At least findit's compile recipe seems affected.

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win_chromium_variable/5

fails the gclient runhooks step with the following error:
subprocess.CalledProcessError: Command '['E:\\b\\s\\w\\ir\\cipd_bin_packages\\vpython.exe', 'e:\\b\\s\\w\\ir\\cache\\builder\\win_layout\\src\\third_party\\depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json', 'e:\\b\\s\\w\\ir\\cache\\builder\\win_layout\\src\\build\\win_toolchain.json', 'a9e1098bba66d2acccc377d5ee81265910f29272']' returned non-zero exit status 1


Note that this is blocking our migration to LUCI for windows builds.

Comment 1 by tandrii@chromium.org, Dec 15 2017

Cc: jchin...@chromium.org
+jchinlee I think this will block your Win (dbg) work too.

Comment 2 by estaab@chromium.org, Jan 2 2018

Status: Available (was: Untriaged)

Comment 3 by tandrii@chromium.org, Jan 3 2018 

We've lost the logs again - the above build's Log stream has no annotation entries :(
Project Member

Comment 4 by bugdroid1@chromium.org, Jan 4 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/1adc91f99fe0aa7675dac6a44f2f320be3ea036f

commit 1adc91f99fe0aa7675dac6a44f2f320be3ea036f
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Jan 04 18:27:14 2018

[download_from_google_storage] Use vpython to invoke gsutil.py.

This will allow all uses of this class to correctly move into
gsutil's personal vpython environment. In particular, the
get_toolchain_if_necessary script is pretty sneaky, and adds
depot_tools to sys.path to import and use this class.  This
means that if get_toolchain_if_necessary is invoked in a vanilla
vpython environment (or an environment otherwise incompatible with
gsutil.py), it's use of this class will fail.

R=hinoka@chromium.org, tandrii@chromium.org

Bug:  795146 
Change-Id: I87309a8b878b465cda7f53db0a41f2f9e6da6f50
Reviewed-on: https://chromium-review.googlesource.com/849663
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/1adc91f99fe0aa7675dac6a44f2f320be3ea036f/download_from_google_storage.py

Comment 5 by iannucci@chromium.org, Jan 5 2018

Status: Fixed (was: Available)
TL;DR: this was actually a missing google storage permission.

Comment 7 by most...@vewd.com, Jan 5 2018

Cc: most...@vewd.com brat...@opera.com
FYI, this seemed to cause some trouble for us downstream, not sure if it's specific to our setup.  Removing ~/.vpython* seems to help.

Comment 8 by tandrii@chromium.org, Jan 5 2018 

can you elaborate on your setup and what specific problems are you are experiencing?

Comment 9 by iannucci@chromium.org, Jan 5 2018 

Yes, I think this is due to cruft left over from an old version of vpython; I've landed a fix for it here (https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/851263) and will be rolling it out to depot_tools shortly.

TL;DR: vpython looks for a spec file called `.vpython` by walking up the directory tree. Old versions of vpython used `~/.vpython/` as the place to put vpython virtual environments. `~/.vpython/` is a directory and not a valid spec file. >_<

Comment 10 by most...@vewd.com, Jan 5 2018 

Excerpt from our (downstream) logs follow.  depot_tools is up to date, we run some hooks from our build configuration but don't use gclient at all during builds.

09:26:11 Failed to fetch file gs://chromium-binutils/0cb5726d9701f8be6a81b199899de1de552922f2 for /home/user/WS/work/chromium/src/third_party/binutils/Linux_x64/binutils.tar.bz2. [Err: [E2018-01-05T08:26:11.544499Z 2677 0 annotate.go:242] original error: read /home/user/.vpython: is a directory
09:26:11 [E2018-01-05T08:26:11.544535Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544550Z 2677 0 annotate.go:242] goroutine 1:
09:26:11 [E2018-01-05T08:26:11.544568Z 2677 0 annotate.go:242] #0 go.chromium.org/luci/vpython/spec/load.go:60 - spec.Load()
09:26:11 [E2018-01-05T08:26:11.544582Z 2677 0 annotate.go:242]   reason: failed to load file from: /home/user/.vpython
09:26:11 [E2018-01-05T08:26:11.544594Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544606Z 2677 0 annotate.go:242] #1 go.chromium.org/luci/vpython/spec/load.go:211 - spec.(*Loader).LoadForScript()
09:26:11 [E2018-01-05T08:26:11.544622Z 2677 0 annotate.go:242] #2 go.chromium.org/luci/vpython/options.go:137 - vpython.(*Options).ResolveSpec()
09:26:11 [E2018-01-05T08:26:11.544637Z 2677 0 annotate.go:242]   reason: failed to load spec for script: {/home/user/WS/depot_tools/gsutil.py}
09:26:11 [E2018-01-05T08:26:11.544656Z 2677 0 annotate.go:242]   internal reason: isModule(false)
09:26:11 [E2018-01-05T08:26:11.544674Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544692Z 2677 0 annotate.go:242] #3 go.chromium.org/luci/vpython/options.go:86 - vpython.(*Options).resolve()
09:26:11 [E2018-01-05T08:26:11.544712Z 2677 0 annotate.go:242]   reason: failed to resolve Python script
09:26:11 [E2018-01-05T08:26:11.544727Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544740Z 2677 0 annotate.go:242] #4 go.chromium.org/luci/vpython/run.go:57 - vpython.Run()
09:26:11 [E2018-01-05T08:26:11.544757Z 2677 0 annotate.go:242]   reason: could not resolve options
09:26:11 [E2018-01-05T08:26:11.544774Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544787Z 2677 0 annotate.go:242] #5 go.chromium.org/luci/vpython/application/application.go:266 - application.(*application).mainImpl()
09:26:11 [E2018-01-05T08:26:11.544809Z 2677 0 annotate.go:242] #6 go.chromium.org/luci/vpython/application/application.go:354 - application.(*Config).Main.func1()
09:26:11 [E2018-01-05T08:26:11.544828Z 2677 0 annotate.go:242] #7 go.chromium.org/luci/vpython/application/support.go:45 - application.run()
09:26:11 [E2018-01-05T08:26:11.544842Z 2677 0 annotate.go:242] #8 go.chromium.org/luci/vpython/application/application.go:353 - application.(*Config).Main()
09:26:11 [E2018-01-05T08:26:11.544857Z 2677 0 annotate.go:242] #9 vpython/main.go:92 - main.mainImpl()
09:26:11 [E2018-01-05T08:26:11.544871Z 2677 0 annotate.go:242] #10 vpython/main.go:98 - main.main()
09:26:11 [E2018-01-05T08:26:11.544885Z 2677 0 annotate.go:242] #11 runtime/proc.go:185 - runtime.main()
09:26:11 [E2018-01-05T08:26:11.544899Z 2677 0 annotate.go:242] #12 runtime/asm_amd64.s:2337 - runtime.goexit()
09:26:11 ]
09:26:11 0> Failed to fetch file gs://chromium-binutils/0cb5726d9701f8be6a81b199899de1de552922f2 for /home/user/WS/work/chromium/src/third_party/binutils/Linux_x64/binutils.tar.bz2, skipping. [Err: [E2018-01-05T08:26:11.544499Z 2677 0 annotate.go:242] original error: read /home/user/.vpython: is a directory
09:26:11 [E2018-01-05T08:26:11.544535Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544550Z 2677 0 annotate.go:242] goroutine 1:
09:26:11 [E2018-01-05T08:26:11.544568Z 2677 0 annotate.go:242] #0 go.chromium.org/luci/vpython/spec/load.go:60 - spec.Load()
09:26:11 [E2018-01-05T08:26:11.544582Z 2677 0 annotate.go:242]   reason: failed to load file from: /home/user/.vpython
09:26:11 [E2018-01-05T08:26:11.544594Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544606Z 2677 0 annotate.go:242] #1 go.chromium.org/luci/vpython/spec/load.go:211 - spec.(*Loader).LoadForScript()
09:26:11 [E2018-01-05T08:26:11.544622Z 2677 0 annotate.go:242] #2 go.chromium.org/luci/vpython/options.go:137 - vpython.(*Options).ResolveSpec()
09:26:11 [E2018-01-05T08:26:11.544637Z 2677 0 annotate.go:242]   reason: failed to load spec for script: {/home/user/WS/depot_tools/gsutil.py}
09:26:11 [E2018-01-05T08:26:11.544656Z 2677 0 annotate.go:242]   internal reason: isModule(false)
09:26:11 [E2018-01-05T08:26:11.544674Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544692Z 2677 0 annotate.go:242] #3 go.chromium.org/luci/vpython/options.go:86 - vpython.(*Options).resolve()
09:26:11 [E2018-01-05T08:26:11.544712Z 2677 0 annotate.go:242]   reason: failed to resolve Python script
09:26:11 [E2018-01-05T08:26:11.544727Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544740Z 2677 0 annotate.go:242] #4 go.chromium.org/luci/vpython/run.go:57 - vpython.Run()
09:26:11 [E2018-01-05T08:26:11.544757Z 2677 0 annotate.go:242]   reason: could not resolve options
09:26:11 [E2018-01-05T08:26:11.544774Z 2677 0 annotate.go:242] 
09:26:11 [E2018-01-05T08:26:11.544787Z 2677 0 annotate.go:242] #5 go.chromium.org/luci/vpython/application/application.go:266 - application.(*application).mainImpl()
09:26:11 [E2018-01-05T08:26:11.544809Z 2677 0 annotate.go:242] #6 go.chromium.org/luci/vpython/application/application.go:354 - application.(*Config).Main.func1()
09:26:11 [E2018-01-05T08:26:11.544828Z 2677 0 annotate.go:242] #7 go.chromium.org/luci/vpython/application/support.go:45 - application.run()
09:26:11 [E2018-01-05T08:26:11.544842Z 2677 0 annotate.go:242] #8 go.chromium.org/luci/vpython/application/application.go:353 - application.(*Config).Main()
09:26:11 [E2018-01-05T08:26:11.544857Z 2677 0 annotate.go:242] #9 vpython/main.go:92 - main.mainImpl()
09:26:11 [E2018-01-05T08:26:11.544871Z 2677 0 annotate.go:242] #10 vpython/main.go:98 - main.main()
09:26:11 [E2018-01-05T08:26:11.544885Z 2677 0 annotate.go:242] #11 runtime/proc.go:185 - runtime.main()
09:26:11 [E2018-01-05T08:26:11.544899Z 2677 0 annotate.go:242] #12 runtime/asm_amd64.s:2337 - runtime.goexit()
09:26:11 ]
09:26:11 Downloading 1 files took 0.075811 second(s)
09:26:11 Downloading /home/user/WS/work/chromium/src/third_party/binutils/Linux_x64/binutils.tar.bz2
09:26:11 Traceback (most recent call last):
09:26:11   File "src/third_party/binutils/download.py", line 130, in <module>
09:26:11     sys.exit(main(sys.argv[1:]))
09:26:11   File "src/third_party/binutils/download.py", line 117, in main
09:26:11     return FetchAndExtract(arch)
09:26:11   File "src/third_party/binutils/download.py", line 82, in FetchAndExtract
09:26:11     '-s', sha1file])
09:26:11   File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
09:26:11     raise CalledProcessError(retcode, cmd)
09:26:11 subprocess.CalledProcessError: Command '['download_from_google_storage', '--no_resume', '--no_auth', '--bucket', 'chromium-binutils', '-s', '/home/user/WS/work/chromium/src/third_party/binutils/Linux_x64/binutils.tar.bz2.sha1']' returned non-zero exit status 1
09:26:11 depot_tools fetch /home/user/WS/depot_tools/.git/FETCH_HEAD at 2018-01-05T08:25:15.390347
09:26:11 Running hook: remove_stale_pyc_files
09:26:11 Running hook: binutils
09:26:11 Traceback (most recent call last):
09:26:11   File "./build/run_opera_hooks.py", line 192, in <module>
09:26:11     RunOperaHooks(sys.argv)
09:26:11   File "./build/run_opera_hooks.py", line 141, in RunOperaHooks
09:26:11     subprocess.check_call(hook['action'], cwd=chromium)
09:26:11   File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
09:26:11     raise CalledProcessError(retcode, cmd)
09:26:11 subprocess.CalledProcessError: Command '['python', 'src/third_party/binutils/download.py']' returned non-zero exit status 1

Comment 11 by iannucci@chromium.org, Jan 5 2018 

Yep, that's the one. I've uploaded the depot_tools roll here:

https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/851847

Comment 12 by iannucci@chromium.org, Jan 5 2018 

(sorry for the trouble! this was a latent bug)
Project Member

Comment 13 by bugdroid1@chromium.org, Jan 5 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/df27bf6f0034ac718433f8c745f9229514aca960

commit df27bf6f0034ac718433f8c745f9229514aca960
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Jan 05 17:51:16 2018

[vpython] Roll vpython to latest version.

This should pick up a fix for where vpython attempts to read
`~/.vpython` folders as spec files (and fails).

R=tandrii@chromium.org

Bug:  795146 
Change-Id: I261944c3d7e3e8703adc4986c88ae742b801d457
Reviewed-on: https://chromium-review.googlesource.com/851847
Commit-Queue: Aaron Gable <agable@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/df27bf6f0034ac718433f8c745f9229514aca960/cipd_manifest.txt
Project Member

Comment 14 by bugdroid1@chromium.org, Jan 9 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/goma/client/+/b97b63efdc84657f0fabccfd08be29cc5bd2cd70

commit b97b63efdc84657f0fabccfd08be29cc5bd2cd70
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Jan 09 00:58:03 2018

Comment 15 by robert...@chromium.org, Jan 17 2018

Status: Available (was: Fixed)
This is happening again (presumably for a different reason)

https://ci.chromium.org/p/chromium/builders/luci.chromium.findit/findit_variable/256

Reopening.

Comment 16 by robert...@chromium.org, Jan 18 2018

Owner: robert...@chromium.org
Status: Assigned (was: Available)
I'll dig into this.
Project Member

Comment 17 by bugdroid1@chromium.org, Jun 14 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e59f04a19f9baafe06cd32351e858919b6980549

commit e59f04a19f9baafe06cd32351e858919b6980549
Author: Roberto Carrillo <robertocn@google.com>
Date: Thu Jun 14 00:20:33 2018

Enable native vpython on findit bucket's builders.

R=iannucci,tandrii

No-Try: True
Bug:  795146 
Change-Id: Ib19e24035fcf46d3563ece2eda7271bc1e176a03
Reviewed-on: https://chromium-review.googlesource.com/1100174
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#567059}
[modify] https://crrev.com/e59f04a19f9baafe06cd32351e858919b6980549/infra/config/global/cr-buildbucket.cfg

Comment 18 by robert...@chromium.org, Jun 25 2018

Status: Fixed (was: Assigned)
►

Sign in to add a comment