New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 877538 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 28
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

CIPD infra go bootstrap fails sometimes when renaming some file

Project Member Reported by tandrii@chromium.org, Aug 24

Issue description

From https://logs.chromium.org/logs/infra/buildbucket/cr-buildbucket.appspot.com/8937296195549818528/+/steps/init_infra_go_env/0/stdout

Searching for a bundle with dependencies in CIPD...
--------------------------------------------------------------------------------
CIPD package name:    infra/go-deps
CIPD package version: go_deps_lock:08c09e39159275d07f95bcce59365a0422ba1f98
--------------------------------------------------------------------------------
[P2836 09:15:06.094 action_plan.go:139 I] In root:
[P2836 09:15:06.094 action_plan.go:145 I]   to install:
[P2836 09:15:06.094 action_plan.go:147 I]     infra/go-deps:hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC
[P2836 09:15:06.094 client.go:550 I] cipd: using instance cache at "e:\\b\\swarm_slave\\cipd_cache\\cache\\instances"
[P2836 09:15:06.094 client.go:1267 I] cipd: instance cache hit for infra/go-deps:hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC
[P2836 09:15:06.116 deployer.go:253 I] Deploying infra/go-deps:hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC into e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\.vendor(/)
[P2836 09:15:06.126 reader.go:341 I] cipd: about to extract 183.3 MB (8254 files)
[P2836 09:15:06.130 reader.go:380 I] cipd: extracting - 0%
[P2836 09:15:08.130 reader.go:380 I] cipd: extracting - 20%
[P2836 09:15:10.130 reader.go:380 I] cipd: extracting - 84%
[P2836 09:15:10.711 reader.go:380 I] cipd: extracting - 100%
[P2836 09:15:11.969 fs.go:384 W] fs: failed to rename("e:\\b\\swarm_slave\\w\\ir\\cache\\infra_gclient_with_go\\infra\\go\\.vendor\\.cipd\\pkgs\\0\\hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC\\src\\github.com\\docker\\docker\\vendor\\github.com\\google\\certificate-transparency\\cpp\\third_party\\isec_partners\\openssl_hostname_validation.c", "e:\\b\\swarm_slave\\w\\ir\\cache\\infra_gclient_with_go\\infra\\go\\.vendor\\src\\github.com\\docker\\docker\\vendor\\github.com\\google\\certificate-transparency\\cpp\\third_party\\isec_partners\\openssl_hostname_validation.c") - The system cannot find the path specified.
[P2836 09:15:11.969 deployer.go:1048 W] Failed to move e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\.vendor\.cipd\pkgs\0\hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC\src\github.com\docker\docker\vendor\github.com\google\certificate-transparency\cpp\third_party\isec_partners\openssl_hostname_validation.c to e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\.vendor\src\github.com\docker\docker\vendor\github.com\google\certificate-transparency\cpp\third_party\isec_partners\openssl_hostname_validation.c: The system cannot find the path specified.
[P2836 09:15:11.969 deployer.go:294 W] Deploy aborted, cleaning up e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\.vendor\.cipd\pkgs\0\hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC
[P2836 09:15:13.148 client.go:1516 E] Failed to install infra/go-deps:hnLM_Pf_1uEOf7v5wMPuNPHbKFg-h_3m2xppcRU31MMC - The system cannot find the path specified.
Error: failed to update packages, see the log.
Failed to install dependencies from the bundle. See logs.
Traceback (most recent call last):
  File "e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\env.py", line 101, in <module>
    main()
  File "e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\env.py", line 83, in main
    deps_only=args.deps_only)
  File "e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\bootstrap.py", line 533, in prepare_go_environ
    bootstrap(layout, logging.INFO)
  File "e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\bootstrap.py", line 495, in bootstrap
    p, layout.toolset_root, force=toolset_updated)
  File "e:\b\swarm_slave\w\ir\cache\infra_gclient_with_go\infra\go\bootstrap.py", line 384, in update_vendor_packages
    subprocess.check_call(cmd, stdout=sys.stderr, env=env)
  File "e:\b\swarm_slave\w\ir\cipd_bin_packages\bin\Lib\subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['e:\\b\\swarm_slave\\w\\ir\\cache\\vpython\\5fbc04\\Scripts\\python.exe', '-u', 'e:\\b\\swarm_slave\\w\\ir\\cache\\infra_gclient_with_go\\infra\\go\\deps.py', '--workspace', 'e:\\b\\swarm_slave\\w\\ir\\cache\\infra_gclient_with_go\\infra\\go', '--goroot', 'e:\\b\\swarm_slave\\w\\ir\\cache\\infra_gclient_with_go\\golang\\go', 'install', '--update-out', 'e:\\b\\swarm_slave\\w\\ir\\cache\\infra_gclient_with_go\\infra\\go\\tmpqv7u82\\deps_updated.json']' returned non-zero exit status 1
step returned non-zero exit code: 1
 
This is MAX_PATH problem. "e:\\b\\swarm_slave\\...\\openssl_hostname_validation.c" is 263 bytes long, MAX_PATH is 260.

I was afraid we'll be getting more of these after switching to SHA256 hash, which is slightly longer (we were at the edge before).

A short term solution for this concrete package may be omitting cpp\third_party\... from the package by modifying https://chromium.googlesource.com/infra/infra/+/master/go/deps.py#508 (I'm 90% sure it is not actually used during compilation).
Labels: -Pri-2 Foundation-Troopers Pri-1
Yeah, depending on the bot, it flakily fails. Retrying helps for now. Bumping priority and sticking into troopers queue.
Cc: rpl@google.com
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 28

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8cd5ea18e9c7c73d0e698d658066581b65d4e9a9

commit 8cd5ea18e9c7c73d0e698d658066581b65d4e9a9
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Aug 28 18:17:43 2018

[go] Exclude path breaking MAX_PATH from the bundle, bump deps.lock.

Also add a mirror for contrib.go.opencensus.io/exporter/stackdriver.

Updated repos:
https://code.googlesource.com/gocloud.git/+log/290bef816a5d..f47691a5dd86
https://chromium.googlesource.com/external/github.com/Microsoft/go-winio.git/+log/78a084671df1..97e4973ce50b
https://chromium.googlesource.com/external/github.com/chromedp/cdproto.git/+log/146d9b5fb76b..d0b1c2b74a0f
https://chromium.googlesource.com/external/github.com/google/skylark.git/+log/5b0e788dfd99..4956ce9eeded
https://chromium.googlesource.com/external/github.com/google/uuid.git/+log/dec09d789f3d..d460ce9f8df2
https://chromium.googlesource.com/external/github.com/gopherjs/gopherjs.git/+log/89baedc74dd7..0210a2f0f73c
https://chromium.googlesource.com/external/github.com/gorilla/websocket.git/+log/3ff3320c2a17..66b9c49e59c6
https://chromium.googlesource.com/external/github.com/mailru/easyjson.git/+log/03f2033d19d5..60711f1a8329
https://chromium.googlesource.com/external/github.com/mitchellh/go-homedir.git/+log/58046073cbff..ae18d6b8b320
https://chromium.googlesource.com/external/github.com/census-instrumentation/opencensus-go.git/+log/bfa2d767dc63..91168ff30a42
https://go.googlesource.com/net.git/+log/922f4815f713..8a410e7b638d
https://go.googlesource.com/tools.git/+log/ef27ca3af5e1..6cd1fcedba52
https://code.googlesource.com/google-api-go-client.git/+log/0873c9a91f71..7954115fcf34
https://chromium.googlesource.com/external/github.com/golang/appengine.git/+log/4216e58b9158..03cac3b07182
https://chromium.googlesource.com/external/github.com/grpc/grpc-go.git/+log/9907ae0e57ce..59dd9b3f1997

R=tandrii@chromium.org
BUG= 877538 

Change-Id: Iacb15b84c632024c257b171253fd32c0d310c4ca
Reviewed-on: https://chromium-review.googlesource.com/1193062
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/8cd5ea18e9c7c73d0e698d658066581b65d4e9a9/go/deps.lock
[modify] https://crrev.com/8cd5ea18e9c7c73d0e698d658066581b65d4e9a9/go/deps.py
[modify] https://crrev.com/8cd5ea18e9c7c73d0e698d658066581b65d4e9a9/go/deps.yaml

Once https://ci.chromium.org/p/infra-internal/g/infra-packagers/console finished building the above revision, this will be resolved.
Owner: vadimsh@chromium.org
Status: Fixed (was: Available)
Thanks to Vadim, this is fixed now.

Sign in to add a comment