cros chrome-sdk fails due to bad cache with confusing error |
||||
Issue description
Manually running `rm -rf build/cros_cache/common/*` fixed the issue, but the error message printed by `cros chrome-sdk` did not make the issue obvious.
$ cros chrome-sdk --board=kevin --log-level=debug
13:45:45: DEBUG: Cache dir lookup.
13:45:45: DEBUG: Configured cache_dir to '/work2/chrome/src/build/cros_cache'
13:45:45: ERROR: cros chrome-sdk failed before completing.
cros: Unhandled exception:
Traceback (most recent call last):
File "/work2/chrome/src/third_party/chromite/bin/cros", line 169, in <module>
DoMain()
File "/work2/chrome/src/third_party/chromite/bin/cros", line 165, in DoMain
commandline.ScriptWrapperMain(FindTarget)
File "/work2/chrome/src/third_party/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
ret = target(argv[1:])
File "/work2/chrome/src/third_party/chromite/scripts/cros.py", line 66, in main
code = _RunSubCommand(subcommand)
File "/work2/chrome/src/third_party/chromite/scripts/cros.py", line 51, in _RunSubCommand
return subcommand.Run()
File "/work2/chrome/src/third_party/chromite/cli/cros/cros_chrome_sdk.py", line 1162, in Run
use_external_config=self.options.use_external_config)
File "/work2/chrome/src/third_party/chromite/cli/cros/cros_chrome_sdk.py", line 137, in __init__
self.gs_ctx = gs.GSContext(cache_dir=cache_dir, init_boto=internal)
File "/work2/chrome/src/third_party/chromite/lib/gs.py", line 530, in __init__
gsutil_bin = self.GetDefaultGSUtilBin(cache_dir, cache_user=cache_user)
File "/work2/chrome/src/third_party/chromite/lib/gs.py", line 424, in GetDefaultGSUtilBin
cls._CompileCrcmod(ref.path)
File "/work2/chrome/src/third_party/chromite/lib/gs.py", line 456, in _CompileCrcmod
osutils.Touch(flag)
File "/work2/chrome/src/third_party/chromite/lib/osutils.py", line 156, in Touch
open(path, 'a').close()
IOError: [Errno 2] No such file or directory: '/work2/chrome/src/build/cros_cache/common/gsutil_4.30.tar.gz/gsutil/third_party/crcmod/.chromite.tried.build'
,
Jul 16
what parts of those paths exist in your system ?
,
Jul 16
Maybe this is a weird racey/flaky bug when the cache moves to a new location for the first time? I'll try to repro.
,
Jul 16
The previous directory structure was build/cros_cache/common $ tree . ├── gsutil_4.30.tar.gz │ └── gsutil │ └── third_party │ ├── argcomplete │ │ └── common.mk │ ├── mock │ │ └── ChangeLog │ └── rsa │ └── doc │ └── _build ├── gsutil_4.30.tar.gz.entry_lock └── staging 9 directories, 3 files
,
Jul 16
> Given the timing I suspect this is somehow related to moving the location of the cache dir, but may be related to other recent changes. I should also mention that I received a warning that my old cache directory was no longer needed, so this seems very likely to me.
,
Jul 16
that dir tree is extremely borked. i'm not sure how it got to be that way, but you should just `rm -rf` that common dir.
,
Jul 16
Yea, that's what I ended up doing (I moved it into a different directory instead of using `rm -rf`, which is how I was able print it above).
,
Jul 17
Couldn't repro in quite the same way, but I did manage to mess with the cache enough that it would fail every gsutil call. rm'ing the whole thing fixed it. I'm not sure how your cache got in the state it did. I'll leave this open for a bit, but if no one else runs into the same issue, I'll just close it out
,
Jul 17
we def don't do any hashing/checking on the cached files. our cache system assumes that, once an item makes into a cache, it stays valid (and we have logic to make sure we don't insert an incomplete package). so it is def easy to corrupt a cache in a way that we don't recover. i would put a pin in this and wait to see if anyone else hits the same problem. otherwise it's prob not worth tracking down :/.
,
Aug 3
This bug has an owner, thus, it's been triaged. Changing status to "assigned".
,
Aug 21
i'll just icebox this until it comes up again |
||||
►
Sign in to add a comment |
||||
Comment 1 by steve...@chromium.org
, Jul 16Owner: bpastene@chromium.org
Status: Untriaged