download_from_google_storage does not restore the executable bit when it's misisng
Reported by
tsniatow...@opera.com,
Sep 9 2016
|
||
Issue description
What steps will reproduce the problem?
(1) download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s src/buildtools/linux64/gn.sha1
(2) C-c at the right moment, or fake with chmod -x src/buildtools/linux64/gn.sha1
(3) download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s src/buildtools/linux64/gn.sha1
The second download_from_google_storage does not re-download, and src/buildtools/linux64/gn.sha1 stays non-executable.
This can happen accidentally when download_from_google_storage is interrupted by a C-c after it has downloaded the file, but before it set the executable bit. This is surprisingly easy to trigger. When I do:
rm src/buildtools/linux64/gn; download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s src/buildtools/linux64/gn.sha1; ls -la src/buildtools/linux64/gn
repeatedly with a manual well-timed C-c, I can get a downloaded (SHA1 matches) binary with no executable bit set roughly 50% of the time.
As it is usually a distant build step that actually tries to run the surprisingly-non-executable file, diagnosing the issue can be bothersome, and it can happen whenever someone aborts a download_from_google_storage.
I believe this happens because the sequence is
(1) download the file
(2) gsutil.check_call('stat', file_url) to see if it should be executable
(3) chmod +x if it should
The gsutil call in (2) is not instant, and a C-c there breaks things. I guess it's preferable to avoid a gsutil hit every time if the file is already present, so perhaps download_from_google_storage should download to a temporary filename, chmod, and then mv to the final location?
,
Oct 5 2016
from download_from_google_storage.py:
code, _, err = gsutil.check_call('cp', file_url, output_filename)
the problem is that download_form_google_storage cp's directly into the target file, as tsniatowski@ pointed out.
IMHO the established pattern for these kind of things is:
- download into a tmp file sitting in the same directory (e.g. /buildtools/linux64/gn.exe.tmp)
- chmod
- do a rename (which on most sensible platforms is an atomic operation)
this will give some basic atomicity guarantee, without leaving around stale files.
,
Dec 21
|
||
►
Sign in to add a comment |
||
Comment 1 by estaab@chromium.org
, Sep 19 2016