New issue
Advanced search Search tips

Issue 645396 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 3
Type: Bug



Sign in to add a comment

download_from_google_storage does not restore the executable bit when it's misisng

Reported by tsniatow...@opera.com, Sep 9 2016

Issue description

What steps will reproduce the problem?
(1) download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s src/buildtools/linux64/gn.sha1
(2) C-c at the right moment, or fake with chmod -x src/buildtools/linux64/gn.sha1
(3) download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s src/buildtools/linux64/gn.sha1

The second download_from_google_storage does not re-download, and src/buildtools/linux64/gn.sha1 stays non-executable.

This can happen accidentally when download_from_google_storage is interrupted by a C-c after it has downloaded the file, but before it set the executable bit. This is surprisingly easy to trigger. When I do:

rm src/buildtools/linux64/gn; download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s src/buildtools/linux64/gn.sha1; ls -la src/buildtools/linux64/gn

repeatedly with a manual well-timed C-c, I can get a downloaded (SHA1 matches) binary with no executable bit set roughly 50% of the time.

As it is usually a distant build step that actually tries to run the surprisingly-non-executable file, diagnosing the issue can be bothersome, and it can happen whenever someone aborts a download_from_google_storage.

I believe this happens because the sequence is
(1) download the file
(2) gsutil.check_call('stat', file_url) to see if it should be executable
(3) chmod +x if it should

The gsutil call in (2) is not instant, and a C-c there breaks things. I guess it's preferable to avoid a gsutil hit every time if the file is already present, so perhaps download_from_google_storage should download to a temporary filename, chmod, and then mv to the final location?
 

Comment 1 by estaab@chromium.org, Sep 19 2016

Components: -Infra>Platform Infra>SDK
from download_from_google_storage.py:
 code, _, err = gsutil.check_call('cp', file_url, output_filename)

the problem is that download_form_google_storage cp's directly into the target file, as  tsniatowski@ pointed out.
 
IMHO the established pattern for these kind of things is:
- download into a tmp file sitting in the same directory (e.g. /buildtools/linux64/gn.exe.tmp)
- chmod
- do a rename (which on most sensible platforms is an atomic operation)

this will give some basic atomicity guarantee, without leaving around stale files.


Status: Available (was: Untriaged)

Sign in to add a comment