prebuilts failing to upload on certain paladins. GS flake? |
|||||
Issue descriptionI've seen a couple of these now, but it seems that a couple of the paladins are failing to upload prebuilts. Opening this bug in case more incidents occur. It may just be a GS flake though. Also, not sure who to assign it too. The two failures I've seen so far: https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/1859 https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu_mobbuild-paladin/builds/2081 Snippet from an error: 07:03:46: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -h x-goog-if-generation-match:0 cp -v -- /b/cbuild/internal_master/chroot/build/guado_moblab/packages/chromeos-base/chromeos-imageburner-0.0.1-r1510.tbz2 gs://chromeos-prebuilt/board/guado_moblab/paladin-R51-8004.0.0-rc3/packages/chromeos-base/chromeos-imageburner-0.0.1-r1510.tbz2 07:03:49: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' acl ch -g 00b4903a97fb6344be6306829e053825e18a04ab0cc5513e9585a2b8c9634c80:FULL_CONTROL -g 00b4903a97ce95daf4ef249a9c21dddd83fdfb7126720b7c3440483b6229a03c:READ -g google.com:READ -u overlay.guado.private@gmail.com:READ gs://chromeos-prebuilt/board/guado_moblab/paladin-R51-8004.0.0-rc3/packages/chromeos-base/chromeos-imageburner-0.0.1-r1510.tbz2 upload_prebuilts: Unhandled exception: Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/bin/upload_prebuilts", line 164, in <module> commandline.ScriptWrapperMain(FindTarget) File "/b/cbuild/internal_master/chromite/lib/commandline.py", line 834, in ScriptWrapperMain ret = target(argv[1:]) File "/b/cbuild/internal_master/chromite/scripts/upload_prebuilts.py", line 889, in main options.toolchain_upload_path) File "/b/cbuild/internal_master/chromite/scripts/upload_prebuilts.py", line 640, in SyncBoardPrebuilts self._UploadPrebuilt(package_path, packages_url_suffix) File "/b/cbuild/internal_master/chromite/scripts/upload_prebuilts.py", line 464, in _UploadPrebuilt RemoteUpload(self._gs_context, self._acl, upload_files) File "/b/cbuild/internal_master/chromite/scripts/upload_prebuilts.py", line 234, in RemoteUpload parallel.RunTasksInProcessPool(upload, tasks, pool) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 808, in RunTasksInProcessPool queue.put((idx, input_args)) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 749, in BackgroundTaskRunner queue.put(_AllTasksComplete()) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 749, in BackgroundTaskRunner queue.put(_AllTasksComplete()) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 560, in ParallelTasks raise BackgroundFailure(exc_infos=errors) chromite.lib.parallel.BackgroundFailure: <class 'chromite.lib.gs.GSContextPreconditionFailed'>: return code: 1; command: /b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' acl ch -g 00b4903a97fb6344be6306829e053825e18a04ab0cc5513e9585a2b8c9634c80:FULL_CONTROL -g 00b4903a97ce95daf4ef249a9c21dddd83fdfb7126720b7c3440483b6229a03c:READ -g google.com:READ -u overlay.guado.private@gmail.com:READ gs://chromeos-prebuilt/board/guado_moblab/paladin-R51-8004.0.0-rc3/packages/chromeos-base/chromeos-imageburner-0.0.1-r1510.tbz2 ERROR 0303 07:04:03.303673 retry_decorator.py] Retrying in 0.99 seconds ... Traceback (most recent call last): File "/b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/third_party/retry-decorator/retry_decorator/retry_decorator.py", line 20, in f_retry return f(*args, **kwargs) File "/b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gslib/commands/acl.py", line 449, in ApplyAclChanges generation=url.generation) File "/b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gslib/cloud_api_delegator.py", line 215, in PatchObjectMetadata generation=generation, preconditions=preconditions, fields=fields) File "/b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gslib/gcs_json_api.py", line 875, in PatchObjectMetadata generation=generation) File "/b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gslib/gcs_json_api.py", line 1376, in _TranslateExceptionAndRaise raise translated_exception PreconditionException: PreconditionException: 412 Precondition Failed CommandException: Received bad request from server: BadRequestException: 400 Invalid argument.
,
Mar 4 2016
,
Mar 5 2016
Reply from bigstore team as "Won't Fix (intended behavior)": "This precondition failure can occur if the request is retried but the original request was successful. When this happens, it's up to your application to handle the precondition failure because gsutil can't guarantee that the object is in the state you desire (even if it performed a GET to check the ACLs, it's possible other changes to the object could have occurred in this timeframe)."
,
Mar 16 2016
,
Mar 20 2016
Ryan, as someone who dealt with gsutil before, can you PTAL and maybe re-triage? This has been happening on our infra waterfall, too. I wonder if we should add retries.
,
Mar 21 2016
Looks like it's a wontfix? Retries on the chromite gsutil stack also seems like a good idea. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by sbasi@chromium.org
, Mar 3 2016