"no space left on device" during Paygen |
||||||||
Issue descriptionOn wizpig-release build 662. https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/662 22:11:13: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cp gs://chromeos-releases/dev-channel/wizpig/9079.0.0/ChromeOS-test-R57-9079.0.0-wizpig.tar.xz /b/cbuild/internal_master/.cache/paygen_cache/cache/9c2fa7b26c4fe25e09cc40e948b990c6 22:12:26: INFO: RunCommand: tar -xJf /tmp/cbuildbot-tmpqZqhjR/tmpnzl1Ts/paygen_payload.sCGHK3/image-archive-rneyFe.tar.xz chromiumos_test_image.bin in /tmp/cbuildbot-tmpqZqhjR/tmpnzl1Ts/paygen_payload.sCGHK3 22:13:59: INFO: Preparing image from gs://chromeos-releases/dev-channel/wizpig/9000.15.0/ChromeOS-test-R56-9000.15.0-wizpig.tar.xz as /tmp/cbuildbot-tmpqZqhjR/tmpnzl1Ts/paygen_payload.sCGHK3/src_image.bin 22:22:24: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -h x-goog-if-generation-match:1481609467979000 rm gs://chromeos-releases/dev-channel/wizpig/9079.0.0/payloads/LOCK_flag 22:22:24: ERROR: Failed: Build definition (board='wizpig', version='9079.0.0', channel='dev-channel') @@@STEP_FAILURE@@@ 22:22:26: ERROR: <type 'exceptions.IOError'>: [Errno 28] No space left on device Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 602, in TaskRunner task(*x, **task_kwargs) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 800, in <lambda> fn = lambda idx, task_args: out_queue.put((idx, task(*task_args))) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 268, in _GenerateSinglePayload dry_run=dry_run) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 837, in CreateAndUploadPayload dry_run=dry_run).Run() File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 707, in Run self._drm(self._VerifyPayload) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__ return self.Run(func, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 82, in Run return self._Call(func, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call return func(*args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 681, in _VerifyPayload self._ApplyPayload(payload, is_delta) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload payload.Apply(bspatch_path=bspatch_path, **part_files) File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply old_rootfs_part=old_rootfs_part) File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run self.payload.manifest.old_rootfs_info) File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 517, in _ApplyToPartition new_part_file, new_part_info.size) File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 458, in _ApplyOperations self._ApplyReplaceOperation(op, op_name, data, new_part_file, part_size) File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 269, in _ApplyReplaceOperation part_file.write(out_data[data_start:data_end]) IOError: [Errno 28] No space left on device <type 'exceptions.IOError'>: [Errno 28] No space left on device Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 602, in TaskRunner task(*x, **task_kwargs) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 800, in <lambda> fn = lambda idx, task_args: out_queue.put((idx, task(*task_args))) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 268, in _GenerateSinglePayload dry_run=dry_run) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 837, in CreateAndUploadPayload dry_run=dry_run).Run() File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 705, in Run self._drm(self._Create) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__ return self.Run(func, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 82, in Run return self._Call(func, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call return func(*args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 552, in _Create self._PrepareImage(self.payload.src_image, self.src_image_file) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 262, in _PrepareImage self.cache.GetFileCopy(image.uri, download_file) File "/b/cbuild/internal_master/chromite/lib/paygen/download_cache.py", line 365, in GetFileCopy shutil.copyfileobj(src, dest) File "/usr/lib/python2.7/shutil.py", line 52, in copyfileobj fdst.write(buf) IOError: [Errno 28] No space left on device
,
Jan 5 2017
Let's use this bug as the reference. These are the recent builds that failed for this reason. I'll try to dig out the actual builders next. asuka/706 no space asuka/709 no space: 2/186 asuka/721 no space: 3/186 (last 2) asuka/722 no space: 4/186 (last 3) asuka/724 no space: 5/186 (last 4) asuka/725 no space: 6/186 (last 6) asuka/726 no space: 7/187 (last 7) asuka/729 no space: 9/189 (last 9) auron_yuna/706 no space auron_yuna/709 no space: 2/186 auron_yuna/721 no space: 3/186 (last 2) auron_yuna/722 no space: 4/186 (last 3) auron_yuna/724 no space: 5/186 (last 4) auron_yuna/725 no space: 6/186 (last 6) auron_yuna/726 no space: 7/187 (last 7) auron_yuna/728 no space: 8/188 (last 8) auron_yuna/730 no space: 9/189 (last 9) banon/726 no space banon/728 no space: 2/188 (last 2) banon/730 no space: 3/189 (last 3) caroline/222 no space caroline/225 no space: 2/186 caroline/231 no space: 3/186 (last 2) cave/532 no space cave/534 no space: 2/186 celes/706 no space celes/709 no space: 2/186 celes/721 no space: 3/186 (last 2) celes/722 no space: 4/186 (last 3) celes/724 no space: 5/186 (last 4) celes/726 no space: 6/187 (last 5) celes/730 no space: 8/189 (last 3) gandof/720 no space gandof/722 no space: 2/189 (last 2) lulu/689 no space lulu/729 no space: 2/188 lulu/731 no space: 3/189 (last 2) reks/730 no space umaro/733 no space wizpig/648 no space wizpig/662 no space: 3/186 (last 2) wizpig/665 no space: 4/186 wizpig/666 no space: 5/186 (last 2) wizpig/668 no space: 6/186 (last 3) wizpig/671 no space: 7/186 (last 4) wizpig/678 no space: 8/186 (last 5) wizpig/681 no space: 9/186 (last 6) wizpig/687 no space: 10/186 (last 7)
,
Jan 5 2017
For future reference: The build slave is listed at the bottom of the builder page for a board. For instance: https://uberchromegw.corp.google.com/i/chromeos/builders/nyan_big-release Search for "buildslaves".
,
Jan 5 2017
asuka: cros-beefy11-c2 auron_yuna: cros-beefy13-c2 banon: cros-beefy15-c2 caroline: cros-beefy20-c2 cave: cros-beefy19-c2 celes: cros-beefy21-c2 gandof: cros-beefy32-c2 lulu: cros-beefy44-c2 reks: cros-beefy56-c2 umaro: cros-beefy69-c2 wizpig: cros-beefy82-c2
,
Jan 5 2017
I'll reinstance them, when the in-progress builds finish.
,
Jan 6 2017
All of the builders listed have been reinstanced with 2TB drives. The 6 PM builds on them will be a little slower than usual as they re-fetch things that are normally cached.
,
Jan 6 2017
My expectations is that the existing problems are fixed, though I still hope to do a more general reinstancing for all builders.
,
Jan 6 2017
Thank you Don. We'll see what happens tonight.
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by benchan@chromium.org
, Dec 14 2016