Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users
Status: Archived
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment
no space left on device for kefka-release during paygenbuild (paygen build)
Project Member Reported by semenzato@chromium.org, Jan 12 2017 Back to list

https://uberchromegw.corp.google.com/i/chromeos/builders/kefka-release/builds/754/steps/PaygenBuildCanary/logs/stdio

23:26:04: INFO: Checking payload integrity
23:26:05: INFO: RunCommand: /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/au-generator/cros_generate_update_payload --outside_chroot --extract --image /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/tgt_image.bin --src_image /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/src_image.bin in /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/au-generator
23:26:12: INFO: Applying delta payload and verifying result
23:26:54: INFO: Uploading payload to gs://chromeos-releases/canary-channel/kefka/9176.0.0/payloads/chromeos_9170.0.0-9176.0.0_kefka_canary-channel_delta_test.bin-3c9a2cf8fd2833734f50cfbe6ed364a9.
23:26:54: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cp /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/delta.bin gs://chromeos-releases/canary-channel/kefka/9176.0.0/payloads/chromeos_9170.0.0-9176.0.0_kefka_canary-channel_delta_test.bin-3c9a2cf8fd2833734f50cfbe6ed364a9
23:26:55: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cp /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/delta.log gs://chromeos-releases/canary-channel/kefka/9176.0.0/payloads/chromeos_9170.0.0-9176.0.0_kefka_canary-channel_delta_test.bin-3c9a2cf8fd2833734f50cfbe6ed364a9.log
23:26:56: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cp /tmp/cbuildbot-tmp203pPj/tmpjCmu2u/paygen_payload.xj_6uU/delta.json gs://chromeos-releases/canary-channel/kefka/9176.0.0/payloads/chromeos_9170.0.0-9176.0.0_kefka_canary-channel_delta_test.bin-3c9a2cf8fd2833734f50cfbe6ed364a9.json
23:26:57: INFO: * Finished payload generation in 0:28:58.622379
23:30:00: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -h x-goog-if-generation-match:1484204265415794 rm gs://chromeos-releases/canary-channel/kefka/9176.0.0/payloads/LOCK_flag
23:30:01: ERROR: Failed: Build definition (board='kefka', version='9176.0.0', channel='canary-channel')

@@@STEP_FAILURE@@@
23:30:01: ERROR: <class 'chromite.lib.paygen.paygen_payload_lib.PayloadVerificationError'>: Resulting rootfs partition corrupted
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 267, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 837, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 707, in Run
    self._drm(self._VerifyPayload)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 681, in _VerifyPayload
    self._ApplyPayload(payload, is_delta)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 656, in _ApplyPayload
    raise PayloadVerificationError('Resulting rootfs partition corrupted')
PayloadVerificationError: Resulting rootfs partition corrupted

<type 'exceptions.IOError'>: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 267, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 837, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 707, in Run
    self._drm(self._VerifyPayload)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 681, in _VerifyPayload
    self._ApplyPayload(payload, is_delta)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload
    payload.Apply(bspatch_path=bspatch_path, **part_files)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply
    old_rootfs_part=old_rootfs_part)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run
    self.payload.manifest.old_rootfs_info)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 517, in _ApplyToPartition
    new_part_file, new_part_info.size)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 458, in _ApplyOperations
    self._ApplyReplaceOperation(op, op_name, data, new_part_file, part_size)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 269, in _ApplyReplaceOperation
    part_file.write(out_data[data_start:data_end])
IOError: [Errno 28] No space left on device




 
Comment 1 by h...@chromium.org, Jan 12 2017
Cc: ihf@chromium.org h...@chromium.org
Owner: dgarr...@chromium.org
If I understand this correctly, we're running out of HDD space of the *builder*, not out of space on the rootfs partition of the image.

So this is not about updating the disk_layout.json. We may need to upgrade the HDD of the builders.
Status: Started
The planned reimage of all GCE builders to increase their drive sizes. That may get delayed for a few days, but I'll still redo this one (cros-beefy38-c2) today as soon as the current build finished.
And yes, this is a builder space issue, not an image issue.
Status: Fixed
All GCE builders have been reimaged with 2T drives. This problem should be solved.
Comment 5 by dchan@google.com, Mar 4 2017
Labels: VerifyIn-58
Labels: VerifyIn-59
Labels: VerifyIn-60
Labels: VerifyIn-61
Comment 9 by dchan@chromium.org, Oct 14 (4 days ago)
Status: Archived
Sign in to add a comment