New issue
Advanced search Search tips

Issue 673846 link

Starred by 2 users

Issue metadata

Status: Archived
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

"no space left on device" during Paygen

Project Member Reported by semenzato@chromium.org, Dec 13 2016

Issue description

On wizpig-release build 662.

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/662

22:11:13: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cp gs://chromeos-releases/dev-channel/wizpig/9079.0.0/ChromeOS-test-R57-9079.0.0-wizpig.tar.xz /b/cbuild/internal_master/.cache/paygen_cache/cache/9c2fa7b26c4fe25e09cc40e948b990c6
22:12:26: INFO: RunCommand: tar -xJf /tmp/cbuildbot-tmpqZqhjR/tmpnzl1Ts/paygen_payload.sCGHK3/image-archive-rneyFe.tar.xz chromiumos_test_image.bin in /tmp/cbuildbot-tmpqZqhjR/tmpnzl1Ts/paygen_payload.sCGHK3
22:13:59: INFO: Preparing image from gs://chromeos-releases/dev-channel/wizpig/9000.15.0/ChromeOS-test-R56-9000.15.0-wizpig.tar.xz as /tmp/cbuildbot-tmpqZqhjR/tmpnzl1Ts/paygen_payload.sCGHK3/src_image.bin
22:22:24: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -h x-goog-if-generation-match:1481609467979000 rm gs://chromeos-releases/dev-channel/wizpig/9079.0.0/payloads/LOCK_flag
22:22:24: ERROR: Failed: Build definition (board='wizpig', version='9079.0.0', channel='dev-channel')

@@@STEP_FAILURE@@@
22:22:26: ERROR: <type 'exceptions.IOError'>: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 268, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 837, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 707, in Run
    self._drm(self._VerifyPayload)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 681, in _VerifyPayload
    self._ApplyPayload(payload, is_delta)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload
    payload.Apply(bspatch_path=bspatch_path, **part_files)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply
    old_rootfs_part=old_rootfs_part)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run
    self.payload.manifest.old_rootfs_info)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 517, in _ApplyToPartition
    new_part_file, new_part_info.size)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 458, in _ApplyOperations
    self._ApplyReplaceOperation(op, op_name, data, new_part_file, part_size)
  File "/b/cbuild/internal_master/src/platform/dev/host/lib/update_payload/applier.py", line 269, in _ApplyReplaceOperation
    part_file.write(out_data[data_start:data_end])
IOError: [Errno 28] No space left on device

<type 'exceptions.IOError'>: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 268, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 837, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 705, in Run
    self._drm(self._Create)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 552, in _Create
    self._PrepareImage(self.payload.src_image, self.src_image_file)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 262, in _PrepareImage
    self.cache.GetFileCopy(image.uri, download_file)
  File "/b/cbuild/internal_master/chromite/lib/paygen/download_cache.py", line 365, in GetFileCopy
    shutil.copyfileobj(src, dest)
  File "/usr/lib/python2.7/shutil.py", line 52, in copyfileobj
    fdst.write(buf)
IOError: [Errno 28] No space left on device

 
Labels: -Pri-2 Pri-1
wizpig-release has been failing in paygen with "no space left on device" since build 659 (the latest build is 665). 
Cc: -mtomasz@chromium.org -sonnyrao@chromium.org dgarr...@chromium.org snanda@chromium.org
Owner: dgarr...@chromium.org
Let's use this bug as the reference.

These are the recent builds that failed for this reason.  I'll try to dig out the actual builders next.

asuka/706   no space
asuka/709   no space: 2/186
asuka/721   no space: 3/186 (last 2)
asuka/722   no space: 4/186 (last 3)
asuka/724   no space: 5/186 (last 4)
asuka/725   no space: 6/186 (last 6)
asuka/726   no space: 7/187 (last 7)
asuka/729   no space: 9/189 (last 9)
auron_yuna/706   no space
auron_yuna/709   no space: 2/186
auron_yuna/721   no space: 3/186 (last 2)
auron_yuna/722   no space: 4/186 (last 3)
auron_yuna/724   no space: 5/186 (last 4)
auron_yuna/725   no space: 6/186 (last 6)
auron_yuna/726   no space: 7/187 (last 7)
auron_yuna/728   no space: 8/188 (last 8)
auron_yuna/730   no space: 9/189 (last 9)
banon/726   no space
banon/728   no space: 2/188 (last 2)
banon/730   no space: 3/189 (last 3)
caroline/222   no space
caroline/225   no space: 2/186
caroline/231   no space: 3/186 (last 2)
cave/532   no space
cave/534   no space: 2/186
celes/706   no space
celes/709   no space: 2/186
celes/721   no space: 3/186 (last 2)
celes/722   no space: 4/186 (last 3)
celes/724   no space: 5/186 (last 4)
celes/726   no space: 6/187 (last 5)
celes/730   no space: 8/189 (last 3)
gandof/720   no space
gandof/722   no space: 2/189 (last 2)
lulu/689   no space
lulu/729   no space: 2/188
lulu/731   no space: 3/189 (last 2)
reks/730   no space
umaro/733   no space
wizpig/648   no space
wizpig/662   no space: 3/186 (last 2)
wizpig/665   no space: 4/186
wizpig/666   no space: 5/186 (last 2)
wizpig/668   no space: 6/186 (last 3)
wizpig/671   no space: 7/186 (last 4)
wizpig/678   no space: 8/186 (last 5)
wizpig/681   no space: 9/186 (last 6)
wizpig/687   no space: 10/186 (last 7)

For future reference:

The build slave is listed at the bottom of the builder page for a board.  For instance:

https://uberchromegw.corp.google.com/i/chromeos/builders/nyan_big-release

Search for "buildslaves".

asuka:      cros-beefy11-c2
auron_yuna: cros-beefy13-c2
banon:      cros-beefy15-c2
caroline:   cros-beefy20-c2
cave:       cros-beefy19-c2
celes:      cros-beefy21-c2
gandof:     cros-beefy32-c2
lulu:       cros-beefy44-c2
reks:       cros-beefy56-c2
umaro:      cros-beefy69-c2
wizpig:     cros-beefy82-c2


I'll reinstance them, when the in-progress builds finish.
All of the builders listed have been reinstanced with 2TB drives.

The 6 PM builds on them will be a little slower than usual as they re-fetch things that are normally cached.
Status: Fixed (was: Untriaged)
My expectations is that the existing problems are fixed, though I still hope to do a more general reinstancing for all builders.
Thank you Don.  We'll see what happens tonight.

Comment 9 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 10 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 11 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 13 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment