New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 675645 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

no space left on device for wizpig is not reported clearly by payload generator

Project Member Reported by semenzato@chromium.org, Dec 19 2016

Issue description

The wizpig release build failure is apparently due to out-of-space.  However, the logs are confusing.

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/681/steps/Paygen/logs/stdio

The first error is here:

06:40:30: INFO: Generating unsigned payload as /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/delta.bin
06:40:30: INFO: RunCommand: /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/au-generator/cros_generate_update_payload --outside_chroot --output /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/delta.bin --image /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/tgt_image.bin --channel canary-channel --board wizpig --version 9102.0.0 --key mp-v8 --build_channel canary-channel --build_version 9102.0.0 in /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/au-generator
06:44:44: ERROR: Nonzero exit code (1), dumping command output:
Generating full update
Truncated root to 1835008000 bytes.
Using rootfs partition size: 2097152000
Running delta_generator with args: -major_version=1 -out_file=/tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/delta.bin -private_key= -new_image=/tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/cros_generate_update_payload.YXQxLl -new_kernel=/tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/cros_generate_update_payload.ePl50Q -new_channel=canary-channel -new_board=wizpig -new_version=9102.0.0 -new_key=mp-v8 -new_build_channel=canary-channel -new_build_version=9102.0.0 -rootfs_partition_size=2097152000
[1219/064305:WARNING:generate_delta_main.cc(438)] --new_partitions is empty, using deprecated --new_image and --new_kernel flags.
[1219/064305:INFO:generate_delta_main.cc(520)] Using provided major_version=1
[1219/064305:INFO:generate_delta_main.cc(539)] Auto-detected minor_version=0
[1219/064305:INFO:generate_delta_main.cc(556)] Generating full update
[1219/064305:INFO:delta_diff_generator.cc(89)] Partition name: system
[1219/064305:INFO:delta_diff_generator.cc(90)] Partition size: 1835008000
[1219/064305:INFO:delta_diff_generator.cc(91)] Block count: 448000
[1219/064305:INFO:delta_diff_generator.cc(117)] Using generator FullUpdateGenerator().
[1219/064305:INFO:full_update_generator.cc(143)] Compressing partition system from /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/cros_generate_update_payload.YXQxLl splitting in chunks of 512 blocks (4096 bytes each) using 32 threads
[1219/064313:INFO:blob_file_writer.cc(35)] 10% complete 88/875 ops (output size: 83618235)
[1219/064317:INFO:blob_file_writer.cc(35)] 20% complete 175/875 ops (output size: 170772744)
[1219/064322:INFO:blob_file_writer.cc(35)] 30% complete 263/875 ops (output size: 255329843)
[1219/064325:INFO:blob_file_writer.cc(35)] 40% complete 350/875 ops (output size: 313363677)
[1219/064329:INFO:blob_file_writer.cc(35)] 50% complete 438/875 ops (output size: 372203358)
[1219/064337:INFO:blob_file_writer.cc(35)] 60% complete 525/875 ops (output size: 522392597)
[1219/064346:INFO:blob_file_writer.cc(35)] 70% complete 613/875 ops (output size: 685749660)
[1219/064352:INFO:blob_file_writer.cc(35)] 80% complete 700/875 ops (output size: 790950007)
[1219/064355:INFO:blob_file_writer.cc(35)] 90% complete 788/875 ops (output size: 871581292)
[1219/064359:INFO:blob_file_writer.cc(35)] 100% complete 875/875 ops (output size: 977824975)
[1219/064404:INFO:delta_diff_utils.cc(748)] /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/cros_generate_update_payload.YXQxLl: size=1835008000 hash=PZriNtxCP6czzU2pgw3Bkcr5Uh8X/4eNuybX9z96iwk=
[1219/064404:INFO:delta_diff_generator.cc(89)] Partition name: boot
[1219/064404:INFO:delta_diff_generator.cc(90)] Partition size: 16777216
[1219/064404:INFO:delta_diff_generator.cc(91)] Block count: 4096
[1219/064404:INFO:delta_diff_generator.cc(117)] Using generator FullUpdateGenerator().
[1219/064404:INFO:full_update_generator.cc(143)] Compressing partition boot from /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/cros_generate_update_payload.ePl50Q splitting in chunks of 512 blocks (4096 bytes each) using 32 threads
[1219/064404:INFO:blob_file_writer.cc(35)] 12% complete 1/8 ops (output size: 977825023)
[1219/064404:INFO:blob_file_writer.cc(35)] 25% complete 2/8 ops (output size: 977825071)
[1219/064404:INFO:blob_file_writer.cc(35)] 37% complete 3/8 ops (output size: 977825119)
[1219/064404:INFO:blob_file_writer.cc(35)] 50% complete 4/8 ops (output size: 977825167)
[1219/064404:INFO:blob_file_writer.cc(35)] 62% complete 5/8 ops (output size: 977825215)
[1219/064405:INFO:blob_file_writer.cc(35)] 75% complete 6/8 ops (output size: 979073408)
[1219/064405:INFO:blob_file_writer.cc(35)] 87% complete 7/8 ops (output size: 981113798)
[1219/064405:INFO:blob_file_writer.cc(35)] 100% complete 8/8 ops (output size: 983210950)
[1219/064405:INFO:delta_diff_utils.cc(748)] /tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/cros_generate_update_payload.ePl50Q: size=16777216 hash=eMwiyF4j3fz2DTH4JIqZX6CwzemEz2fT9NPEQ/krQVI=
[1219/064405:INFO:delta_diff_generator.cc(138)] Writing payload file...
[1219/064408:ERROR:payload_file.cc(303)] writer.Write(buf.data(), buf.size()) failed.
[1219/064443:ERROR:payload_file.cc(111)] ReorderDataBlobs(data_blobs_path, ordered_blobs_path) failed.
[1219/064443:ERROR:delta_diff_generator.cc(141)] payload.WritePayload(output_path, temp_file_path, private_key_path, metadata_size) failed.
ERROR   : Mon Dec 19 06:44:44 PST 2016
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 32724:  '--outside_chroot' '--output' '/tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/delta.bin' '--image' '/tmp/cbuildbot-tmpXEeDkE/tmp2QQ4bJ/paygen_payload.2ukdcO/tgt_image.bin' '--channel' 'canary-channel' '--board' 'wizpig' '--version' '9102.0.0' '--key' 'mp-v8' '--build_channel' 'canary-channel' '--build_version' '9102.0.0'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  cros_generate_update_payload:345:main(), called: cleanup_on_error  
ERROR   :  cros_generate_update_payload:136:cleanup_on_error(), called: die 'Cleanup success after an error.' 
ERROR   : 
ERROR   : Error was:
ERROR   :   Cleanup success after an error.

The first indication of an error is at the line

06:44:44: ERROR: Nonzero exit code (1), dumping command output:

and then there are these errors:

[1219/064405:INFO:delta_diff_generator.cc(138)] Writing payload file...
[1219/064408:ERROR:payload_file.cc(303)] writer.Write(buf.data(), buf.size()) failed.
[1219/064443:ERROR:payload_file.cc(111)] ReorderDataBlobs(data_blobs_path, ordered_blobs_path) failed.
[1219/064443:ERROR:delta_diff_generator.cc(141)] payload.WritePayload(output_path, temp_file_path, private_key_path, metadata_size) failed.

so they are likely to be ENOSPC, but why isn't the program logging that error?  "writer.Write() failed" is a poor message.


 
Cc: nxia@chromium.org adlr@chromium.org snanda@chromium.org
Also found here.

https://uberchromegw.corp.google.com/i/chromeos/builders/celes-release/builds/725/steps/Paygen/logs/stdio

I may have already mentioned this in a separate bug: the update engine needs an owner.

Comment 2 by nxia@chromium.org, Jun 8 2018

Cc: -nxia@chromium.org

Sign in to add a comment