New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 601096 link

Starred by 2 users

Issue metadata

Status: Duplicate
Merged: issue 516795
Owner:
User never visited
Closed: Apr 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Chell fails paygen with timeout

Project Member Reported by bhthompson@chromium.org, Apr 6 2016

Issue description

Now that Cave is in a separate build group Chell should have plenty of time to build, however we are currently timing out in the Paygen step.

...
04:00:58: INFO: Uploading payload to gs://chromeos-releases/canary-channel/chell/8156.0.0/payloads/chromeos_8156.0.0-8156.0.0_chell_canary-channel_delta_test.bin-3ea539717a230d9403839f51831f1256.
04:01:00: INFO: * Finished payload generation in 0:03:46.954980
03:57:13: INFO: * Starting payload generation
03:57:13: INFO: Generating delta payload chromeos_8060.0.0-8156.0.0_chell_canary-channel_delta_test.bin-6abcc987594dd671769ef58fb7b4af5c
03:57:13: INFO: Preparing au-generator.zip from gs://chromeos-releases/canary-channel/chell/8156.0.0/au-generator.zip.
03:57:13: INFO: lock /b/cbuild/internal_master/.cache/paygen_cache/lock/310eced4cce1bd76f9816ddba3e5363a: blocking while taking a shared lock
03:57:15: INFO: Preparing image from gs://chromeos-releases/canary-channel/chell/8156.0.0/ChromeOS-test-R51-8156.0.0-chell.tar.xz as /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/tgt_image.bin
03:57:15: INFO: lock /b/cbuild/internal_master/.cache/paygen_cache/lock/ab0b0094a7372432d3c3fecf2c514f56: blocking while taking a shared lock
03:57:40: INFO: RunCommand: tar -xJf /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/image-archive-qznrLU.tar.xz chromiumos_test_image.bin in /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2
03:58:48: INFO: Preparing image from gs://chromeos-releases/canary-channel/chell/8060.0.0/ChromeOS-test-R51-8060.0.0-chell.tar.xz as /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/src_image.bin
03:59:08: INFO: RunCommand: tar -xJf /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/image-archive-gMyS19.tar.xz chromiumos_test_image.bin in /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2
04:00:08: INFO: Generating unsigned payload as /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/delta.bin
04:00:08: INFO: RunCommand: /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/au-generator/cros_generate_update_payload --outside_chroot --output /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/delta.bin --image /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/tgt_image.bin --channel canary-channel --board chell --version 8156.0.0 --key test --build_channel canary-channel --build_version 8156.0.0 --src_image /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/src_image.bin --src_channel canary-channel --src_board chell --src_version 8060.0.0 --src_key test --src_build_channel canary-channel --src_build_version 8060.0.0 in /tmp/cbuildbot-tmpoCUBqC/tmpvYa9Q0/paygen_payload.KGeqt2/au-generator

@@@STEP_FAILURE@@@
04:05:00: ERROR: Timeout occurred- waited 27628 seconds, failing. Timeout reason: Slave reached the timeout deadline set by master.

@@@STEP_FAILURE@@@
04:05:00: ERROR: Timeout occurred- waited 27965 seconds, failing. Timeout reason: Slave reached the timeout deadline set by master.
04:05:00: INFO: Detected sys.exit(Timeout occurred- waited 27628 seconds, failing. Timeout reason: Slave reached the timeout deadline set by master.)
04:05:00: INFO: Running cidb query on pid 777, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x524a210>
04:05:01: INFO: Running cidb query on pid 777, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x4cf1d50>
04:05:02: INFO: Running cidb query on pid 777, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x5259050>
...

https://uberchromegw.corp.google.com/i/chromeos/builders/glados-release-group/builds/864

Are we just taking too long to build Glados systems?
 

Comment 1 by autumn@chromium.org, Apr 11 2016

Cc: dgarr...@chromium.org
Owner: msartori@chromium.org
Matthew, can you take a look since Don is out? 
Cc: puneetster@chromium.org
This same (I believe) error appears on other platforms as well.  Here's an auron_paine builder that fails with it too.

https://uberchromegw.corp.google.com/i/chromeos/builders/auron-release-group/builds/1816
This is a lab issue. It's the HW Testing of payloads that is failing.
Both the chell and auron builders hit the 8-hour canary brick
wall timeout.  Isn't this just  bug 516795 ?

Mergedinto: 516795
Status: Duplicate (was: Untriaged)

Comment 6 by benhenry@google.com, Apr 27 2016

Components: Infra>Client>ChromeOS
Labels: -Infra-ChromeOS

Sign in to add a comment