New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 677534 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

cyan-release build timed out

Project Member Reported by semenzato@chromium.org, Dec 29 2016

Issue description

It happened in paygen, about 7h50m after start.  The time spent in paygen looks normal.

https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-release/builds/713

The build started at 18:08 and timed out at 1:51.  Some of the time stamps:

** Finished Stage InitSDK - Wed, 28 Dec 2016 18:43:15 -0800 (PST)
** Finished Stage BuildPackages [afdo_use] - Wed, 28 Dec 2016 22:49:26 -0800 (PST)
** Finished Stage BuildImage [afdo_use] - Wed, 28 Dec 2016 22:59:05 -0800 (PST)
** Finished Stage UnitTest - Wed, 28 Dec 2016 23:31:29 -0800 (PST)
** Finished Stage Signing - Thu, 29 Dec 2016 00:24:40 -0800 (PST)

paygen stdout snippet:

01:12:08: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cp -a public-read /tmp/cbuildbot-tmpetWfvO/tmpMMdwuX/paygen_build-control_files.7yCS7H/paygen_au_dev_control.tar.bz2 gs://chromeos-image-archive/cyan-release/R57-9131.0.0/paygen_au_dev_control.tar.bz2
01:12:09: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpetWfvO/tmpBNQop4/temp_summary.json --raw-cmd --task-name cyan-release/R57-9131.0.0-paygen_au_dev --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:paygen_au_dev' '--tags=build:cyan-release/R57-9131.0.0' '--tags=task_name:cyan-release/R57-9131.0.0-paygen_au_dev' '--tags=board:cyan' -- /usr/local/autotest/site_utils/run_suite.py --build cyan-release/R57-9131.0.0 --board cyan --suite_name paygen_au_dev --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --suite_min_duts 2 -c
Autotest instance: cautotest
12-29-2016 [01:12:15] Submitted create_suite_job rpc
12-29-2016 [01:12:20] Created suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=93242853
@@@STEP_LINK@Link to suite@http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=93242853@@@
--create_and_return was specified, terminating now.
Will return from run_suite with status: OK
01:12:22: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpetWfvO/tmp4Hmx3q/temp_summary.json --raw-cmd --task-name cyan-release/R57-9131.0.0-paygen_au_dev --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:paygen_au_dev' '--tags=build:cyan-release/R57-9131.0.0' '--tags=task_name:cyan-release/R57-9131.0.0-paygen_au_dev' '--tags=board:cyan' -- /usr/local/autotest/site_utils/run_suite.py --build cyan-release/R57-9131.0.0 --board cyan --suite_name paygen_au_dev --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --suite_min_duts 2 -m 93242853

@@@STEP_FAILURE@@@
01:50:44: ERROR: Timeout occurred- waited 27701 seconds, failing. Timeout reason: This build has reached the timeout deadline set by the master. Either this stage or a previous one took too long (see stage timing historical summary in ReportStage) or the build failed to start on time.

@@@STEP_FAILURE@@@
01:50:45: ERROR: Timeout occurred- waited 27314 seconds, failing. Timeout reason: This build has reached the timeout deadline set by the master. Either this stage or a previous one took too long (see stage timing historical summary in ReportStage) or the build failed to start on time.
01:50:45: INFO: Running cidb query on pid 30948, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x55dfbd0>
01:50:50: INFO: cidb query succeeded after 1 retries


---------------

There were many errors building chrome:

chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_cyan/Release/test_case.html.mock-http-headers': No such file or directory
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_cyan/Release/test_page.css': No such file or directory
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_cyan/Release/test_url_loader_data': No such file or directory
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/test_src
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/chrome/test/functional" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/content/common/gpu/testdata" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/third_party/accessibility-developer-tools/gen/axs_testing.js" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/chrome/test/functional" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/content/common/gpu/testdata" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/third_party/accessibility-developer-tools/gen/axs_testing.js" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/test_src
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/pdf/test" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-57.0.2965.0_rc-r1: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/pdf/test" failed: No such file or directory (2)
chromeos-chrome-57.0.2965.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/test_src
chromeos-chrome-57.0.2965.0_rc-r1: Copying Page Cycler Data into /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/page_cycler_src
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/page_cycler_src
chromeos-chrome-57.0.2965.0_rc-r1: Copying Perf Data into /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/perf_data_src
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/perf_data_src
chromeos-chrome-57.0.2965.0_rc-r1: Copying Telemetry Framework into /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/telemetry_src
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/telemetry_src
chromeos-chrome-57.0.2965.0_rc-r1:  * install_test_resources to /build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/telemetry_src
chromeos-chrome-57.0.2965.0_rc-r1:  * Installing telemetry binaries: bitmaptools clear_system_cache minidump_stackwalk
chromeos-chrome-57.0.2965.0_rc-r1:  * Preparing tests
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/jpeg_decode_accelerator_unittest' to 'chrome_test/test_src/out/Release/jpeg_decode_accelerator_unittest': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/media_unittests' to 'chrome_test/test_src/out/Release/media_unittests': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/sandbox_linux_unittests' to 'chrome_test/test_src/out/Release/sandbox_linux_unittests': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/video_decode_accelerator_unittest' to 'chrome_test/test_src/out/Release/video_decode_accelerator_unittest': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/video_encode_accelerator_unittest' to 'chrome_test/test_src/out/Release/video_encode_accelerator_unittest': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/chrome_sandbox' to 'chrome_test/test_src/out/Release/chrome_sandbox': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/out/Release/setup_test_links.sh' to 'chrome_test/test_src/out/Release/setup_test_links.sh': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/base/base_paths_posix.cc' to 'chrome_test/test_src/base/base_paths_posix.cc': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/avatar1.jpg' to 'chrome_test/test_src/chrome/test/data/chromeos/avatar1.jpg': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/avatar2.jpg' to 'chrome_test/test_src/chrome/test/data/chromeos/avatar2.jpg': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/avatar3.png' to 'chrome_test/test_src/chrome/test/data/chromeos/avatar3.png': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/mobile_activation.html' to 'chrome_test/test_src/chrome/test/data/chromeos/mobile_activation.html': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/oobe_webui_browsertest.js' to 'chrome_test/test_src/chrome/test/data/chromeos/oobe_webui_browsertest.js': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/app_mode/red16x16.png' to 'chrome_test/test_src/chrome/test/data/chromeos/app_mode/red16x16.png': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/key.pem' to 'chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/key.pem': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/src/main.js' to 'chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/src/main.js': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/src/manifest.json' to 'chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/src/manifest.json': Invalid cross-device link
chromeos-chrome-57.0.2965.0_rc-r1: cp: cannot create hard link '/build/cyan/tmp/portage/chromeos-base/chromeos-chrome-57.0.2965.0_rc-r1/work/autotest-work/client/deps/chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/src/window.html' to 'chrome_test/test_src/chrome/test/data/chromeos/app_mode/enterprise_kiosk_test_app/src/window.html': Invalid cross-device link



 
Is this still happening? 
Cc: -dtor@chromium.org -yhanada@chromium.org -martinroth@chromium.org -kevcheng@chromium.org nxia@chromium.org johnylin@chromium.org
Yes, cyan is still timing out.  Also see issue 676684 for the more general problem.  (Maybe we can duplicate to that one.)

Comment 3 by autumn@chromium.org, Jan 11 2017

Owner: dgarr...@chromium.org
Mergedinto: 676684
Status: Duplicate (was: Untriaged)

Sign in to add a comment