push to prod failed to update servers and crashed |
||||
Issue description
I think push to prod passed, but it also threw errors. Please confirm if this is expected
Below is the push log:
autotest:
git log --oneline d8a535dac..b45c84267
b45c84267 [autotest] Fix check_diskspace path existence check
3e5ff4a2c autotest: suite: privatize get_result_reporter
194b870d7 [autotest] Fix import order in label_cleaner
f672ae22c autotest: suite: always use Email reporter
07f1d3e95 [autotest] Make the diskspace check more robust.
c314a4e17 [autotest] Fix backwards compatibility of sysinfo excludes.
0e9214d43 [autotest] Remove rpc_hostname feature from shards
0248035d3 [autotest] Remove puppylab code
944ac4624 [autotest] Merge server/autotest.client_logger
817b3f1e8 [autotest] Supply port when setting up SSH tunnel with host
6b53c3842 autotest: temporarily short-cut some broken reporter logic
3dabea54f [autotest] pass-through all host attribute updates to master
54c47bfd4 autotest: Switch dbus call from LibCrosService to DisplayService
chromite:
git log --oneline 80329de4..847bb287
847bb287 Update config settings by config-updater.
71c09cdc Remove GCC builders from the waterfall.
30ee94e3 cbuildbot: create bluestreak-pre-cq
22b5ab52 Update config settings by config-updater.
1341bdda cros_workon: drop shared parser logic
13f7f691 chromeos_config: Add internal to betty-vmtest-informational
135a1eec cros tryjob: Add support for local tryjobs.
9f36a55a cros tryjob: Add support for local patches.
1e1fbd05 cbuildbot: Pass --nouse-image to cros_sdk
593014dc constant: move waterfall constants to own file
c98f4561 Revert "Add basic OrderedDict fallback implementation."
8cdbe778 chromeos_config: add betty-chrome-pfq as experimental to chrome-pfq
4ad25dcf portage_util: Use unstable ebuild to determine subdirs
5fed0a54 simple_builders: Make pre-cq vm tests unforgiving
77f85661 mark_android_as_stable: add x86_64 emulator artifacts
33242471 chromite: cros flash sometimes fails without any logging for debug
7d7c2da8 cros tryjob: Create a new command.
916358b9 chromiumos-sdk: Update timeout to 18 hrs from 16 hrs.
a60f7085 chromeos_config: Add new builder for vmtest
9fa1311c cbuildbot: Remove chroot.img when removing chroot dir
e9842fe5 chromeos_config: enable --latest_toolchain in llvm_toolchain template.
386b9eb8 cros_sdk: Create chroots on a loopback device.
eade7bf9 Mark guado_moblab experiment temporarily.
f0a602ea cros_setup_toolchains: Add clang to list of MANUAL_PKGS.
20cfc6ce Add libcxx and libcxxabi to host toolchain packages.
4753160c Revert 'generic_stages: temporarily ignore all target prebuilts.'
Traceback (most recent call last):
File "./automated_deploy.py", line 218, in <module>
sys.exit(main(sys.argv))
File "./automated_deploy.py", line 200, in main
kick_off_deploy()
File "./automated_deploy.py", line 178, in kick_off_deploy
infra.local_runner(deploy_cmd, stream_output=True)
File "/usr/local/google/home/nxia/chromiumos/src/third_party/autotest/files/site_utils/lib/infra.py", line 60, in local_runner
raise subprocess.CalledProcessError(return_code, cmd, output[:1024])
subprocess.CalledProcessError: Command 'runlocalssh ./site_utils/deploy_server.py --afe=cautotest' returned non-zero exit status 1
,
Aug 8 2017
Successfully updated server chromeos-server28.mtv.corp.google.com. Failed to update server chromeos-server29.mtv.corp.google.com. Error: Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server29.mtv.corp.google.com Auto packing deferred; not on AC Auto packing deferred; not on AC Fetching project chromiumos/infra_virtualenv Fetching project chromiumos/chromite Your sources have been sync'd successfully. sysmon stop/waiting sysmon start/running, process 19813 rpc-logserver stop/waiting rpc-logserver start/running, process 19847 Checking tree status: Tree status: clean Updating Repo. Updating server to prod branch Removing .pyc files Updating ~chromeos-test/chromiumos Removing .pyc files Running update commands: build_externals Running: build_externals: /usr/local/autotest/utils/build_externals.py Restarting Services: sysmon, rpc-logserver Restarting: sysmon Restarting: rpc-logserver Changes: autotest: No Change. autotest/site_utils/autotest_private: No Change.Successfully updated server chromeos-server30.cbf.corp.google.com. Successfully updated server chromeos-server34.mtv.corp.google.com. Successfully updated server chromeos-server38.cbf.corp.google.com. Does it mean chromeos-server29.mtv.corp.google.com is the only server failed? need to update it separately ?
,
Aug 8 2017
yes, you can login to the server, and update manually, see what is the output $ cd /usr/local/autotest $ ./site_utils/deploy_server_local.py
,
Aug 8 2017
chromeos-test@chromeos-server29:/usr/local/autotest$ ./site_utils/deploy_server_local.py Checking tree status: Tree status: clean Updating Repo. Auto packing deferred; not on AC Fetching projects: 50% (1/2) Auto packing deferred; not on AC Fetching projects: 100% (2/2), done. Updating server to prod branch Removing .pyc files Updating ~chromeos-test/chromiumos Fetching project chromiumos/infra_virtualenv Fetching project chromiumos/chromite remote: Counting objects: 1852, done remote: Finding sources: 100% (4/4) remote: Total 4 (delta 3), reused 4 (delta 3) From https://chromium.googlesource.com/chromiumos/chromite 4b7ddead..56bccac4 master -> cros/master Fetching projects: 100% (2/2), done. Your sources have been sync'd successfully. Removing .pyc files Running update commands: build_externals Running: build_externals: /usr/local/autotest/utils/build_externals.py Restarting Services: sysmon, rpc-logserver Restarting: sysmon sysmon stop/waiting sysmon start/running, process 28373 Restarting: rpc-logserver rpc-logserver stop/waiting rpc-logserver start/running, process 28407 Changes: autotest: No Change. autotest/site_utils/autotest_private: No Change. same output and no error, does it server29 has been updated successfully?
,
Aug 9 2017
today's push failed at chromeos-server27.mtv.corp.google.com, going to check what's wrong Time used to update server chromeos-server27.mtv.corp.google.com: 718.003012896 Failed to update server chromeos-server27.mtv.corp.google.com. Error: Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server27.mtv.corp.google.com Auto packing deferred; not on AC Auto packing deferred; not on AC Fetching project chromiumos/infra_virtualenv Fetching project chromiumos/chromite Your sources have been sync'd successfully. sysmon stop/waiting sysmon start/running, process 10980 scheduler stop/waiting scheduler start/running, process 11244 host-scheduler stop/waiting host-scheduler start/running, process 11693 shard-client stop/waiting shard-client start/running, process 13460 gs_offloader stop/waiting gs_offloader start/running, process 13845 gs_offloader_s stop/waiting gs_offloader_s start/running, process 14254 rpc-logserver stop/waiting rpc-logserver start/running, process 14458 Checking tree status: Tree status: clean Updating Repo. Updating server to prod branch Removing .pyc files Updating ~chromeos-test/chromiumos Removing .pyc files Running update commands: build_externals Running: build_externals: /usr/local/autotest/utils/build_externals.py Restarting Services: sysmon, scheduler, host-scheduler, shard-client, gs_offloader, gs_offloader_s, rpc-logserver Restarting: sysmon Restarting: scheduler Restarting: host-scheduler Restarting: shard-client Restarting: gs_offloader Restarting: gs_offloader_s Restarting: rpc-logserver Changes: autotest: No Change. autotest/site_utils/autotest_private: No Change. Errors updating: chromeos-server27.mtv.corp.google.com
,
Aug 9 2017
manually run the deployment scripts afterwards
,
Aug 9 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by shuqianz@chromium.org
, Aug 8 2017