fail push_to_prod or push_to_staging if build_externals fails |
||||||||
Issue descriptionFollow up to Issue 769419 , once it is resolved. build_externals randomly breaks on subsets of our production servers, and we don't notice. That's no bueno.
,
Oct 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/af98d4701a709569ea3439950c6878db2ef56af2 commit af98d4701a709569ea3439950c6878db2ef56af2 Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Oct 05 10:04:19 2017 autotest: move most packages to chromeos mirror 4 packages were untouched because there's currently a checksum mismatch between the existing mirror's version and the expected version. BUG= chromium:770351 TEST=utils/build_externals.py Change-Id: If4e34fb84002b7a41afe18d46aac20bc781e5079 Reviewed-on: https://chromium-review.googlesource.com/699737 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/af98d4701a709569ea3439950c6878db2ef56af2/utils/external_packages.py
,
Oct 9 2017
do a clean build_externals from empty on staging test
,
Oct 16 2017
,
Oct 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ccbf6437aab055ff5c2787e3cb521e4235aaedcd commit ccbf6437aab055ff5c2787e3cb521e4235aaedcd Author: Aviv Keshet <akeshet@chromium.org> Date: Fri Oct 20 21:38:49 2017 autotest: deploy_server_local: add a flag for clearing external packages BUG= chromium:770351 TEST=deploy_server_local locally with and without flag prior to adding dryrun check. Then, added dryrun check. Change-Id: I78a1152fc22f7cebdc030432016bd90780fe1eb3 Reviewed-on: https://chromium-review.googlesource.com/728863 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Shuqian Zhao <shuqianz@chromium.org> [modify] https://crrev.com/ccbf6437aab055ff5c2787e3cb521e4235aaedcd/site_utils/deploy_server_local.py
,
Oct 24 2017
,
Oct 26 2017
,
Nov 1 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/e9ac2c3de87694e1d891975a90e9befe3bd0b294 commit e9ac2c3de87694e1d891975a90e9befe3bd0b294 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed Nov 01 00:31:03 2017
,
Nov 1 2017
fixed (will be marked at meeting)
,
Nov 2 2017
[chromeos-staging-master2.hot.corp.google.com] out: Running update commands: build_externals [chromeos-staging-master2.hot.corp.google.com] out: Running: build_externals: /usr/local/autotest/utils/build_externals.py --use_chromite_master [chromeos-staging-master2.hot.corp.google.com] out: FAILED: [chromeos-staging-master2.hot.corp.google.com] out: 09:00:24 INFO | imported setuptools version 18.0.1. [chromeos-staging-master2.hot.corp.google.com] out: 09:00:24 INFO | A new Setuptools is not needed on this system. [chromeos-staging-master2.hot.corp.google.com] out: 09:00:24 INFO | Module MySQLdb is installed in /usr/lib/python2.7/dist-packages/MySQLdb/__init__.pyc, rather than /usr/local/autotest/site-packages. The module will be forced to be installed in /usr/local/autotest/site-packages. [chromeos-staging-master2.hot.corp.google.com] out: 09:00:24 INFO | Fetching http://commondatastorage.googleapis.com/chromeos-mirror/gentoo/distfiles/MySQL-python-1.2.3.tar.gz .. .. .. a whole bunch of spam .. .. [chromeos-staging-master2.hot.corp.google.com] out: 09:03:28 INFO | unzip -q -o -d '/usr/local/autotest/site-packages' 'dist/btsocket-1.0-py2.7-linux-x86_64. egg' [chromeos-staging-master2.hot.corp.google.com] out: 09:03:28 INFO | compiling .py files in /usr/local/autotest/site-packages to .pyc [chromeos-staging-master2.hot.corp.google.com] out: Compiling /usr/local/autotest/site-packages/astroid/tests/testdata/python2/data/invalid_encoding.py ... [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site-packages/astroid/tests/testdata/python2/data/invalid_encoding.py", line 0 [chromeos-staging-master2.hot.corp.google.com] out: SyntaxError: unknown encoding: lala [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: Compiling /usr/local/autotest/site-packages/astroid/tests/testdata/python3/data/invalid_encoding.py ... [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site-packages/astroid/tests/testdata/python3/data/invalid_encoding.py", line 0 [chromeos-staging-master2.hot.corp.google.com] out: SyntaxError: unknown encoding: lala .. .. .. [chromeos-staging-master2.hot.corp.google.com] out: assert (True, 'error msg') # [assert-on-tuple] [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: Compiling /usr/local/autotest/site-packages/pylint/test/functional/async_functions.py ... [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site-packages/pylint/test/functional/async_functions.py", line 5 [chromeos-staging-master2.hot.corp.google.com] out: async def next(): # [redefined-builtin] [chromeos-staging-master2.hot.corp.google.com] out: ^ [chromeos-staging-master2.hot.corp.google.com] out: SyntaxError: invalid syntax .. .. [chromeos-staging-master2.hot.corp.google.com] out: SyntaxError: invalid syntax [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: Compiling /usr/local/autotest/site-packages/pylint/test/regrtest_data/syntax_error.py ... [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site-packages/pylint/test/regrtest_data/syntax_error.py", line 1 [chromeos-staging-master2.hot.corp.google.com] out: class A extends B {} [chromeos-staging-master2.hot.corp.google.com] out: ^ [chromeos-staging-master2.hot.corp.google.com] out: SyntaxError: invalid syntax [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: 09:03:32 INFO | chmod -R a+rX '/usr/local/autotest/site-packages' [chromeos-staging-master2.hot.corp.google.com] out: 09:03:32 ERROR| Unable to download PyMox [chromeos-staging-master2.hot.corp.google.com] out: stderr: [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: From https://chromium.googlesource.com/chromiumos/chromite [chromeos-staging-master2.hot.corp.google.com] out: 5a9bf85c..29f3e3fd master -> cros/master [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: Traceback (most recent call last): [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 540, in <module> [chromeos-staging-master2.hot.corp.google.com] out: sys.exit(main(sys.argv[1:])) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 532, in main [chromeos-staging-master2.hot.corp.google.com] out: use_chromite_master=behaviors.update_push_servers) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 364, in run_deploy_actions [chromeos-staging-master2.hot.corp.google.com] out: use_chromite_master=use_chromite_master) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 257, in update_command [chromeos-staging-master2.hot.corp.google.com] out: stderr=subprocess.STDOUT) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/lib/python2.7/subprocess.py", line 573, in check_output [chromeos-staging-master2.hot.corp.google.com] out: raise CalledProcessError(retcode, cmd, output=output) [chromeos-staging-master2.hot.corp.google.com] out: subprocess.CalledProcessError: Command '/usr/local/autotest/utils/build_externals.py --use_chromite_maste r' returned non-zero exit status 1 [chromeos-staging-master2.hot.corp.google.com] out:
,
Nov 2 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/acbafc435859591ec8f12c2a2d7673f2151589af commit acbafc435859591ec8f12c2a2d7673f2151589af Author: Aviv Keshet <akeshet@google.com> Date: Thu Nov 02 16:13:57 2017
,
Nov 3 2017
,
Nov 4 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/adbff7d92785a18dbf9bf22dd353e1a13d82a99c commit adbff7d92785a18dbf9bf22dd353e1a13d82a99c Author: Aviv Keshet <akeshet@chromium.org> Date: Sat Nov 04 04:56:43 2017
,
Nov 6 2017
This is currently breaking push-to-prod. Yay! we broke it.
,
Nov 7 2017
,
Nov 7 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/9e52d1b46c79d9a57fe92e30293ba10a98cf9a57 commit 9e52d1b46c79d9a57fe92e30293ba10a98cf9a57 Author: Aviv Keshet <akeshet@chromium.org> Date: Tue Nov 07 20:55:49 2017
,
Nov 27 2017
Believed fixed. However, push to prod is failing a bunch. I don't think that is related to this, but ccing secondary deputy. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by akes...@chromium.org
, Oct 2 2017Owner: akes...@chromium.org
Summary: fail push_to_prod or push_to_staging if build_externals fails (was: add monarch metric and alert about build_externals)