push_to_prod error |
||||
Issue descriptionpush_to_prod gave errors: logs: Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server48.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [2/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com Traceback (most recent call last): File "/usr/local/autotest/site_utils/deploy_server_local.py", line 27, in <module> from autotest_lib.server import utils as server_utils File "/usr/local/autotest/server/utils.py", line 14, in <module> from autotest_lib.server.site_utils import * File "/usr/local/autotest/server/site_utils.py", line 29, in <module> from chromite.lib import ts_mon_config File "/usr/local/autotest/site-packages/chromite/lib/ts_mon_config.py", line 20, in <module> from chromite.lib import parallel File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 31, in <module> from chromite.lib import failures_lib File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 17, in <module> class StepFailure(Exception): File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 32, in StepFailure EXCEPTION_CATEGORY = constants.EXCEPTION_CATEGORY_UNKNOWN AttributeError: 'module' object has no attribute 'EXCEPTION_CATEGORY_UNKNOWN' chromeos-server48.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server48.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 1 [2/5] Try to update server chromeos-server48.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server48.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [3/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [4/5] Try to update server chromeos-server31.hot.corp.google.com Traceback (most recent call last): File "/usr/local/autotest/site_utils/deploy_server_local.py", line 27, in <module> from autotest_lib.server import utils as server_utils File "/usr/local/autotest/server/utils.py", line 14, in <module> from autotest_lib.server.site_utils import * File "/usr/local/autotest/server/site_utils.py", line 29, in <module> from chromite.lib import ts_mon_config File "/usr/local/autotest/site-packages/chromite/lib/ts_mon_config.py", line 20, in <module> from chromite.lib import parallel File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 31, in <module> from chromite.lib import failures_lib File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 17, in <module> class StepFailure(Exception): File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 32, in StepFailure EXCEPTION_CATEGORY = constants.EXCEPTION_CATEGORY_UNKNOWN AttributeError: 'module' object has no attribute 'EXCEPTION_CATEGORY_UNKNOWN' Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server48.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server48.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 1 [3/5] Try to update server chromeos-server48.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server48.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 Time used to update server chromeos-server31.hot.corp.google.com: 2033.6792109 Failed to update server chromeos-server31.hot.corp.google.com. Error: Traceback (most recent call last): File "/usr/local/autotest/site_utils/deploy_server_local.py", line 27, in <module> from autotest_lib.server import utils as server_utils File "/usr/local/autotest/server/utils.py", line 14, in <module> from autotest_lib.server.site_utils import * File "/usr/local/autotest/server/site_utils.py", line 29, in <module> from chromite.lib import ts_mon_config File "/usr/local/autotest/site-packages/chromite/lib/ts_mon_config.py", line 20, in <module> from chromite.lib import parallel File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 31, in <module> from chromite.lib import failures_lib File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 17, in <module> class StepFailure(Exception): File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 32, in StepFailure EXCEPTION_CATEGORY = constants.EXCEPTION_CATEGORY_UNKNOWN AttributeError: 'module' object has no attribute 'EXCEPTION_CATEGORY_UNKNOWN' chromeos-server48.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server48.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 1 [4/5] Try to update server chromeos-server48.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server48.hot.corp.google.com Traceback (most recent call last): File "/usr/local/autotest/site_utils/deploy_server_local.py", line 27, in <module> from autotest_lib.server import utils as server_utils File "/usr/local/autotest/server/utils.py", line 14, in <module> from autotest_lib.server.site_utils import * File "/usr/local/autotest/server/site_utils.py", line 29, in <module> from chromite.lib import ts_mon_config File "/usr/local/autotest/site-packages/chromite/lib/ts_mon_config.py", line 20, in <module> from chromite.lib import parallel File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 31, in <module> from chromite.lib import failures_lib File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 17, in <module> class StepFailure(Exception): File "/usr/local/autotest/site-packages/chromite/lib/failures_lib.py", line 32, in StepFailure EXCEPTION_CATEGORY = constants.EXCEPTION_CATEGORY_UNKNOWN AttributeError: 'module' object has no attribute 'EXCEPTION_CATEGORY_UNKNOWN' chromeos-server48.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server48.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 1 Time used to update server chromeos-server48.hot.corp.google.com: 2040.8441 Failed to update server chromeos-server48.hot.corp.google.com. Errors updating: chromeos-server31.hot.corp.google.com chromeos-server48.hot.corp.google.com To retry: Traceback (most recent call last): File "/tmp/autotest/site_utils/deploy_server.py", line 273, in <module> sys.exit(main(sys.argv)) File "/tmp/autotest/site_utils/deploy_server.py", line 269, in main (str(args[0]), str(' '.join(failed + skipped)))) TypeError: sequence item 2: expected string, tuple found
,
Nov 9 2016
,
Nov 9 2016
There are two issues here: A) EXCEPTION_CATEGORY_UNKNOWN This is the real error. Cause unknown. Fix unknown, but I suggest logging into both servers, removing all .pyc files in /usr/local, then re-pushing. B) The script did not correctly report all of the servers that failed to update. There are many others that did not update. crbug.com/660173. My hopeful fix just landed.
,
Nov 9 2016
If removing the .pyc files fixes the problem, then crbug.com/616277 is what caused failure A.
,
Nov 10 2016
I deleted all the pyc files, and I was able to deploy chromeos-server48.hot.corp.google.com. But I still got failures updating chromeos-server31.hot.corp.google.com. nxia@nxia:/tmp/autotest/site_utils$ runlocalssh ./deploy_server.py --afe=cautotest chromeos-server31.hot.corp.google.com WARNING:root:Unable to import chromite. Monarch is disabled. WARNING:root:Unable to import chromite. Retrieving server status... Will update (in this order): ============================== Group 1 (0 servers) ============================== ============================== Group 2 (1 servers) ============================== chromeos-server31.hot.corp.google.com: primary ['golo_proxy'] ============================== Group 3 (0 servers) ============================== Updating server chromeos-server31.hot.corp.google.com... [0/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [1/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [2/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [3/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 [4/5] Try to update server chromeos-server31.hot.corp.google.com Running '/usr/local/autotest/site_utils/deploy_server_local.py ' on chromeos-server31.hot.corp.google.com chromeos-server31.hot.corp.google.com: Command failed with error: Command '['googlesh', '-s', '-uchromeos-test', '-mchromeos-server31.hot.corp.google.com', '/usr/local/autotest/site_utils/deploy_server_local.py ']' returned non-zero exit status 255 Time used to update server chromeos-server31.hot.corp.google.com: 12.5591619015 Failed to update server chromeos-server31.hot.corp.google.com. Error: Errors updating: chromeos-server31.hot.corp.google.com To retry: /tmp/autotest/site_utils/deploy_server.py <options> chromeos-server31.hot.corp.google.com
,
Nov 10 2016
shuqianz@, can you take a look at the error?
,
Nov 10 2016
nxia@ has login permission issue to chromeos-server31.hot server, so the push fails. I will fix the crbug.com/616277 issue to remove the .pyc files. |
||||
►
Sign in to add a comment |
||||
Comment 1 by nxia@chromium.org
, Nov 9 2016