scheduler on testing push servers crashed because failed to get the global config value |
||||
Issue descriptionHere is the scheduler log: ts_mon: sending 2 metrics to PubSub 01/17 10:19:27.427 INFO | client:0570| Attempting refresh to obtain initial access_token 01/17 10:19:27.479 INFO | client:0872| Refreshing access_token 01/17 10:19:27.530 ERROR| base_utils:0280| [stderr] Traceback (most recent call last): 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] File "/usr/local/autotest/scheduler/drone_utility.py", line 18, in <module> 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] from autotest_lib.client.common_lib import utils, global_config, error 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] File "/usr/local/autotest/client/common_lib/utils.py", line 14, in <module> 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] from autotest_lib.client.common_lib.site_utils import * 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] File "/usr/local/autotest/client/common_lib/site_utils.py", line 23, in <module> 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] CONFIG = global_config.global_config 01/17 10:19:27.531 ERROR| base_utils:0280| [stderr] AttributeError: 'module' object has no attribute 'global_config' 01/17 10:19:27.541 ERROR|site_drone_manager:0076| Failed to initialize drone chromeos-drone2-staging.hot.corp.google.com with error: command execution error * Command: /usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_n6wT8Vssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpOwhp4a -o BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l chromeos-test -p 22 chromeos-drone2-staging.hot.corp.google.com " if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::call|_execute_calls|_execute_calls_impl] -> ssh_run(python /usr/local/autotest/scheduler/drone_utility.py --call_time 1484677167.37)\";fi; python /usr/local/autotest/scheduler/drone_utility.py --call_time 1484677167.37" Exit status: 1 Duration: 0.123951911926 stderr: Traceback (most recent call last): File "/usr/local/autotest/scheduler/drone_utility.py", line 18, in <module> from autotest_lib.client.common_lib import utils, global_config, error File "/usr/local/autotest/client/common_lib/utils.py", line 14, in <module> from autotest_lib.client.common_lib.site_utils import * File "/usr/local/autotest/client/common_lib/site_utils.py", line 23, in <module> CONFIG = global_config.global_config AttributeError: 'module' object has no attribute 'global_config' 01/17 10:19:27.543 ERROR| email_manager:0082| Uncaught exception; terminating monitor_db Traceback (most recent call last): File "/usr/local/autotest/scheduler/monitor_db.py", line 172, in main_without_exception_handling initialize() File "/usr/local/autotest/scheduler/monitor_db.py", line 243, in initialize _drone_manager.initialize(RESULTS_DIR, drone_list, results_host) File "/usr/local/autotest/scheduler/drone_manager.py", line 201, in initialize raise DroneManagerError('No valid drones found') DroneManagerError: No valid drones found pprabhu@, I saw that you recently committed several changes related to the global_config.py Can you take a look of this?
,
Jan 17 2017
Deleting /usr/local/autotest/client/common_lib/global_config.pyc fixed this issue. I don't know why (1) that file wasn't updated (but this is known to happen (2) before the update, the dir(global_config) showed a module with no global variables. This shouldn't have been the case even before my CL. How did this file even get here?
,
Jan 17 2017
Because I have removed global_config.pyc, the next test_push will likely succeed, but we still run the risk of this issue occurring on prod.
,
Jan 17 2017
,
Jan 17 2017
test_push has gotten past the testing stage. Handing to dshi@ as an FYI when he tries to push-to-prod. Please assign back to me if you want me to take any action.
,
Feb 6 2017
this should be fixed in another bug to remove all pyc file during push. |
||||
►
Sign in to add a comment |
||||
Comment 1 by pprabhu@chromium.org
, Jan 17 2017