New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 681902 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Feb 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 616277



Sign in to add a comment

scheduler on testing push servers crashed because failed to get the global config value

Project Member Reported by shuqianz@chromium.org, Jan 17 2017

Issue description

Here is the scheduler log:

ts_mon: sending 2 metrics to PubSub
01/17 10:19:27.427 INFO |            client:0570| Attempting refresh to obtain initial access_token
01/17 10:19:27.479 INFO |            client:0872| Refreshing access_token
01/17 10:19:27.530 ERROR|        base_utils:0280| [stderr] Traceback (most recent call last):
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 18, in <module>
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr]     from autotest_lib.client.common_lib import utils, global_config, error
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr]   File "/usr/local/autotest/client/common_lib/utils.py", line 14, in <module>
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr]     from autotest_lib.client.common_lib.site_utils import *
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr]   File "/usr/local/autotest/client/common_lib/site_utils.py", line 23, in <module>
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr]     CONFIG = global_config.global_config
01/17 10:19:27.531 ERROR|        base_utils:0280| [stderr] AttributeError: 'module' object has no attribute 'global_config'
01/17 10:19:27.541 ERROR|site_drone_manager:0076| Failed to initialize drone chromeos-drone2-staging.hot.corp.google.com with error: command execution error
* Command: 
    /usr/bin/ssh -a -x  -o ControlPath=/tmp/_autotmp_n6wT8Vssh-master/socket
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpOwhp4a -o
    BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l
    chromeos-test -p 22 chromeos-drone2-staging.hot.corp.google.com " if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::call|_execute_calls|_execute_calls_impl] -> ssh_run(python
    /usr/local/autotest/scheduler/drone_utility.py --call_time
    1484677167.37)\";fi; python /usr/local/autotest/scheduler/drone_utility.py
    --call_time 1484677167.37"
Exit status: 1
Duration: 0.123951911926

stderr:
Traceback (most recent call last):
  File "/usr/local/autotest/scheduler/drone_utility.py", line 18, in <module>
    from autotest_lib.client.common_lib import utils, global_config, error
  File "/usr/local/autotest/client/common_lib/utils.py", line 14, in <module>
    from autotest_lib.client.common_lib.site_utils import *
  File "/usr/local/autotest/client/common_lib/site_utils.py", line 23, in <module>
    CONFIG = global_config.global_config
AttributeError: 'module' object has no attribute 'global_config'
01/17 10:19:27.543 ERROR|     email_manager:0082| Uncaught exception; terminating monitor_db
Traceback (most recent call last):
  File "/usr/local/autotest/scheduler/monitor_db.py", line 172, in main_without_exception_handling
    initialize()
  File "/usr/local/autotest/scheduler/monitor_db.py", line 243, in initialize
    _drone_manager.initialize(RESULTS_DIR, drone_list, results_host)
  File "/usr/local/autotest/scheduler/drone_manager.py", line 201, in initialize
    raise DroneManagerError('No valid drones found')
DroneManagerError: No valid drones found


pprabhu@, I saw that you recently committed several changes related to the global_config.py Can you take a look of this?

 
Status: Started (was: Untriaged)
Deleting /usr/local/autotest/client/common_lib/global_config.pyc fixed this issue.

I don't know why
(1) that file wasn't updated (but this is known to happen
(2) before the update, the dir(global_config) showed a module with no global variables. This shouldn't have been the case even before my CL. How did this file even get here?

Because I have removed global_config.pyc, the next test_push will likely succeed, but we still run the risk of this issue occurring on prod.
Blockedon: 616277
Cc: -dshi@chromium.org pprabhu@chromium.org
Owner: dshi@chromium.org
Status: Assigned (was: Started)
test_push has gotten past the testing stage.
Handing to dshi@ as an FYI when he tries to push-to-prod.

Please assign back to me if you want me to take any action.

Comment 6 by dshi@chromium.org, Feb 6 2017

Status: Fixed (was: Assigned)
this should be fixed in another bug to remove all pyc file during push.

Sign in to add a comment