New issue
Advanced search Search tips

Issue 818346 link

Starred by 1 user
Status: Unconfirmed
Owner: ----
Cc:
pprabhu@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

cros-full-0027 raised permission exception when trying to create a path owned by root

Project Member Reported by nxia@chromium.org, Mar 2 2018 
autotest scheduler went down   http://shortn/_TZ3kn38mtj

5:46:38 ERROR| [stderr] Traceback (most recent call last):
15:46:38 ERROR| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 670, in <module>
15:46:38 ERROR| [stderr]     main()
15:46:38 ERROR| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 665, in main
15:46:38 ERROR| [stderr]     return_value = drone_utility.execute_calls(calls)
15:46:38 ERROR| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 422, in execute_calls
15:46:38 ERROR| [stderr]     results.append(method_call.execute_on(self))
15:46:38 ERROR| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 66, in execute_on
15:46:38 ERROR| [stderr]     return method(*self._args, **self._kwargs)
15:46:38 ERROR| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 240, in write_to_file
15:46:38 ERROR| [stderr]     self._ensure_directory_exists(os.path.dirname(file_path))
15:46:38 ERROR| [stderr]   File "/usr/local/autotest/scheduler/drone_utility.py", line 191, in _ensure_directory_exists
15:46:38 ERROR| [stderr]     os.makedirs(path)
15:46:38 ERROR| [stderr]   File "/usr/lib/python2.7/os.py", line 150, in makedirs
15:46:38 ERROR| [stderr]     makedirs(head, mode)
15:46:38 ERROR| [stderr]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
15:46:38 ERROR| [stderr]     mkdir(name, mode)
15:46:38 ERROR| [stderr] OSError: [Errno 13] Permission denied: '/usr/local/autotest/results/hosts/chromeos6-row1-rack7-host21/62619156-repair'
15:46:38 ERROR| drone_manager.execute_queue.cros-full-0027_mtv_corp_google_com raised an exception that will be re-raised by the thread pool manager.

...



15:46:38 ERROR| Uncaught exception, terminating monitor_db.
Traceback (most recent call last):
  File "/usr/local/autotest/scheduler/monitor_db.py", line 190, in main_without_exception_handling
    dispatcher.tick()
  File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 483, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/autotest/scheduler/monitor_db.py", line 415, in tick
    _drone_manager.execute_actions()
  File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 483, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/autotest/scheduler/drone_manager.py", line 575, in execute_actions
    self._drones.values())
  File "/usr/local/autotest/scheduler/thread_lib.py", line 167, in execute
    return self.get_results() if wait else None
  File "/usr/local/autotest/scheduler/thread_lib.py", line 115, in get_results
    self.wait_on_drones()
  File "/usr/local/autotest/scheduler/thread_lib.py", line 103, in wait_on_drones
    raise drone_task_queue.DroneTaskQueueException(exception_msg)
DroneTaskQueueException: Drone cros-full-0027.mtv.corp.google.com raised Exception command execution error
* Command: 
    /usr/bin/ssh -a -x  -o ControlPath=/tmp/_autotmp_XiMQuBssh-master/socket
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpxz87sw -o
    BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=1 -l chromeos-test -p 22 cros-
    full-0027.mtv.corp.google.com " if type \"logger\" > /dev/null 2>&1; then
    logger -tag \"autotest\"
    \"server[stack::_execute_calls|_execute_calls_impl|run] -> ssh_run(python
    /usr/local/autotest/scheduler/drone_utility.py --call_time
    1520034398.03)\";fi; python /usr/local/autotest/scheduler/drone_utility.py
    --call_time 1520034398.03"
Exit status: 1
Duration: 0.174740076065

stderr:
Traceback (most recent call last):
  File "/usr/local/autotest/scheduler/drone_utility.py", line 670, in <module>
    main()
  File "/usr/local/autotest/scheduler/drone_utility.py", line 665, in main
    return_value = drone_utility.execute_calls(calls)
  File "/usr/local/autotest/scheduler/drone_utility.py", line 422, in execute_calls
    results.append(method_call.execute_on(self))
  File "/usr/local/autotest/scheduler/drone_utility.py", line 66, in execute_on
    return method(*self._args, **self._kwargs)
  File "/usr/local/autotest/scheduler/drone_utility.py", line 240, in write_to_file
    self._ensure_directory_exists(os.path.dirname(file_path))
  File "/usr/local/autotest/scheduler/drone_utility.py", line 191, in _ensure_directory_exists
    os.makedirs(path)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/autotest/results/hosts/chromeos6-row1-rack7-host21/62619156-repair'

Comment 1 by nxia@chromium.org, Mar 3 2018 

autotest/scheduler/tick   http://shortn/_TZ3kn38mtj

Comment 2 by nxia@chromium.org, Mar 3 2018 

the folder is owned by root in cros-full-0027.mtv.corp.google.com 

chromeos-test@cros-full-0027:/usr/local/autotest/results/hosts$ ls -lts chromeos6-row1-rack7-host21
total 4
4 drwxr-xr-x 7 root root 4096 Feb 21 19:18 62495156-repair

Comment 3 by nxia@chromium.org, Mar 3 2018

Labels: Chase-Pending
I removed the folder /usr/local/autotest/results/hosts/chromeos6-row1-rack7-host21

the log it contains is https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos6-row1-rack7-host21/62495156-repair/debug/

Comment 4 by nxia@chromium.org, Mar 3 2018

Summary: cros-full-0027 raised permission exception when trying to create a path owned by root (was: master scheduler went down )

Comment 5 by nxia@chromium.org, Mar 3 2018

Description: Show this description

Comment 6 by akes...@chromium.org, Mar 5 2018

Labels: -Pri-1 -Chase-Pending Pri-2
Status: Unconfirmed (was: Untriaged)
Awaiting further occurrences.

Comment 7 by nxia@chromium.org, Jun 8 2018

Cc: -nxia@chromium.org
►

Sign in to add a comment