New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 773093 link

Starred by 1 user
Status: Duplicate
Merged: issue 770868
Owner:
█ nxia@chromium.org
Last visit > 30 days ago
Closed: Oct 2017
Cc:
█ dshi@chromium.org
█ shuqianz@chromium.org
dgarr...@chromium.org
ayatane@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: Bug



Sign in to add a comment

push-to-prod on chromeos-server104 failed with UnstableService gs_offloader

Project Member Reported by nxia@chromium.org, Oct 9 2017 
chromeos-test@chromeos-server104:/usr/local/.repo/projects/autotest.git/objects$  /usr/local/autotest/site_utils/deploy_server_local.py
Checking tree status:
Tree status: clean
Updating Repo.
Fetching projects: 100% (2/2), done.  

Updating server to prod branch
Removing .pyc files
Updating ~chromeos-test/chromiumos
Fetching project chromiumos/infra_virtualenv
Fetching project chromiumos/chromite
Fetching projects: 100% (2/2), done.  

Your sources have been sync'd successfully.
Removing .pyc files
Running update commands: build_externals
Running: build_externals: /usr/local/autotest/utils/build_externals.py
Restarting Services: sysmon, scheduler, host-scheduler, shard-client, gs_offloader, gs_offloader_s, rpc-logserver
Restarting: sysmon
sysmon stop/waiting
sysmon start/running, process 31386
Restarting: scheduler
scheduler stop/waiting
scheduler start/running, process 31892
Restarting: host-scheduler
host-scheduler stop/waiting
host-scheduler start/running, process 2622
Restarting: shard-client
shard-client stop/waiting
shard-client start/running, process 5164
Restarting: gs_offloader
gs_offloader stop/waiting
gs_offloader start/running, process 5514
Restarting: gs_offloader_s
gs_offloader_s stop/waiting
gs_offloader_s start/running, process 5589
Restarting: rpc-logserver
rpc-logserver stop/waiting
rpc-logserver start/running, process 5670
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 519, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 511, in main
    use_chromite_master=behaviors.update_push_servers)
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 360, in run_deploy_actions
    skip_service_status=skip_service_status)
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 330, in restart_services
    raise UnstableServices(unstable_services)
__main__.UnstableServices: ['gs_offloader']

Comment 1 by nxia@chromium.org, Oct 9 2017 

gs_offloader has permission issues:

chromeos-test@chromeos-server104:/var/log$ cat gs_offloader.log.1 
DEBUG:root:Set process to nice value: 10
DEBUG:root:Set process to ionice IDLE
DEBUG:root:Offloading Autotest results in /usr/local/autotest/results
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal
DEBUG:root:Offloading to: gs://chromeos-autotest-results/
INFO:root:Offloader multiprocessing is set to:False
NOTICE:root:ts_mon was set up.
Copying file:///tmp/tmpAXjXw1 [Content-Type=inode/x-empty]...
Removing gs://chromeos-autotest-results/tmpAXjXw1...:            0 B    
DEBUG:root:Start of offload cycle - found 41 new jobs
INFO:oauth2client.client:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
DEBUG:root:Running 'gsutil cp -eR 147792412-kkashamsetty/chromeos4-row12-rack8-host11/cheets_CTS_N.CtsDramTestCases/results/android-cts/2017.10.09_15.23.33.zip gs://chromeos-cts-apfe/edgar-release/R61-9765.81.0/147792411/cheets_CTS_N.CtsDramTestCases/147792412-kkashamsetty_2017.10.09_15.23.33/'
DEBUG:root:Upload 147792412-kkashamsetty/chromeos4-row12-rack8-host11/cheets_CTS_N.CtsDramTestCases/results/android-cts/2017.10.09_15.23.33.zip to gs://chromeos-cts-apfe/edgar-release/R61-9765.81.0/147792411/cheets_CTS_N.CtsDramTestCases/147792412-kkashamsetty_2017.10.09_15.23.33/ 
DEBUG:root:Running 'gsutil cp -eR 147792412-kkashamsetty/chromeos4-row12-rack8-host11/cheets_CTS_N.CtsDramTestCases/results/android-cts/2017.10.09_15.23.33/test_result.xml.gz gs://chromeos-cts-results/cheets_CTS_N.CtsDramTestCases/147792412-kkashamsetty_2017.10.09_15.23.33/'
DEBUG:root:Zip and upload 147792412-kkashamsetty/chromeos4-row12-rack8-host11/cheets_CTS_N.CtsDramTestCases/results/android-cts/2017.10.09_15.23.33/test_result.xml.gz to gs://chromeos-cts-results/cheets_CTS_N.CtsDramTestCases/147792412-kkashamsetty_2017.10.09_15.23.33/
DEBUG:root:Removing symlink 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/vmlog/vmlog.LATEST
DEBUG:root:Creating marker 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/vmlog/vmlog.LATEST
DEBUG:root:Removing symlink 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/vmlog/vmlog.PREVIOUS
DEBUG:root:Creating marker 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/vmlog/vmlog.PREVIOUS
DEBUG:root:Removing symlink 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/power_manager/powerd.PREVIOUS
DEBUG:root:Creating marker 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/power_manager/powerd.PREVIOUS
DEBUG:root:Removing symlink 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/power_manager/powerd.LATEST
DEBUG:root:Creating marker 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/power_manager/powerd.LATEST
DEBUG:root:Removing symlink 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/ui/ui.LATEST
DEBUG:root:Creating marker 147804011-gkihumba/chromeos2-row4-rack1-host6/sysinfo/mnt/stateful_partition/unencrypted/preserve/log/ui/ui.LATEST
DEBUG:root:Running 'gsutil cp -eR 147792414-kkashamsetty/chromeos2-row5-rack7-host19/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/results/android-cts/2017.10.09_15.23.41.zip gs://chromeos-cts-apfe/edgar-release/R61-9765.81.0/147792411/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/147792414-kkashamsetty_2017.10.09_15.23.41/'
DEBUG:root:Upload 147792414-kkashamsetty/chromeos2-row5-rack7-host19/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/results/android-cts/2017.10.09_15.23.41.zip to gs://chromeos-cts-apfe/edgar-release/R61-9765.81.0/147792411/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/147792414-kkashamsetty_2017.10.09_15.23.41/ 
DEBUG:root:Running 'gsutil cp -eR 147792414-kkashamsetty/chromeos2-row5-rack7-host19/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/results/android-cts/2017.10.09_15.23.41/test_result.xml.gz gs://chromeos-cts-results/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/147792414-kkashamsetty_2017.10.09_15.23.41/'
DEBUG:root:Zip and upload 147792414-kkashamsetty/chromeos2-row5-rack7-host19/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/results/android-cts/2017.10.09_15.23.41/test_result.xml.gz to gs://chromeos-cts-results/cheets_CTS_N.CtsNetSecConfigInvalidPinTestCases/147792414-kkashamsetty_2017.10.09_15.23.41/
INFO:root:Waiting for ts_mon flushing process to finish...
INFO:root:Finished waiting for ts_mon process.
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 1146, in <module>
    main()
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 1100, in main
    offloader.offload_once()
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 914, in offload_once
    _enqueue_offload(job, queue, self._upload_age_limit)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 751, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 751, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 562, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
chromite.lib.parallel.BackgroundFailure: <type 'exceptions.OSError'>: [Errno 13] Permission denied
Traceback (most recent call last):
  File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line 603, in TaskRunner
    task(*x, **task_kwargs)
  File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 483, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 561, in offload
    self._offload(dir_entry, dest_path, stdout_file, stderr_file)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 607, in _offload
    sanitize_dir(dir_entry)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 211, in sanitize_dir
    _escape_rename_dir_contents(dirpath)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 227, in _escape_rename_dir_contents
    _escape_rename_dir_contents(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 227, in _escape_rename_dir_contents
    _escape_rename_dir_contents(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 227, in _escape_rename_dir_contents
    _escape_rename_dir_contents(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 227, in _escape_rename_dir_contents
    _escape_rename_dir_contents(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 227, in _escape_rename_dir_contents
    _escape_rename_dir_contents(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 227, in _escape_rename_dir_contents
    _escape_rename_dir_contents(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 223, in _escape_rename_dir_contents
    _escape_rename(path)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 238, in _escape_rename
    os.rename(path, sanitized_path)
OSError: [Errno 13] Permission denied

Comment 2 by nxia@chromium.org, Oct 9 2017

Cc: dshi@chromium.org ayatane@chromium.org

Comment 3 by shuqianz@chromium.org, Oct 9 2017

Mergedinto: 770868
Status: Duplicate (was: Untriaged)
I think this is the same issue we discussed in today's meeting?

Comment 4 by nxia@chromium.org, Oct 9 2017 

restarting is the last step of deploy_server_local? I can assume the push-to-prod for this server completed and move on the the Group 3?

Comment 5 by shuqianz@chromium.org, Oct 9 2017 

yes, service restart is the last step. You can move on.

Comment 6 by dgarr...@chromium.org, Oct 9 2017 

You can also fix the issues (temporarily) with:

  sudo chown -R chromeos-test:eng /usr/local/autotest/results
►

Sign in to add a comment