New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 734647 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Jul 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 747056



Sign in to add a comment

Moblab failing to offload

Project Member Reported by haddowk@chromium.org, Jun 19 2017

Issue description

moblab@localhost /var/log/apache2 $ /usr/bin/python /usr/local/autotest/site_utils/gs_offloader.py --age_to_upload=0 --age_to_delete=3 -i --log_size=200
DEBUG:root:Set process to nice value: 10
DEBUG:root:Set process to ionice IDLE
DEBUG:root:Offloading Autotest results in /usr/local/autotest/results
DEBUG:root:Running 'readlink -f /sys/class/net/eth0'
DEBUG:root:Running 'readlink -f /sys/class/net/eth1'
INFO:root:Eth0 is a USB dongle, use eth1 as moblab nic.
DEBUG:root:Running 'ip addr show eth1 | grep link/ether'
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal
DEBUG:root:Offloading to: gs://chromeos-moblab-haddowktest/54:ab:3a:00:b2:4b/d02cc28e-0db1-11e7-9e64-54ab3a00b24b/
INFO:root:Offloader multiprocessing is set to:False
Copying file:///tmp/tmpqQcLlM [Content-Type=application/octet-stream]...
INFO:root:Configuration file does not exist, ignoring: /etc/chrome-infra/ts-mon.json
ERROR:root:ts_mon monitoring is disabled because the endpoint provided is invalid or not supported: 
NOTICE:root:ts_mon was set up.
Removing gs://chromeos-moblab-haddowktest/54:ab:3a:00:b2:4b/d02cc28e-0db1-11e7-9e64-54ab3a00b24b/tmpqQcLlM...
DEBUG:root:Start of offload cycle - found 12 new jobs
INFO:root:Waiting for ts_mon flushing process to finish...
INFO:root:Finished waiting for ts_mon process.
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 1042, in <module>
    main()
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 1035, in main
    offloader.offload_once()
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 858, in offload_once
    job.enqueue_offload(queue, self._upload_age_limit)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 561, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
chromite.lib.parallel.BackgroundFailure: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'seek'
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/usr/lib64/python2.7/site-packages/chromite/lib/metrics.py", line 306, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/autotest/site_utils/gs_offloader.py", line 641, in offload_dir
    stdout_file.seek(0)
AttributeError: 'NoneType' object has no attribute 'seek'



 
Cc: dshi@chromium.org jrbarnette@chromium.org ayatane@chromium.org
Cc: ntang@chromium.org
Labels: OS-Chrome

Comment 4 by ntang@google.com, Jun 19 2017

probably that is also why i could not see logs from the subprocess.

Comment 5 Deleted

I think one of my gs_offloader refactor commits fixes this, but those commits are stuck in pre-cq hell because of mox tests that only fail on trybots

Anyway, the moral of the story is to use context managers instead of managing things like

stdout_file = None
try:
    <100s of lines>
    stdout_file = open()
    <100s of lines>
except error:
    <100s of lines>
    stdout_file.seek()
finally:
    stdout_file.close()

Owner: ayatane@chromium.org
is your fix landed?
Blocking: 747056
Status: Fixed (was: Untriaged)
Yes, it should no longer be possible for stdout_file to be None

https://cs.corp.google.com/android/external/autotest/site_utils/gs_offloader.py?q=gs_offloader&dr=C&l=537

Comment 10 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment