Moblab failing to offload |
|||||||
Issue description
moblab@localhost /var/log/apache2 $ /usr/bin/python /usr/local/autotest/site_utils/gs_offloader.py --age_to_upload=0 --age_to_delete=3 -i --log_size=200
DEBUG:root:Set process to nice value: 10
DEBUG:root:Set process to ionice IDLE
DEBUG:root:Offloading Autotest results in /usr/local/autotest/results
DEBUG:root:Running 'readlink -f /sys/class/net/eth0'
DEBUG:root:Running 'readlink -f /sys/class/net/eth1'
INFO:root:Eth0 is a USB dongle, use eth1 as moblab nic.
DEBUG:root:Running 'ip addr show eth1 | grep link/ether'
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal
DEBUG:root:Offloading to: gs://chromeos-moblab-haddowktest/54:ab:3a:00:b2:4b/d02cc28e-0db1-11e7-9e64-54ab3a00b24b/
INFO:root:Offloader multiprocessing is set to:False
Copying file:///tmp/tmpqQcLlM [Content-Type=application/octet-stream]...
INFO:root:Configuration file does not exist, ignoring: /etc/chrome-infra/ts-mon.json
ERROR:root:ts_mon monitoring is disabled because the endpoint provided is invalid or not supported:
NOTICE:root:ts_mon was set up.
Removing gs://chromeos-moblab-haddowktest/54:ab:3a:00:b2:4b/d02cc28e-0db1-11e7-9e64-54ab3a00b24b/tmpqQcLlM...
DEBUG:root:Start of offload cycle - found 12 new jobs
INFO:root:Waiting for ts_mon flushing process to finish...
INFO:root:Finished waiting for ts_mon process.
Traceback (most recent call last):
File "/usr/local/autotest/site_utils/gs_offloader.py", line 1042, in <module>
main()
File "/usr/local/autotest/site_utils/gs_offloader.py", line 1035, in main
offloader.offload_once()
File "/usr/local/autotest/site_utils/gs_offloader.py", line 858, in offload_once
job.enqueue_offload(queue, self._upload_age_limit)
File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
queue.put(_AllTasksComplete())
File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
queue.put(_AllTasksComplete())
File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 561, in ParallelTasks
raise BackgroundFailure(exc_infos=errors)
chromite.lib.parallel.BackgroundFailure: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'seek'
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/chromite/lib/parallel.py", line 602, in TaskRunner
task(*x, **task_kwargs)
File "/usr/lib64/python2.7/site-packages/chromite/lib/metrics.py", line 306, in wrapper
return fn(*args, **kwargs)
File "/usr/local/autotest/site_utils/gs_offloader.py", line 641, in offload_dir
stdout_file.seek(0)
AttributeError: 'NoneType' object has no attribute 'seek'
,
Jun 19 2017
,
Jun 19 2017
,
Jun 19 2017
probably that is also why i could not see logs from the subprocess.
,
Jun 19 2017
I think one of my gs_offloader refactor commits fixes this, but those commits are stuck in pre-cq hell because of mox tests that only fail on trybots
Anyway, the moral of the story is to use context managers instead of managing things like
stdout_file = None
try:
<100s of lines>
stdout_file = open()
<100s of lines>
except error:
<100s of lines>
stdout_file.seek()
finally:
stdout_file.close()
,
Jul 20 2017
is your fix landed?
,
Jul 20 2017
,
Jul 20 2017
Yes, it should no longer be possible for stdout_file to be None https://cs.corp.google.com/android/external/autotest/site_utils/gs_offloader.py?q=gs_offloader&dr=C&l=537
,
Jan 22 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by haddowk@chromium.org
, Jun 19 2017