gs_offloader can crash - probably a threading race condition |
||||
Issue description
gs_offloader crashes
Traceback (most recent call last):
File "/usr/local/autotest/site_utils/gs_offloader.py", line 1211, in <module>
main()
File "/usr/local/autotest/site_utils/gs_offloader.py", line 1165, in main
offloader.offload_once()
File "/usr/local/autotest/site_utils/gs_offloader.py", line 970, in offload_once
_enqueue_offload(job, queue, self._upload_age_limit)
File "/usr/local/autotest/site_utils/gs_offloader.py", line 1047, in _enqueue_offload
if job.process_gs_instructions():
File "/usr/local/autotest/site_utils/job_directories.py", line 181, in process_gs_instructions
if not os.listdir(self.dirname):
OSError: [Errno 2] No such file or directory: '1737-moblab'
,
Jul 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3bc3be032e5005fe7a9206f24c29696a83811495 commit 3bc3be032e5005fe7a9206f24c29696a83811495 Author: Keith Haddow <haddowk@chromium.org> Date: Mon Jul 16 23:31:59 2018 [autotest] Fix issue where offloaded jobs are never pruned. Jobs are open until they are pruned, not necessarily just uploaded change the code to reflect this. When checking a directory exists an OSError can be thrown, handle that as if the directory does not exist. TEST=tryjob BUG= chromium:863470 , chromium:863185 Change-Id: I58127c521dbd666551465e117b44f78b6730a173 Reviewed-on: https://chromium-review.googlesource.com/1136793 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/3bc3be032e5005fe7a9206f24c29696a83811495/site_utils/job_directories.py [modify] https://crrev.com/3bc3be032e5005fe7a9206f24c29696a83811495/site_utils/gs_offloader.py
,
Jul 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c5625e52d4682ca7e2366f1dce895429f7a38475 commit c5625e52d4682ca7e2366f1dce895429f7a38475 Author: Keith Haddow <haddowk@chromium.org> Date: Mon Jul 16 23:41:23 2018 [autotest] Fix issue where offloaded jobs are never pruned. Jobs are open until they are pruned, not necessarily just uploaded change the code to reflect this. When checking a directory exists an OSError can be thrown, handle that as if the directory does not exist. TEST=tryjob BUG= chromium:863470 , chromium:863185 Change-Id: I58127c521dbd666551465e117b44f78b6730a173 Reviewed-on: https://chromium-review.googlesource.com/1136793 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> (cherry picked from commit 3bc3be032e5005fe7a9206f24c29696a83811495) Reviewed-on: https://chromium-review.googlesource.com/1139113 Reviewed-by: Keith Haddow <haddowk@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [modify] https://crrev.com/c5625e52d4682ca7e2366f1dce895429f7a38475/site_utils/job_directories.py [modify] https://crrev.com/c5625e52d4682ca7e2366f1dce895429f7a38475/site_utils/gs_offloader.py
,
Jul 16
,
Jul 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cb0298cb76c8dae7609f7b512bc7fa451890da08 commit cb0298cb76c8dae7609f7b512bc7fa451890da08 Author: Keith Haddow <haddowk@chromium.org> Date: Mon Jul 16 23:44:06 2018 [autotest] Fix issue where offloaded jobs are never pruned. Jobs are open until they are pruned, not necessarily just uploaded change the code to reflect this. When checking a directory exists an OSError can be thrown, handle that as if the directory does not exist. TEST=tryjob BUG= chromium:863470 , chromium:863185 Change-Id: I58127c521dbd666551465e117b44f78b6730a173 Reviewed-on: https://chromium-review.googlesource.com/1136793 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> (cherry picked from commit 3bc3be032e5005fe7a9206f24c29696a83811495) Reviewed-on: https://chromium-review.googlesource.com/1139114 Reviewed-by: Keith Haddow <haddowk@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [modify] https://crrev.com/cb0298cb76c8dae7609f7b512bc7fa451890da08/site_utils/job_directories.py [modify] https://crrev.com/cb0298cb76c8dae7609f7b512bc7fa451890da08/site_utils/gs_offloader.py |
||||
►
Sign in to add a comment |
||||
Comment 1 by haddowk@chromium.org
, Jul 13Owner: haddowk@chromium.org
Status: Assigned (was: Untriaged)