Staging test failing because skylab-drone is out of space |
|||
Issue description[chromeos-staging-master2.hot.corp.google.com] out: stdout: [chromeos-staging-master2.hot.corp.google.com] out: Updating server: pprabhu-skylab-drone-2.cbf.corp.google.com [chromeos-staging-master2.hot.corp.google.com] out: Checking tree status: [chromeos-staging-master2.hot.corp.google.com] out: Tree status: clean [chromeos-staging-master2.hot.corp.google.com] out: Updating Repo. [chromeos-staging-master2.hot.corp.google.com] out: stderr: [chromeos-staging-master2.hot.corp.google.com] out: error: file write error: No space left on device [chromeos-staging-master2.hot.corp.google.com] out: fatal: unable to write sha1 file [chromeos-staging-master2.hot.corp.google.com] out: fatal: unpack-objects failed [chromeos-staging-master2.hot.corp.google.com] out: error: file write error: No space left on device [chromeos-staging-master2.hot.corp.google.com] out: fatal: unable to write sha1 file [chromeos-staging-master2.hot.corp.google.com] out: fatal: unpack-objects failed [chromeos-staging-master2.hot.corp.google.com] out: error: file write error: No space left on device [chromeos-staging-master2.hot.corp.google.com] out: fatal: unable to write sha1 file [chromeos-staging-master2.hot.corp.google.com] out: fatal: unpack-objects failed [chromeos-staging-master2.hot.corp.google.com] out: error: file write error: No space left on device [chromeos-staging-master2.hot.corp.google.com] out: fatal: unable to write sha1 file [chromeos-staging-master2.hot.corp.google.com] out: fatal: unpack-objects failed [chromeos-staging-master2.hot.corp.google.com] out: error: Cannot fetch chromiumos/third_party/autotest [chromeos-staging-master2.hot.corp.google.com] out: [chromeos-staging-master2.hot.corp.google.com] out: error: Exited sync due to fetch errors [chromeos-staging-master2.hot.corp.google.com] out: Traceback (most recent call last): [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 545, in <module> [chromeos-staging-master2.hot.corp.google.com] out: sys.exit(main(sys.argv[1:])) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 526, in main [chromeos-staging-master2.hot.corp.google.com] out: repo_sync(behaviors.update_push_servers) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/local/autotest/site_utils/deploy_server_local.py", line 179, in repo_sync [chromeos-staging-master2.hot.corp.google.com] out: subprocess.check_output(['repo', 'sync', '--force-sync']) [chromeos-staging-master2.hot.corp.google.com] out: File "/usr/lib/python2.7/subprocess.py", line 573, in check_output [chromeos-staging-master2.hot.corp.google.com] out: raise CalledProcessError(retcode, cmd, output=output) [chromeos-staging-master2.hot.corp.google.com] out: subprocess.CalledProcessError: Command '['repo', 'sync', '--force-sync']' returned non-zero exit status 1
,
Jul 19
Started rising Jul 12: http://shortn/_4rLn8lb5Sr
,
Jul 19
,
Jul 19
pprabhu-skylab-drone-3 shows similar behaviour: http://shortn/_q6pVzQQBkO
,
Jul 19
Actually, a lot of space is being eaten up by autotest logs: chromeos-test@pprabhu-skylab-drone-2:/usr/local/autotest/logs$ du -sh . 39G . Full of tons of non-trivial at times gs_offloader logs: chromeos-test@pprabhu-skylab-drone-2:/usr/local/autotest/logs$ ls gs_offloader* | wc 33450 33450 1421587 chromeos-test@pprabhu-skylab-drone-2:/usr/local/autotest/logs$ ls -Shl | head -n 50 total 39G -rw-r--r-- 1 chromeos-test eng 2.0G Jul 17 13:01 gs_offloader_hosts_log_20180717_010124.txt -rw-r--r-- 1 chromeos-test eng 2.0G Jul 17 13:01 gs_offloader_jobs_log_20180717_010124.txt -rw-r--r-- 1 chromeos-test eng 1.2G Jul 17 01:01 gs_offloader_hosts_log_20180716_170157.txt -rw-r--r-- 1 chromeos-test eng 1.2G Jul 17 01:01 gs_offloader_jobs_log_20180716_170157.txt -rw-r--r-- 1 chromeos-test eng 678M Jul 14 09:01 gs_offloader_jobs_log_20180714_010110.txt -rw-r--r-- 1 chromeos-test eng 678M Jul 14 09:01 gs_offloader_hosts_log_20180714_010111.txt -rw-r--r-- 1 chromeos-test eng 591M Jul 16 17:01 gs_offloader_hosts_log_20180716_130117.txt -rw-r--r-- 1 chromeos-test eng 591M Jul 16 17:01 gs_offloader_jobs_log_20180716_130111.txt -rw-r--r-- 1 chromeos-test eng 575M Jul 16 13:01 gs_offloader_jobs_log_20180716_090106.txt -rw-r--r-- 1 chromeos-test eng 575M Jul 16 13:01 gs_offloader_hosts_log_20180716_090106.txt -rw-r--r-- 1 chromeos-test eng 566M Jul 16 09:01 gs_offloader_hosts_log_20180716_050115.txt -rw-r--r-- 1 chromeos-test eng 566M Jul 16 09:01 gs_offloader_jobs_log_20180716_050115.txt -rw-r--r-- 1 chromeos-test eng 552M Jul 16 05:01 gs_offloader_hosts_log_20180716_010122.txt -rw-r--r-- 1 chromeos-test eng 552M Jul 16 05:01 gs_offloader_jobs_log_20180716_010121.txt -rw-r--r-- 1 chromeos-test eng 536M Jul 16 01:01 gs_offloader_hosts_log_20180715_210125.txt -rw-r--r-- 1 chromeos-test eng 536M Jul 16 01:01 gs_offloader_jobs_log_20180715_210125.txt -rw-r--r-- 1 chromeos-test eng 522M Jul 15 21:01 gs_offloader_jobs_log_20180715_170107.txt -rw-r--r-- 1 chromeos-test eng 522M Jul 15 21:01 gs_offloader_hosts_log_20180715_170107.txt -rw-r--r-- 1 chromeos-test eng 500M Jul 15 17:01 gs_offloader_hosts_log_20180715_130110.txt -rw-r--r-- 1 chromeos-test eng 500M Jul 15 17:01 gs_offloader_jobs_log_20180715_130110.txt -rw-r--r-- 1 chromeos-test eng 493M Jul 15 13:01 gs_offloader_hosts_log_20180715_090102.txt -rw-r--r-- 1 chromeos-test eng 493M Jul 15 13:01 gs_offloader_jobs_log_20180715_090101.txt -rw-r--r-- 1 chromeos-test eng 487M Jul 15 09:01 gs_offloader_hosts_log_20180715_050106.txt -rw-r--r-- 1 chromeos-test eng 487M Jul 15 09:00 gs_offloader_jobs_log_20180715_050105.txt -rw-r--r-- 1 chromeos-test eng 477M Jul 15 05:01 gs_offloader_hosts_log_20180715_010111.txt -rw-r--r-- 1 chromeos-test eng 477M Jul 15 05:01 gs_offloader_jobs_log_20180715_010111.txt -rw-r--r-- 1 chromeos-test eng 454M Jul 15 01:01 gs_offloader_hosts_log_20180714_210137.txt -rw-r--r-- 1 chromeos-test eng 454M Jul 15 01:01 gs_offloader_jobs_log_20180714_210136.txt -rw-r--r-- 1 chromeos-test eng 429M Jul 14 21:01 gs_offloader_hosts_log_20180714_170113.txt -rw-r--r-- 1 chromeos-test eng 429M Jul 14 21:01 gs_offloader_jobs_log_20180714_170112.txt -rw-r--r-- 1 chromeos-test eng 398M Jul 14 17:01 gs_offloader_jobs_log_20180714_130105.txt -rw-r--r-- 1 chromeos-test eng 398M Jul 14 17:01 gs_offloader_hosts_log_20180714_130106.txt -rw-r--r-- 1 chromeos-test eng 372M Jul 14 13:01 gs_offloader_hosts_log_20180714_090111.txt -rw-r--r-- 1 chromeos-test eng 371M Jul 14 13:01 gs_offloader_jobs_log_20180714_090111.txt -rw-r--r-- 1 chromeos-test eng 366M Jul 18 15:01 gs_offloader_hosts_log_20180718_130118.txt -rw-r--r-- 1 chromeos-test eng 364M Jul 18 15:01 gs_offloader_jobs_log_20180718_130118.txt -rw-r--r-- 1 chromeos-test eng 363M Jul 18 13:01 gs_offloader_jobs_log_20180718_110116.txt -rw-r--r-- 1 chromeos-test eng 363M Jul 18 13:01 gs_offloader_hosts_log_20180718_110117.txt -rw-r--r-- 1 chromeos-test eng 360M Jul 18 11:01 gs_offloader_hosts_log_20180718_090112.txt -rw-r--r-- 1 chromeos-test eng 360M Jul 18 11:01 gs_offloader_jobs_log_20180718_090112.txt -rw-r--r-- 1 chromeos-test eng 354M Jul 18 09:01 gs_offloader_hosts_log_20180718_070107.txt -rw-r--r-- 1 chromeos-test eng 354M Jul 18 09:01 gs_offloader_jobs_log_20180718_070107.txt -rw-r--r-- 1 chromeos-test eng 350M Jul 18 07:01 gs_offloader_jobs_log_20180718_050102.txt -rw-r--r-- 1 chromeos-test eng 350M Jul 18 07:01 gs_offloader_hosts_log_20180718_050103.txt -rw-r--r-- 1 chromeos-test eng 350M Jul 17 21:01 gs_offloader_hosts_log_20180717_190106.txt -rw-r--r-- 1 chromeos-test eng 349M Jul 17 21:01 gs_offloader_jobs_log_20180717_190106.txt -rw-r--r-- 1 chromeos-test eng 349M Jul 18 05:01 gs_offloader_hosts_log_20180718_030104.txt -rw-r--r-- 1 chromeos-test eng 349M Jul 18 05:01 gs_offloader_jobs_log_20180718_030104.txt -rw-r--r-- 1 chromeos-test eng 349M Jul 18 03:01 gs_offloader_hosts_log_20180718_010101.txt
,
Jul 19
About 30G was taken by autotest results, which were not being offloaded due to issue 863192
,
Jul 19
gs_offloader logs are mostly full of: 2018-07-17 01:01:28,407 - DEBUG - Error opening .ready_for_offload for swarming-3ea8f8e5f1223411: [Errno 2] No such file or directory: 'swarming-3ea8f8e5f1223411/.ready_for_offload' 2018-07-17 01:01:28,407 - DEBUG - Error opening .ready_for_offload for swarming-3eace0d8485c9e11: [Errno 2] No such file or directory: 'swarming-3eace0d8485c9e11/.ready_for_offload' 2018-07-17 01:01:28,407 - DEBUG - Error opening .ready_for_offload for swarming-3eb701809d388b11: [Errno 2] No such file or directory: 'swarming-3eb701809d388b11/.ready_for_offload' 2018-07-17 01:01:28,407 - DEBUG - Error opening .ready_for_offload for swarming-3eb38ffd4fff7811: [Errno 2] No such file or directory: 'swarming-3eb38ffd4fff7811/.ready_for_offload' 2018-07-17 01:01:28,407 - DEBUG - Error opening .ready_for_offload for swarming-3ea891e7bec44211: [Errno 2] No such file or directory: 'swarming-3ea891e7bec44211/.ready_for_offload' 2018-07-17 01:01:28,407 - DEBUG - Error opening .ready_for_offload for swarming-3eaf3115fd563611: [Errno 2] No such file or directory: 'swarming-3eaf3115fd563611/.ready_for_offload' 2018-07-17 01:01:28,408 - DEBUG - Error opening .ready_for_offload for swarming-3eaeb8629ad65011: [Errno 2] No such file or directory: 'swarming-3eaeb8629ad65011/.ready_for_offload' 2018-07-17 01:01:28,408 - DEBUG - Error opening .ready_for_offload for swarming-3eb3370adc06c611: [Errno 2] No such file or directory: 'swarming-3eb3370adc06c611/.ready_for_offload' 2018-07-17 01:01:28,408 - DEBUG - Error opening .ready_for_offload for swarming-3eb56a17c7e6c011: [Errno 2] No such file or directory: 'swarming-3eb56a17c7e6c011/.ready_for_offload' 2018-07-17 01:01:28,408 - DEBUG - Error opening .ready_for_offload for swarming-3eafbefbe8026911: [Errno 2] No such file or directory: 'swarming-3eafbefbe8026911/.ready_for_offload' 2018-07-17 01:01:28,408 - DEBUG - Error opening .ready_for_offload for swarming-3ea8d0dff3828711: [Errno 2] No such file or directory: 'swarming-3ea8d0dff3828711/.ready_for_offload' 2018-07-17 01:01:28,408 - DEBUG - Error opening .ready_for_offload for swarming-3eb9c06208cb8111: [Errno 2] No such file or directory: 'swarming-3eb9c06208cb8111/.ready_for_offload'
,
Jul 19
Manually cleaned the drones for now, this should unblockd staging lab. Have some CLs to reduce spam and cleanup logdir better in flight.
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f587aabefdcf1188ede7a7e7fb7efbb527aac85c commit f587aabefdcf1188ede7a7e7fb7efbb527aac85c Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Jul 20 12:31:27 2018 autotest: Do not log missing file for unsealed job directories This is the expected path for all jobs that are not yet finished. Logging (even at debug level) is creating too much span in gs_offloader logs. BUG= chromium:865561 TEST=None Change-Id: I09eadaefc53282e874f23c5873177ddd063659bf Reviewed-on: https://chromium-review.googlesource.com/1144180 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/f587aabefdcf1188ede7a7e7fb7efbb527aac85c/site_utils/job_directories.py
,
Jul 20
,
Jul 24
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/3d4a003647c7c58c024bef9a17e8465664fbddf6 commit 3d4a003647c7c58c024bef9a17e8465664fbddf6 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Jul 24 15:36:52 2018 |
|||
►
Sign in to add a comment |
|||
Comment 1 by pprabhu@chromium.org
, Jul 19