New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 755226 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Aug 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

result_tools/utils failed with utils_lib.NotEnoughDiskError: Not enough disk space after saving the summary file.

Project Member Reported by warx@chromium.org, Aug 14 2017

Issue description

see it in tricky-tot-chrome-pfq-informational #5621
error log:

08/14 10:12:11.801 ERROR|   logging_manager:0626| tko parser: MACHINE NAME: chromeos4-row2-rack3-host6
08/14 10:12:11.801 ERROR|   logging_manager:0626| tko parser: MACHINE GROUP: tricky
08/14 10:12:11.801 ERROR|   logging_manager:0626| tko parser: parsing partial test ---- SERVER_JOB
08/14 10:12:11.802 ERROR|   logging_manager:0626| tko parser: parsing partial test None CLIENT_JOB.0
08/14 10:12:11.802 ERROR|   logging_manager:0626| tko parser: RUNNING: RUNNING
08/14 10:12:11.802 ERROR|   logging_manager:0626| CLIENT_JOB.0
08/14 10:12:11.802 ERROR|   logging_manager:0626| 
08/14 10:12:11.803 ERROR|   logging_manager:0626| tko parser: parsing partial test platform_ToolchainOptions platform_ToolchainOptions
08/14 10:12:11.803 ERROR|   logging_manager:0626| tko parser: RUNNING: RUNNING
08/14 10:12:11.803 ERROR|   logging_manager:0626| Subdir: platform_ToolchainOptions
08/14 10:12:11.803 ERROR|   logging_manager:0626| Testname: platform_ToolchainOptions
08/14 10:12:11.804 ERROR|   logging_manager:0626| 
08/14 10:12:11.804 ERROR|   logging_manager:0626| tko parser: update RUNNING reason: Autotest client terminated unexpectedly: DUT rebooted during the test run.
08/14 10:12:11.804 ERROR|   logging_manager:0626| tko parser: parsing test platform_ToolchainOptions platform_ToolchainOptions
08/14 10:12:11.804 ERROR|   logging_manager:0626| tko parser: ADD: FAIL
08/14 10:12:11.805 ERROR|   logging_manager:0626| Subdir: platform_ToolchainOptions
08/14 10:12:11.805 ERROR|   logging_manager:0626| Testname: platform_ToolchainOptions
08/14 10:12:11.805 ERROR|   logging_manager:0626| Autotest client terminated unexpectedly: DUT rebooted during the test run.
08/14 10:12:11.805 ERROR|   logging_manager:0626| tko parser: parsing test None CLIENT_JOB.0
08/14 10:12:11.806 ERROR|   logging_manager:0626| tko parser: ADD: GOOD
08/14 10:12:11.806 ERROR|   logging_manager:0626| Subdir: None
08/14 10:12:11.806 ERROR|   logging_manager:0626| Testname: CLIENT_JOB.0
08/14 10:12:11.807 ERROR|   logging_manager:0626| 
08/14 10:12:11.807 ERROR|   logging_manager:0626| tko parser: parsing test ---- SERVER_JOB
08/14 10:12:23.413 ERROR|             utils:0280| [stderr] Traceback (most recent call last):
08/14 10:12:23.413 ERROR|             utils:0280| [stderr]   File "/usr/local/autotest/result_tools/utils.py", line 424, in <module>
08/14 10:12:23.414 ERROR|             utils:0280| [stderr]     main()
08/14 10:12:23.414 ERROR|             utils:0280| [stderr]   File "/usr/local/autotest/result_tools/utils.py", line 420, in main
08/14 10:12:23.414 ERROR|             utils:0280| [stderr]     execute(options.path, options.max_size_KB)
08/14 10:12:23.414 ERROR|             utils:0280| [stderr]   File "/usr/local/autotest/result_tools/utils.py", line 373, in execute
08/14 10:12:23.414 ERROR|             utils:0280| [stderr]     (free_space, len(summary_json)))
08/14 10:12:23.414 ERROR|             utils:0280| [stderr] utils_lib.NotEnoughDiskError: Not enough disk space after saving the summary file. Available free disk: 0 bytes. Summary file size: 61 bytes.
08/14 10:12:23.421 ERROR|            runner:0121| Non-critical failure: Failed to create directory summary for /dev/pstore/console-ramoops.
 

Comment 1 by warx@chromium.org, Aug 14 2017

Cc: warx@chromium.org
Owner: nxia@chromium.org
more like an infra issue, assigned to infra deputy

Comment 2 by nxia@chromium.org, Aug 14 2017

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/135019024-chromeos-test/chromeos4-row2-rack3-host6/crashinfo.chromeos4-row2-rack3-host6/

I see chrome crashes during the test, can you please have someone check if the crashes caused the reboot?

Comment 3 by nxia@chromium.org, Aug 14 2017

Cc: -warx@chromium.org nxia@chromium.org
Owner: warx@chromium.org

Comment 4 by nxia@chromium.org, Aug 14 2017

other issue I see from the log is "08/14 10:12:23.414 ERROR| utils:0280| [stderr] utils_lib.NotEnoughDiskError: Not enough disk space after saving the summary file. Available free disk: 0 bytes. Summary file size: 61 bytes. 08/14 10:12:23.421 ERROR| runner:0121| Non-critical failure: Failed to create directory summary for /dev/pstore/console-ramoops."

Comment 5 by warx@chromium.org, Aug 14 2017

Labels: -Pri-1 Pri-2
p2 for it since it doesn't fail consistently. Chrome crash fix is landed. We could wait to see if this error happens again.

Comment 6 by nxia@chromium.org, Aug 15 2017

see similar issues on another run. 


https://luci-milo.appspot.com/buildbot/chromeos/quawks-paladin/418

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/134739586-chromeos-test/chromeos6-row1-rack21-host19/


08/12 22:45:19.113 DEBUG|             utils:0280| [stdout] 2017-08-12 22:45:18,008 Running result_tools/utils on path: /dev/pstore/console-ramoops
08/12 22:45:19.160 ERROR|             utils:0280| [stderr] Traceback (most recent call last):
08/12 22:45:19.161 ERROR|             utils:0280| [stderr]   File "/usr/local/autotest/result_tools/utils.py", line 424, in <module>
08/12 22:45:19.161 ERROR|             utils:0280| [stderr]     main()
08/12 22:45:19.162 ERROR|             utils:0280| [stderr]   File "/usr/local/autotest/result_tools/utils.py", line 420, in main
08/12 22:45:19.162 ERROR|             utils:0280| [stderr]     execute(options.path, options.max_size_KB)
08/12 22:45:19.163 ERROR|             utils:0280| [stderr]   File "/usr/local/autotest/result_tools/utils.py", line 373, in execute
08/12 22:45:19.164 ERROR|             utils:0280| [stderr]     (free_space, len(summary_json)))
08/12 22:45:19.165 ERROR|             utils:0280| [stderr] utils_lib.NotEnoughDiskError: Not enough disk space after saving the summary file. Available free disk: 0 bytes. Summary file size: 63 bytes.
08/12 22:45:19.167 ERROR|            runner:0121| Non-critical failure: Failed to create directory summary for /dev/pstore/console-ramoops.
Traceback (most recent call last):
  File "/usr/local/autotest/client/bin/result_tools/runner.py", line 114, in run_on_client
    timeout=_BUILD_DIR_SUMMARY_TIMEOUT)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 316, in run
    return self.run_very_slowly(*args, **kwargs)
  File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 458, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 307, in run_very_slowly
    ssh_failure_retry_ok)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 258, in _run
    raise error.AutoservRunError("command execution error", result)
AutoservRunError: command execution error
* Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_IN4L3Tssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos6-row1-rack21-host19 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::run_on_client|run|wrapper] ->
    ssh_run(/usr/local/autotest/result_tools/utils.py -p /dev/pstore/console-
    ramoops )\";fi; /usr/local/autotest/result_tools/utils.py -p /dev/pstore
    /console-ramoops "
Exit status: 1
Duration: 0.38264799118

Comment 7 by nxia@chromium.org, Aug 15 2017

Summary: result_tools/utils failed with utils_lib.NotEnoughDiskError: Not enough disk space after saving the summary file. (was: platform_ToolchainOptions: FAIL: Autotest client terminated unexpectedly: DUT rebooted during the test run)

Comment 8 by warx@chromium.org, Aug 29 2017

Status: Archived (was: Assigned)
tricky-tot-chrome-pfq-informational is healthy now

Sign in to add a comment