Not enough free space on /usr/local/autotest | intermittently on gale/jetstream/related |
|||||||||
Issue descriptionhttps://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/1098 Failed to install image during autoupdate test, due to not enough disk space error. "Not enough free space on /usr/local/autotest - 0.607GB free, want 0.700GB". 05/04 10:51:34.307 DEBUG| ssh_host:0284| Running (ssh) 'test -x /usr/local/autotest/bin/autotest' 05/04 10:51:34.694 DEBUG| ssh_host:0284| Running (ssh) 'test -w /usr/local/autotest' 05/04 10:51:35.084 DEBUG| autotest:0119| Found existing autodir at /usr/local/autotest 05/04 10:51:35.085 INFO | base_classes:0352| Checking for >= 0.7 GB of space under /usr/local/autotest on machine chromeos4-row9-rack9-host5 05/04 10:51:35.085 DEBUG| ssh_host:0284| Running (ssh) 'df -PB 1000000 /usr/local/autotest | tail -1' 05/04 10:51:35.476 DEBUG| utils:0297| [stdout] /dev/mmcblk0p1 11057 9869 607 95% /usr/local 05/04 10:51:35.518 ERROR| repair:0332| Failed: Legacy host verification checks Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 329, in _verify_host self.verify(host) File "/usr/local/autotest/server/hosts/repair.py", line 55, in verify host.verify_software() File "/usr/local/autotest/server/hosts/cros_host.py", line 1481, in verify_software super(CrosHost, self).verify_software() File "/usr/local/autotest/server/hosts/abstract_ssh.py", line 722, in verify_software self.AUTOTEST_GB_DISKSPACE_REQUIRED) File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 356, in check_diskspace raise error.AutoservDiskFullHostError(path, gb, free_space_gb) AutoservDiskFullHostError: Not enough free space on /usr/local/autotest - 0.607GB free, want 0.700GB 05/04 10:51:35.519 INFO | server_job:0184| FAIL ---- verify.cros timestamp=1493920295 localtime=May 04 10:51:35 Not enough free space on /usr/local/autotest - 0.607GB free, want 0.700GB
,
Aug 24 2017
The assigned owner "akeel@chromium.org" is not able to receive e-mails, please re-triage. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Sep 15 2017
This happened again in the latest run of gale-paladin: https://uberchromegw.corp.google.com/i/chromeos/builders/gale-paladin/builds/3882
,
Sep 15 2017
will take a look
,
Sep 15 2017
I login to the server, and run the following command root@chromeos6-row21-jetstream-host3 $ df -PB 1000000 /usr/local/autotest | tail -1 /dev/mmcblk0p1 2054 325 1607 17% /usr/local The /usr/local only consumes 17%, not the percentage shown in the task. During the test runtime, is there anything under autotest consuming the disk? Richard?
,
Sep 15 2017
> I login to the server, and run the following command The failure is a verify failure on the DUT. The shortage of space is on the DUT.
,
Sep 15 2017
Sorry, I man I login to the DUT, the output is from the DUT
,
Sep 15 2017
> Sorry, I man I login to the DUT, the output is from the DUT
Oh, so it is. Having enough space on the DUT is actually normal
after the failure. Here's the relevant history:
2017-09-15 12:11:50 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row21-jetstream-host3/61405945-repair/
2017-09-15 11:51:11 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row21-jetstream-host3/61405793-provision/
2017-09-15 09:48:56 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/142270284-chromeos-test/
2017-09-15 09:37:40 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row21-jetstream-host3/61404918-provision/
So, the provision failure triggered repair, and repair fixed the problem.
If you look in the repair logs, you'll see that the space check verifier
failed repeatedly, until finally repair powerwashed the DUT and
re-installed the software.
,
Sep 15 2017
hmm, what caused this issue? and how to avoid it?
,
Sep 15 2017
> hmm, what caused this issue? and how to avoid it?
Basically, something in stateful got too big. A few points of note:
* This is jetstream, which has a much smaller stateful than a
typical chromebook.
* The stuff that takes up space in the results directory can be
seen by looking at "result_summary.html" in the provision logs.
Some of that stuff likely contributed to the overflow.
* Stateful includes a lot of software packages specific to test
images. If someone's recently added new packages to test images
(or if package upgrades made stuff bigger) that will cause
trouble.
,
Sep 18 2017
,
Sep 19 2017
,
Sep 20 2017
Happened on gale-paladin. OP was about a different board, but that was a while ago. Repurposing this bug to be about gale/jetstream related siblings. https://luci-milo.appspot.com/buildbot/chromeos/gale-paladin/3922
,
Sep 20 2017
|
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by jen...@chromium.org
, May 4 2017