autotest: out of space when running profiler |
|||||||
Issue descriptiontelemetry_AFDOGenerate failed on chell-chrome-pfq because of this. https://uberchromegw.corp.google.com/i/chromeos/builders/chell-chrome-pfq/builds/1133 telemetry_AFDOGenerate ABORT: Autotest client terminated unexpectedly: DUT is pingable, SSHable and did NOT restart un-expectedly. We probably lost connectivity during the test., client.bin.job.__init__ failed: [Errno 28] No space left on device
,
Jan 2 2018
[It appears that a bunch of old cros issues bulk-added the "Infra" component recently, but they should probably be "Infra>Client>ChromeOS".]
,
Jan 3 2018
,
Jan 4 2018
,
Feb 5 2018
,
Nov 12
I got similar issue running telemetry_AFDOGenerate, the script was trying to copy /build/${BOARD}/usr/local/build/autotest/client/* into /tmp/profiler directory on DUT.
The autotest/client directory is around 3.0GB, while /tmp only reserved around 2.0GB space. And it leads to this error:
rsync: write failed on "/tmp/profilers/autoserv-ulJobH/deps/telemetry_dep/test_src/media/test/data/tulip2.webm": No space left on device (28)
I don't totally understand why it is going to copy such a big directory onto DUT, but I think we need to figure out a way to avoid it.
,
Nov 12
(FWIW, the workaround we came up with was to bind mount the stateful partition to /tmp/profilers. It 'worked', but we ran into some later seemingly-unrelated issues anyway. Example commands for the workaround on the crbook: $ mkdir /mnt/stateful_partition/whee $ mount --bind !$ /tmp/profilers ...We should still probably fix the 3GB of copying, though :) )
,
Nov 13
I have a very simple fix for this in my local chroot (if anyone is interested): In src/third_party/autotest/files/server/profilers.py, change: PROFILER_TMPDIR = '/tmp/profilers' to PROFILER_TMPDIR = '/usr/local/profilers' There is enough room in /usr/local that this is not a problem.
,
Nov 26
I while back, I tried to push the change the Caroline suggests into autotest and had to revert it because I broke autotest badly. But, if we don't use the profilers functionality in autotest, we dont need this fix. Why is zhizhouy finding this?
,
Nov 26
I discussed with Caroline offline later on this. And you are right we are going to replace this with perf running on DUT directly. And this will not be an issue after the change. I caught this bug just because I wanted to run the original profiler to see its behavior. So I will assign this issue to me and link related patch to it. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by zalcorn@chromium.org
, Dec 26 2017