New issue
Advanced search Search tips

Issue 774247 link

Starred by 1 user

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

autotest: out of space when running profiler

Project Member Reported by laszio@chromium.org, Oct 12 2017

Issue description

telemetry_AFDOGenerate failed on chell-chrome-pfq because of this.

https://uberchromegw.corp.google.com/i/chromeos/builders/chell-chrome-pfq/builds/1133

  telemetry_AFDOGenerate     ABORT: Autotest client terminated unexpectedly: DUT is pingable, SSHable and did NOT restart un-expectedly. We probably lost connectivity during the test., client.bin.job.__init__ failed: [Errno 28] No space left on device

 
Components: Infra

Comment 2 by mmoss@chromium.org, Jan 2 2018

Components: -Infra Infra>Client>ChromeOS
[It appears that a bunch of old cros issues bulk-added the "Infra" component recently, but they should probably be "Infra>Client>ChromeOS".]
Status: Archived (was: Untriaged)
Components: -Infra>Client>ChromeOS
Status: Available (was: Archived)
Components: Tools>ChromeOS-Toolchain
Cc: zhizhouy@chromium.org g...@chromium.org
Owner: llozano@chromium.org
Status: Assigned (was: Available)
I got similar issue running telemetry_AFDOGenerate, the script was trying to copy /build/${BOARD}/usr/local/build/autotest/client/* into /tmp/profiler directory on DUT.

The autotest/client directory is around 3.0GB, while /tmp only reserved around 2.0GB space. And it leads to this error:

rsync: write failed on "/tmp/profilers/autoserv-ulJobH/deps/telemetry_dep/test_src/media/test/data/tulip2.webm": No space left on device (28)


I don't totally understand why it is going to copy such a big directory onto DUT, but I think we need to figure out a way to avoid it.
(FWIW, the workaround we came up with was to bind mount the stateful partition to /tmp/profilers. It 'worked', but we ran into some later seemingly-unrelated issues anyway.

Example commands for the workaround on the crbook:
$ mkdir /mnt/stateful_partition/whee
$ mount --bind !$ /tmp/profilers

...We should still probably fix the 3GB of copying, though :) )
I have a very simple fix for this in my local chroot (if anyone is interested):  In src/third_party/autotest/files/server/profilers.py, change:

PROFILER_TMPDIR = '/tmp/profilers'

to 

PROFILER_TMPDIR = '/usr/local/profilers'

There is enough room in /usr/local that this is not a problem.

I while back, I tried to push the change the Caroline suggests into autotest and had to revert it because I broke autotest badly.

But, if we don't use the profilers functionality in autotest, we dont need this fix. Why is zhizhouy finding this? 
Cc: llozano@chromium.org cmt...@chromium.org
Owner: zhizhouy@chromium.org
Status: Started (was: Assigned)
I discussed with Caroline offline later on this.

And you are right we are going to replace this with perf running on DUT directly. And this will not be an issue after the change.

I caught this bug just because I wanted to run the original profiler to see its behavior.

So I will assign this issue to me and link related patch to it.

Sign in to add a comment