Actually this is bad.  It filled up around Feb 5, and has probably failed the health check since then, so it's not been used.

On the other hand, it's interesting to find out that the other devservers were enough...

We should add alerting from this.

I suspect this is from switching over to use the SSD for the images and perhaps there's some issue with detecting when the FS is full and things need to be cleaned?

There is a cron job on the devserver to clean staged images every 30 mins. I manually ran the job, and it failed:

chromeos-test@chromeos4-devserver7:~$ /home/chromeos-test/chromiumos/src/third_party/autotest/files/site_utils/admin/clean_staged_images.py --max-age 16 --max-paladin-age 16 /home/chromeos-test/images
-bash: /home/chromeos-test/chromiumos/src/third_party/autotest/files/site_utils/admin/clean_staged_images.py: No such file or directory

Then I manually searched for the file, but not found it.

Allen, any puppet change recently?

chromeos4-devserver7 doesn't run repo sync when crbug.com/687237 is fixed.

Now it has, I'm running clean_staged_images.py . We should check all devserver and force them them to run repo sync to avoid such cases.

Sorry #5, could you clarify this sentence:

"chromeos4-devserver7 doesn't run repo sync when crbug.com/687237 is fixed."

Do you mean "will not run repo sync until issue 687237 is fixed"?  (I took a quick look at the bug but it was too much work to understand what was going on.)

Thanks!

Re #6,

Devserver itself always contain the autotest repo. One day we remove it, but find devserver still need to use this 'clean_staged_images' to clean images. Then we add it back. It's the whole story in crbug.com/687237.

But after we add it back in xml, we need to run repo sync to let devserver sync the autotest repo back. In crbug.com/687237 deputy (Aviv) said he already kicked off repo sync to all devservers. But for some unknown reasons, chromeos4-devserver7 missed this step, and it still doesn't contain autotest package. So it cannot run its normal clean script :( 

#7 Thanks!  So what you meant in #5 was:

"chromeos-devserver7 didn't run repo sync after issue 687237 was fixed"

or even better:

"after issue 687237 was fixed, we updated the code on the devservers (with repo sync) but somehow that operation failed on chromeos4-devserver7"

(Also note that if you use "issue ###" or "bug ###" instead of crbug.com/###, you get clickable links which also show the open/closed status of the bug.)

Sorry, I don't mean to be a nitpick, I was really not understanding.

oh, skill learned! thanks :)