Haven't had any luck reproducing, going through logs has not revealed anything. Next step is probably going to be either getting a local repro or sshing to one of the bots and applying reverts until the build succeeds.

example failures:
 - https://luci-milo.appspot.com/buildbot/chromeos/reef-chrome-pfq/780
 - https://luci-milo.appspot.com/buildbot/chromeos/chell-chrome-pfq/849
 - https://luci-milo.appspot.com/buildbot/chromeos/caroline-chrome-pfq/295

Potential blamelist:

cros_workon:
 - https://chromium-review.googlesource.com/c/602629
 - https://chromium-review.googlesource.com/c/602254
 - https://chromium-review.googlesource.com/c/602627
 - https://chromium-review.googlesource.com/c/602530

cbuildbot:
 - https://chromium-review.googlesource.com/c/602514

tooling:
 - https://chromium-review.googlesource.com/c/602683

(possible) data add:
 - https://chromium-review.googlesource.com/c/597533
 - https://chromium-review.googlesource.com/c/601056

vapier@, ihf@, any ideas here? Both yusukes@ and I have been unable to repro locally.

I'm planning on submitting a clobber PFQ run later today to see what happens.

Image sizes seem to have been going (slightly) down over time per https://chromeperf.appspot.com/report?sid=d2c8d8b3c8780bb331d2aaff94189c36d7a2dceb373903b03424f9982282032f.

looking at the logs in comment #0 & comment #1, the pfq's are all failing while merging telemetry to /usr/local.  don't confuse that with the rootfs (which is referenced in comment #4).  /usr/local is the stateful partition and independent of the rootfs.

telemetry has a history of being super fat.  maybe it just got much fatter?

at any rate, i threw together this CL:
  https://chromium-review.googlesource.com/611724

unless i'm forgetting something obvious, should fix these problems.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/8558cbe1b9ae90d9ebc910253b10f757ffeff9c3

commit 8558cbe1b9ae90d9ebc910253b10f757ffeff9c3
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Aug 14 21:57:26 2017

disk layout: raise default state filesystem size to 2 GiB

We've been (close to) filling the stateful for a while now due to all the
test data we throw into it, so make it bigger.  This shouldn't impact AU
at all as we only process the rootfs & kernel, and we're only increasing
the filesystem size, not the partition size.

BUG= chromium:753498 
TEST=precq passes

Change-Id: Ic3fa9d63607172c196d4bce76e8a949e0729f044
Reviewed-on: https://chromium-review.googlesource.com/611724
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jacob Dufault <jdufault@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/8558cbe1b9ae90d9ebc910253b10f757ffeff9c3/build_library/legacy_disk_layout.json

assigning to current gardener to verify actually fixes things :)

Chrome has been rolled to 62.0.3185.0 at R62-9842.0.0!

yep!