https://cs.corp.google.com/chromeos_public/src/aosp/system/update_engine/payload_generator/delta_diff_generator.cc?q=%22shirking+the+filesystem+size%22&dr=C&l=107

From the build logs:

06:48:56: INFO: All payloads for the build:
06:48:56: INFO:   1: chromeos_9283.0.0_snappy_canary-channel_full_premp.bin-692bf4298ca6b716dc378901d3c9049a.signed
06:48:56: INFO:   2: chromeos_9176.0.0_snappy_canary-channel_full_premp.bin-a5bffcb170384bbe02fdbe8990c18ef6.signed (exists)
06:48:56: INFO:   3: chromeos_9176.0.0-9283.0.0_snappy_canary-channel_delta_premp.bin-8868fab67da2712ecf33741570d21714.signed
06:48:56: INFO:   4: chromeos_9283.0.0_snappy_canary-channel_full_test.bin-fd272abef14ffc6230bac680a1638c62
06:48:56: INFO:   5: chromeos_9176.0.0_snappy_canary-channel_full_test.bin-b2e5c515cd5b31e39ee7ed39fd40de09 (exists)
06:48:56: INFO:   6: chromeos_9283.0.0-9283.0.0_snappy_canary-channel_delta_test.bin-38f3799f94b1ec268c633f2e83039315
06:48:56: INFO:   7: chromeos_9176.0.0-9283.0.0_snappy_canary-channel_delta_test.bin-c81109ed9c18d51046cfec49ebcfa246

Assigning to this week's deputy.

We need to blacklist the older builds from paygen testing.
That would seem to be at minimum the 9176.0.0 build.
dgarrett@ should know how to find all builds needing to
be blacklisted.  He may also know of alternative solutions.

Why is 9176 being used? These boards are having problems bootstrapping, but can we just bump the attribute for what the base should be in the hopes of getting over the hurdle? Otherwise we're just at the whim of successive filesystem sizes. Presumably crbug.com/192136 should be fixed?

The error is happening because the rootfs filesystem configuration shrank in size, which is not a supported thing for delta payloads (you can still use full payloads).

9176.0.0 is being used because GE says it's the currently released version in the field.

Two solutions:
1) Grow the rootfs filesystem size back to whatever it was, and live with that.
2) Convince GE that there is no current release. Devices in the field will still update via full payloads.

I'm not sure how hard 2 is, the GE team can speak to that.

PS: Dev channel has the previous release as: 9202.12.0

Which appears to be fine in terms of FS size.

cc leecy@

Do we know where the rootfs filesystem changed?  I think the easiest thing to do is to do a manual canary push of whatever version we think makes sense (i.e. has the rootfs the size we want).  Presumably no canary channel delta payload was generated (can you confirm Don?) so we would essentially just be pushing the full payload if we do a manual push here.

Once that is live in the field, and assuming the newest one is is the same rootfs, this will correct itself since that will be the newest n-1 for generating deltas.

Could we just pick a very recent M58 canary to manually push reef/pyro/snappy to so we can clear this?

In practice, I'm almost certain that there are no actual users on Canary at this point since basically all systems are assigned to dogfooders on Dev (about to transition to Beta), and developers don't tend to sit on Canary.

Just confirmed in GE, that we only see the full payloads associated with the newer builds.  I think we can just push those boards as is manually.

Don, do you know how to make a manual push?

Bernie just pushed today's canary (9285.0.0) for those 3 boards, so this should be the new n-1 for the next set of canary builders.  

Thanks Bernie! Let's see how the next canary behaves. If we pass PaygenBuildCanary, i'll mark this as fixed.

I do see 9285.0.0 marked as live for these boards now, so assuming a delta can be produced from it, it should be OK.

Excellent.

Good news : https://uberchromegw.corp.google.com/i/chromeos/builders/reef-release/builds/849/steps/PaygenBuildCanary/logs/stdio

https://uberchromegw.corp.google.com/i/chromeos/builders/pyro-release/builds/395/steps/PaygenBuildCanary/logs/stdio

https://uberchromegw.corp.google.com/i/chromeos/builders/snappy-release/builds/380/steps/PaygenBuildCanary/logs/stdio

PaygenBuildCanary steps all passed today.