+dgarrett to take a look

+don isn't this a build problem? Not a dev/code problem.

I looked into it but I'm not really expert on how the GE works. The source version for payload (e.g. 11021.56.0) comes from the GE config json which is I think gs://chromeos-build-release-console/paygen.json.  There is no other way to get these numbers in paygen. So the only theory I have is that the artifacts and paygen started building before that json file is pushed!!!?? Or somehow gsbucket cached the older json config version?!! I don't really know.

#4 sounds like the most likely scenario.

Is it feasible to use payloads tryjobs to recover?

So what I think happens is these channel level delta payloads are based on the current serving channel version at the time the build happens, so any time we take a build that was built before the same channel had an update pushed, it will have an out of sync set of delta payloads. In other words if we try to do two releases to a channel back to back we can run into this issue.

https://cros-goldeneye.corp.google.com/chromeos/console/pushHistoryRelease?releaseName=M70-BETA-CHROMEOS-7 was mostly pushed by 2018-10-24 14:08

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=3067252 built started at 2018-10-24 00:12

Since the push for the channel happened after the build, the build had outdated payloads. 

I think a set of payload tryjobs might regenerate them, but OTOH for beta not having delta payloads is not a critical optimization to have IMO.

I think so. But I'm guessing with <board>-payloads and not with <board>payloads-tryjob??!! The json file has been updated so I should just regenerate newer deltas and save them to the same bucket.

Yes, this is the expected behavior.  The build will generate N-1 payloads based on what is currently N-1, if it changes after the build, then you can generate new payloads via the tryjob commands (Don may know what the best commands to run are for this).

I believe it's now:

cros tryjob --production --version XXX --channel YYY <board>-payloads

That needs to be done for every affected board.

Having red is confusing.  if this is expected behavior, we should fix the way it's build (regenerate the load) or if delta payload is not important for beta, we should remove the test for delta.

I'm not certain why this shows red in GE.

The build generated what it was told to, and reported everything it built as passing. However, there I'm not sure how results are reported from the build to GE, and that path is likely not straight forward.

The Payload tests ARE critically important. Those tests are the last minute sanity check that a ChromeOS device can successfully update away from the release being shipped out.

However, the delta from previous release is nice-to-have (ie: not critical) for beta.

Delta payloads are missing for https://cros-goldeneye.corp.google.com/chromeos/console/qaRelease?releaseName=M71-BETA-CHROMEOS-2.
As a result, test for delta are failing.