The fix is being tested here:

http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8949420733278071712

But we need to record the series of related events, and figure out how things broke.

The change that introduced the problem (or at least the one whose revert fixed it) was crrev.com/i/604790.

Can we explain why the Android PFQ passed and performed an uprev after the initial change landed?

I was curious about the same thing that Don is asking about in c#5.

A good question, which needs followup.

-> jclinton@

Need more data. How can https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1010518 be related to the SHA256 of aapt changing?
 
The Comment #1 fix being tested doesn't really tell me anything either.

That was the point, essentially. A change in Autotest should not fail in BuildPackages.  The relevant CL was https://chrome-internal-review.googlesource.com/c/chromeos/overlays/project-cheets-private/+/604790. Reverting that CL fixed the problem. So, how did that CL (604790) make it in the first place? Aviv seems to think you should own figuring this out.

I don't understand this section of OS platform. What's going on here? How can these two things possibly be related? You made the change to the software; do you have any guesses?

From offline conversation: the initial bug report was simply that the PreCQ was broken, not that the CL mentioned was the cause of that breakage. It seems that Don knew at the time what the root cause was (Comment #1). Over to him to see if he knows of another bug that tracked that root cause or more information. Assign back to me once info. is added. Thank you.

From #4, this is the CL that caused the break. https://crrev.com/i/604790

I don't know if the script change or the ebuild change caused the problem (no idea what the actual problem was), just got dragged into reverting a .9999 ebuild revert.

.9999 ebuilds are really just templates that are unused until the next time the relevant ebuild is uprevved. Since the ebuild in question is only uprevved by the Android PFQ, it couldn't have caused a problem until the Android PFQ passed.

So.... how did the Android PFQ pass when the result broke everything else?

Or was the script change the problem?

Or is my analysis wrong somehow?

I think those are the right questions, especially "how did the Android PFQ pass when the result broke everything else?".

This ebuild specifically skips SRC_URI in the CQ if the 9999.ebuild is being modified. https://chrome-internal-review.googlesource.com/c/chromeos/overlays/project-cheets-private/+/604790/2/chromeos-base/android-container-nyc/android-container-nyc-9999.ebuild#30 . Later, when the uprev happens, we are no longer hitting the skip logic.

Over to the team who owns this ebuild and the CL in question to fix this. This ebuild should not have a 9999 PV short-circuit of all testing in the CQ. Just delete that entire if statement; I can't think of any reason it should be there.

>> So.... how did the Android PFQ pass when the result broke everything else?

Android PFQ had non-clean $DISTDIR and did not fetch fresh tools from new Android build (aapt). That is why Manifest was not updated. However during build_packages phase build system fetched new tool and failed due Manifest pointed to old file.

I traced how ebuild was updating the manifest and this is portage behavior. From one point it uses $DISDIR during the Manifest update. From other side it tries to download new binaries during build_packages phase.

We have alternative solution to use build build suffix for such tools. That should avoid this problem.

I think problem with CQ was resolved last week. I also explained why Android PFQ passed but build failed.