https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/1080279 might be the cause of this.  Build compare shows this change as the only one being in common from the three failed builds: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildCompare?lastN=20&masterBuilder=master-paladin&botConfig=falco-full-compile-paladin&buildBotMaster=chromeos#highlightKeyValues=stage_status%3ABuildPackages%E2%86%92fail%3Bstatus%E2%86%92fail&operator=and

I think this is related to the revdeps work. 718 packages is a lot. Are we exhausting the argument length limit at ${NEW_DEPS[*]} ? Maybe we just went over that limit due to the package count increasing?

This is going to take out the active CQ run: https://luci-milo.appspot.com/buildbot/chromeos/nyan-full-compile-paladin/12809

Ugh, not the best time for this to hit ... leaving for Europe for 3w.

We can easily disable the extra revdeps using a flag, but since we have already removed the same logic inside of the CROS_WORKON path in the build_packages shell script it isn't a great idea to do that unless we can just disable the flag in these full paladin builds.

Maybe the right thing to do here would be to temporarily mark these as experimental. Then, introduce a change that skips this logic if it is a full compile. The intent of this change was to get build correctness when doing incremental builds; that should be a noop on full compile.

Marked experimental.

I don't really understand this feature.  But if there's a bad CL which has a lot of dependencies, will it possibly cause build failures for developers if they're doing a full build?

I don't think so: I suspect this line of code only gets a lot of arguments when there's already a large list of modified packages in base Portage calculation. That should only happen on from-source/full builds. So, I think we can pivot on from-source.

falco-full-compile history: https://luci-milo.appspot.com/buildbot/chromeos/falco-full-compile-paladin/
nyan-full-compile history: https://luci-milo.appspot.com/buildbot/chromeos/nyan-full-compile-paladin/

So, it looks like it's not happening now and maybe David's idea in Comment 1 is actually the root cause. If so, maybe we are passing the argument to Portage for that package upgrade in a way that breaks the command invocation.

Interestingly, on that last CQ run where those builders were made experimental, falco didn't fail, only nyan.

The change I mentioned wouldn't have made it in if it wasn't changed mid-CQ run.  It's in pre-CQ now and we'll see what happens.  If it gets in and all the release builds start breaking, that should prove things one way or the other.

From the equery man page: "Note: Additionally, when using --quiet, if no matches are found, instead of printing an error, the equery list module will return with an exit value of 3." That would seem to indicate an argument is being passed that doesn't exist.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/9e658a9550baf51993f54d96f37473866c58b17f

commit 9e658a9550baf51993f54d96f37473866c58b17f
Author: Jason D. Clinton <jclinton@chromium.org>
Date: Wed Jun 06 04:28:38 2018

build_packages: Add log message to help debug  crbug.com/849414 

TEST=none
BUG= chromium:849414 

Change-Id: I6a244aacfdf6732c8964550434fcc9118c303643
Reviewed-on: https://chromium-review.googlesource.com/1087829
Commit-Ready: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Lann Martin <lannm@chromium.org>

[modify] https://crrev.com/9e658a9550baf51993f54d96f37473866c58b17f/build_packages

falco and nyan full compile builders have been green for several days and the reverse dependency fixes have been submitted.