Marking grunt paladin experimental.

Failure is:
chromeos-firmware-grunt-0.0.1-r53:  * Build grunt firmware updater: -c /build/grunt//usr/share/chromeos-config/yaml/config.yaml -i /build/grunt/tmp/portage/chromeos-base/chromeos-firmware-grunt-0.0.1-r53/distdir --tool_base /firmware/utils:/usr/sbin:/usr/bin --root /build/grunt --script updater4.sh
chromeos-firmware-grunt-0.0.1-r53: Traceback (most recent call last):
chromeos-firmware-grunt-0.0.1-r53:   File "./pack_firmware.py", line 1190, in <module>
chromeos-firmware-grunt-0.0.1-r53:     main(sys.argv)
chromeos-firmware-grunt-0.0.1-r53:   File "./pack_firmware.py", line 1187, in main
chromeos-firmware-grunt-0.0.1-r53:     packer.Start(argv[1:])
chromeos-firmware-grunt-0.0.1-r53:   File "./pack_firmware.py", line 1149, in Start
chromeos-firmware-grunt-0.0.1-r53:     firmware_info[model], args, tool_base, model_details)
chromeos-firmware-grunt-0.0.1-r53:   File "./pack_firmware.py", line 1018, in _GenerateOneModel
chromeos-firmware-grunt-0.0.1-r53:     dirname)
chromeos-firmware-grunt-0.0.1-r53:   File "./pack_firmware.py", line 974, in _ExtractFile
chromeos-firmware-grunt-0.0.1-r53:     os.path.join(self._args.imagedir, fname), dirname, suffix)
chromeos-firmware-grunt-0.0.1-r53:   File "./pack_firmware.py", line 655, in _UntarFile
chromeos-firmware-grunt-0.0.1-r53:     with tarfile.open(pathname) as tar:
chromeos-firmware-grunt-0.0.1-r53:   File "/usr/lib64/python2.7/tarfile.py", line 1667, in open
chromeos-firmware-grunt-0.0.1-r53:     return func(name, "r", fileobj, **kwargs)
chromeos-firmware-grunt-0.0.1-r53:   File "/usr/lib64/python2.7/tarfile.py", line 1732, in gzopen
chromeos-firmware-grunt-0.0.1-r53:     fileobj = gzip.GzipFile(name, mode, compresslevel, fileobj)
chromeos-firmware-grunt-0.0.1-r53:   File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
chromeos-firmware-grunt-0.0.1-r53:     fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
chromeos-firmware-grunt-0.0.1-r53: IOError: [Errno 2] No such file or directory: '/build/grunt/tmp/portage/chromeos-base/chromeos-firmware-grunt-0.0.1-r53/distdir/Careena.10795.0.0.tbz2'
chromeos-firmware-grunt-0.0.1-r53:  * ERROR: chromeos-base/chromeos-firmware-grunt-0.0.1-r53::grunt-private failed (compile phase):
chromeos-firmware-grunt-0.0.1-r53:  *   Cannot pack firmware.


https://crrev.com/i/647551 updated the grunt model.yaml file to bump firmware version from 10795.0.0 to 10833.0.0.  
This patch passed its grunt-paladin CQ run [1], during which, locally, chromeos-config was rebuilt thus updating the local config.yaml.
This updated config.yaml file was then processed by the revbumped chromeos-firmware-grunt-0.0.1-r53, which successfully installed the updated firmware, 10833.0.0.
[1] https://uberchromegw.corp.google.com/i/chromeos/builders/grunt-paladin/builds/2250

However, grunt-paladin build #2250 was part of master-paladin build #19018 [2], which itself failed.  Since CL:*647551 was grunt specific, and grunt-paladin passed, the patch was successfully landed and the chromeos-firmware-grunt ebuild was successfully upreved to -r53...  [speculation:] However, since master-paladin failed, the resulting chromeos-config output was *NOT* uploaded as the new prebuilt for chromeos-base/chromeos-config-0.0.1-r21.
[2] https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/19018

For the next grunt-paladin build (#2251) [3], the "CleanUp" stage [4] noticed the previous master-paladin (#19018) had failed, and it reverted to a previous good chroot (presumably from master-paladin #19017).  This chroot included the old config.yaml file, which still referenced the old firmware version (10795.0.0).  However, for some unknown (by me) reason [*], chromeos-config was added to the "final reverse dependency" list, and hence was rebuilt again, which installed the new config.yaml (again), and the grunt-paladin build passed.  Unfortunately, the corresponding master-paladin build (#19019) failed again [5].
[3] https://uberchromegw.corp.google.com/i/chromeos/builders/grunt-paladin/builds/2251
[4] https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fgrunt-paladin%2F2251%2F%2B%2Frecipes%2Fsteps%2FCleanUp%2F0%2Fstdout
[5] https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/19019


For the next grunt-paladin build (#2252) [6], the "CleanUp" stage [7] again noticed the previous master-paladin (#19019) had failed, and it reverted to a previous good chroot (presumably from master-paladin #19017).  However, this time chromeos-config was NOT added to the "final reverse dependency" list, the restored chroot still had the old config.yaml with firmware version 10795.0.0, and thus the build failed with the failure message above.
[6] https://uberchromegw.corp.google.com/i/chromeos/builders/grunt-paladin/builds/2252
[7[ https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fgrunt-paladin%2F2252%2F%2B%2Frecipes%2Fsteps%2FCleanUp%2F0%2Fstdout

[*] The difference in the "reverse dependency" calculation is probably due to this:
#2251: INFO    : calculating reverse dependencies on packages: chromeos-base/ec-devutils chromeos-base/vboot_reference chromeos-base/tast-cmd chromeos-base/tast-remote-tests-cros sys-apps/coreboot-utils chromeos-base/ec-utils
#2252  INFO    : calculating reverse dependencies on packages: chromeos-base/ec-devutils sys-apps/coreboot-utils chromeos-base/ec-utils

A workaround for this should be to manually uprev chromeos-config [8].  A longer term solution may be to change chromeos-config to be a cros_workon ebuilt that uprevs automatically when any of its "source" .yaml files change.

[8] https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1123016

Re: [*] Indeed, the "reverse dependency" line comes from [0]:

[0] https://cs.corp.google.com/chromeos_public/src/scripts/build_packages?type=cs&q=%22calculating+reverse+dependencies+on+packages%22&g=0&l=326

equery-${FLAGS_board} -q depends --indirect

And the magic ebuild with a chromeos-config reverse dependency is...
chromeos-base/vboot_reference

equery-grunt -q depends --indirect chromeos-base/vboot_reference  | grep chromeos-config

        chromeos-base/chromeos-config-bsp-grunt-0.0.1-r3
         virtual/chromeos-config-bsp-3-r2
          chromeos-base/chromeos-config-0.0.1-r21
        chromeos-base/chromeos-config-bsp-grunt-private-9999
         virtual/chromeos-config-bsp-3-r2
        chromeos-base/chromeos-config-tools-0.0.2-r1071
         chromeos-base/chromeos-config-tools-0.0.2-r1071
         chromeos-base/chromeos-config-tools-0.0.2-r1071
              chromeos-base/chromeos-config-tools-0.0.2-r1071
              chromeos-base/chromeos-config-tools-0.0.2-r1071
              chromeos-base/chromeos-config-tools-0.0.2-r1071

From the logs, it looks like the revdeps logic didn't account for the revdeps of chromeos-config-bsp-private even though it was being upgraded. I think this is a bug in the scan of affected packages for the revdeps calculation.

The code [0] seems to indicate that only packages in the "save_install_plan" file are used for the "reverse dependency" lookup ... 

[0] https://cs.corp.google.com/chromeos_public/src/scripts/build_packages?type=cs&q=%22calculating+reverse+dependencies+on+packages%22&g=0&l=311

MOD_PKGS=()
if [[ -n "${FLAGS_save_install_plan}" ]]; then
  [ -f ${FLAGS_save_install_plan} ] \
    || die "${FLAGS_save_install_plan} does not exist"
  readarray NEW_DEPS < ${FLAGS_save_install_plan}
  if [[ ${#NEW_DEPS[@]} -gt 0 ]]; then
    echo "Found new deps: '${NEW_DEPS[*]}'. Converting deps to package names."
    # Convert specific versions into base package names
    MOD_PKGS+=( $(\
      equery-${FLAGS_board} list -p -o --format='$category/$name' \
          ${NEW_DEPS[*]}) )
  fi
  FORCE_LOCAL_BUILD_PKGS+=( ${MOD_PKGS[@]} )
fi


but in the logs,this is just a file name, so I don't know how this file is built or what it contains.  e.g., from https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fgrunt-paladin%2F2251%2F%2B%2Frecipes%2Fsteps%2FBuildPackages%2F0%2Fstdout:

...
00:59:13: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpR9exdY' 'USE=chrome_internal' 'FEATURES=separatedebug' -- ./build_packages '--board=grunt' '--accept_licenses=@CHROMEOS' --withdebugsymbols --skip_chroot_upgrade --withevents '--eventfile=/mnt/host/source/buildbot_archive/grunt-paladin/R69-10835.0.0-rc3/build-events.json' '--save_install_plan=/tmp/grunt_install_plan.2714277' virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all in /b/c/cbuild/repository
00:59:13: NOTICE: /b/c/cbuild/repository/chroot.img is using 64 GiB more than needed.  Running fstrim.
Chromium OS version information:
    CHROME_BASE=
    CHROME_BRANCH=69
    CHROME_VERSION=
    CHROMEOS_BUILD=10835
    CHROMEOS_BRANCH=0
    CHROMEOS_PATCH=2018_07_02_0059
    CHROMEOS_VERSION_STRING=10835.0.2018_07_02_0059
INFO    : saving install plan to /tmp/grunt_install_plan.2714277

Greg, where are we at on this?

Working on it

Quick update: just sat down with gmeinke@ to look at this; it *appears* that the reverse dependency calculation is not working correctly, which prevents chromeos-config changes from being picked up in incremental builds. He is working on that issue. 

looks like the issue is that the flag to capture the non-tool dependencies was optimized out of the call to capture those dependencies ... will have a fix shortly

still testing ... latest patch should solve issue with not excluding chrome from the local build.

latest patch verified and sending to CQ

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/ee71dd15ad02590827674f0e55701d78aa290dd8

commit ee71dd15ad02590827674f0e55701d78aa290dd8
Author: Gregory Meinke <gmeinke@chromium.org>
Date: Wed Jul 11 19:13:10 2018

Pass missing install plan flag to parallel_emerge

BUG= chromium:859556 
TEST=tryjob and local builds

Change-Id: I6c725cf7019731e111045b8484a2b07289d97495
Reviewed-on: https://chromium-review.googlesource.com/1129556
Commit-Ready: Gregory Meinke <gmeinke@chromium.org>
Tested-by: Gregory Meinke <gmeinke@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/ee71dd15ad02590827674f0e55701d78aa290dd8/build_packages

reverse dependencies are now correctly being calculated.