I found that in licensing.LoadPackageInfo of a package building process, gmerge is called. When it's removed from devserver, I get errors:

gmerge-0.0.1-r1032: >>> Compiling source in /build/whirlwind/tmp/portage/chromeos-base/gmerge-0.0.1-r1032/work/gmerge-0.0.1 ...
gmerge-0.0.1-r1032: make -j16 
gmerge-0.0.1-r1032: Preparing devserver modules.
gmerge-0.0.1-r1032: >>> Source compiled.
gmerge-0.0.1-r1032: >>> Test phase [not enabled]: chromeos-base/gmerge-0.0.1-r1032
gmerge-0.0.1-r1032: 
gmerge-0.0.1-r1032: >>> Install gmerge-0.0.1-r1032 into /build/whirlwind/tmp/portage/chromeos-base/gmerge-0.0.1-r1032/image/ category chromeos-base
gmerge-0.0.1-r1032: !!! dobin: gmerge does not exist

Is it expected? Does it mean we can't remove gmerge support from devserver?

I prepared these CLs to remove the gmerge package, thinking it wasn't used any more.

https://chromium-review.googlesource.com/#/c/439435/
https://chromium-review.googlesource.com/#/c/438842/

However, that breaks "cros deploy" to a DUT which doesn't have gmerge installed. I don't understand why, and probably won't dig any further, for now. I'm abandoning these CLs, but added the links above so that they can be found if we dig into this again.

Example cros deploy error:

cros deploy 100.107.151.225 update_engine

 * Generating locale-archive: forcing # of jobs to 1
13:16:26: NOTICE: Cleaning outdated binary packages from /build/lumpy
13:16:28: NOTICE: These are the packages to emerge:
13:16:28: NOTICE: * 1) chromeos-base/update_engine-0.0.3-r2273
 [#################################-----------------------------------------------------------------------------------] 28%Exception during output capturing: RunCommandError('cmd=[\'ssh\', \'-p\', \'22\', \'-oConnectionAttempts=4\', \'-oUserKnownHostsFile=/dev/null\', \'-oProtocol=2\', \'-oConnectTimeout=30\', \'-oServerAliveCountMax=3\', \'-oStrictHostKeyChecking=no\', \'-oServerAliveInterval=10\', \'-oNumberOfPasswordPrompts=0\', \'-oIdentitiesOnly=yes\', \'-i\', \'/tmp/ssh-tmpZGkypw/testing_rsa\', \'root@100.107.151.225\', \'--\', \'FEATURES=-sandbox\', \'PORTAGE_CONFIGROOT=/usr/local\', "CONFIG_PROTECT=\'-*\'", \'PKGDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages\', \'PORTDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD\', \'PORTAGE_TMPDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp\', \'emerge\', \'--usepkg\', u\'/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base/update_engine-0.0.3-r2273.tbz2\', \'--root=/\'], extra env={\'LC_MESSAGES\': \'C\'}', <chromite.lib.cros_build_lib.CommandResult object at 0x7f8e4035ecd0>, None)
Captured stdout was:
sending incremental file list
update_engine-0.0.3-r2273.tbz2
sent 428,970 bytes  received 35 bytes  286,003.33 bytes/sec
total size is 434,251  speedup is 1.01
Captured stderr was:
13:16:29: INFO: RunCommand: strip_package --sysroot /build/lumpy chromeos-base/update_engine-0.0.3-r2273
13:16:30: DEBUG: RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- mkdir -p /usr/local/tmp/cros-deploy '&&' mktemp -d '--tmpdir=/usr/local/tmp/cros-deploy'
13:16:31: DEBUG: The temporary working directory on the device is /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD
13:16:31: DEBUG: RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- echo '${PATH}'
13:16:33: DEBUG: RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- rm -rf /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp '&&' mkdir -p /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp
13:16:34: NOTICE: Copying update_engine-0.0.3-r2273.tbz2 to device.
13:16:34: DEBUG: RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- 'PATH=/usr/local/bin:/usr/local/sbin:$PATH rsync' --version
Warning: Permanently added '100.107.151.225' (RSA) to the list of known hosts.
13:16:37: INFO: Use portage temp dir /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp
13:16:37: NOTICE: Installing update_engine-0.0.3-r2273.tbz2.
13:16:37: INFO: RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- 'FEATURES=-sandbox' 'PORTAGE_CONFIGROOT=/usr/local' "CONFIG_PROTECT='-*'" 'PKGDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages' 'PORTDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD' 'PORTAGE_TMPDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp' emerge --usepkg /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base/update_engine-0.0.3-r2273.tbz2 '--root=/'
Warning: Permanently added '100.107.151.225' (RSA) to the list of known hosts.

!!! /usr/local/etc/portage/make.profile is not a symlink and will probably prevent most merges.
!!! It should point into a profile within /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/profiles/
!!! (You can safely ignore this message when syncing. It's harmless.)

!!! Your current profile is invalid. If you have just changed your profile
!!! configuration, you should revert back to the previous configuration.
!!! Allowed actions are limited to --help, --info, --search, --sync, and
!!! --version.
13:16:39: ERROR: Failed to emerge package update_engine-0.0.3-r2273.tbz2

13:16:40: ERROR: Oops. Something went wrong.
13:16:40: ERROR: cros deploy failed before completing.
13:16:40: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- 'FEATURES=-sandbox' 'PORTAGE_CONFIGROOT=/usr/local' "CONFIG_PROTECT='-*'" 'PKGDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages' 'PORTDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD' 'PORTAGE_TMPDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp' emerge --usepkg /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base/update_engine-0.0.3-r2273.tbz2 '--root=/'
cmd=['ssh', '-p', '22', '-oConnectionAttempts=4', '-oUserKnownHostsFile=/dev/null', '-oProtocol=2', '-oConnectTimeout=30', '-oServerAliveCountMax=3', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=10', '-oNumberOfPasswordPrompts=0', '-oIdentitiesOnly=yes', '-i', '/tmp/ssh-tmpZGkypw/testing_rsa', 'root@100.107.151.225', '--', 'FEATURES=-sandbox', 'PORTAGE_CONFIGROOT=/usr/local', "CONFIG_PROTECT='-*'", 'PKGDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages', 'PORTDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD', 'PORTAGE_TMPDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp', 'emerge', '--usepkg', u'/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base/update_engine-0.0.3-r2273.tbz2', '--root=/'], extra env={'LC_MESSAGES': 'C'}
Traceback (most recent call last):
  File "/mnt/host/source/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/mnt/host/source/chromite/lib/operation.py", line 175, in CaptureOutputInBackground
    func(*args, **kwargs)
  File "/mnt/host/source/chromite/cli/deploy.py", line 890, in _EmergePackages
    _Emerge(device, pkg_path, root, extra_args=emerge_args)
  File "/mnt/host/source/chromite/cli/deploy.py", line 774, in _Emerge
    capture_output=False, debug_level=logging.INFO)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 1058, in RunCommand
    return super(ChromiumOSDevice, self).RunCommand(cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 902, in RunCommand
    return self.BaseRunCommand(cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 849, in BaseRunCommand
    return self.GetAgent().RemoteSh(cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 336, in RemoteSh
    return cros_build_lib.RunCommand(ssh_cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpZGkypw/testing_rsa root@100.107.151.225 -- 'FEATURES=-sandbox' 'PORTAGE_CONFIGROOT=/usr/local' "CONFIG_PROTECT='-*'" 'PKGDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages' 'PORTDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD' 'PORTAGE_TMPDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp' emerge --usepkg /usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base/update_engine-0.0.3-r2273.tbz2 '--root=/'
cmd=['ssh', '-p', '22', '-oConnectionAttempts=4', '-oUserKnownHostsFile=/dev/null', '-oProtocol=2', '-oConnectTimeout=30', '-oServerAliveCountMax=3', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=10', '-oNumberOfPasswordPrompts=0', '-oIdentitiesOnly=yes', '-i', '/tmp/ssh-tmpZGkypw/testing_rsa', 'root@100.107.151.225', '--', 'FEATURES=-sandbox', 'PORTAGE_CONFIGROOT=/usr/local', "CONFIG_PROTECT='-*'", 'PKGDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages', 'PORTDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD', 'PORTAGE_TMPDIR=/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/portage-tmp', 'emerge', '--usepkg', u'/usr/local/tmp/cros-deploy/tmp.bryqN0JeWD/packages/chromeos-base/update_engine-0.0.3-r2273.tbz2', '--root=/'], extra env={'LC_MESSAGES': 'C'}

The upshot is that I was wrong, at least some of the gmerge code is still used.

This is the CL I made for 'removing gmerge from devserver': https://chromium-review.googlesource.com/c/437634/

I will also abandon it at the end of this week and save it for next fixit.

Bulk closing Infra>Client>ChromeOS issues untouched in over a year.