pprabhu:

What's up with Kahlee paladin? It's gotten the slowest by far.

akeshet:

Yes, median BuildPackages time for it has grown steadily http://shortn/_UODNEPpWFa especially in the tail.

Looking at a recent example https://luci-milo.appspot.com/buildbot/chromeos/kahlee-paladin/1465# I see:
1) amd-firmware was retried, but that doesn't really explain things. Filed https://bugs.chromium.org/p/chromium/issues/detail?id=816604 anyway
2) even though there's only 1 relevant CL, I see a large number of packages that are being build from ebuild (ie being rebuilt) rather than from binary package and BuildPackages stage is pushing 50 minutes. Is something wrong with our binary package publishing?

akeshet:

Digging more into  https://luci-milo.appspot.com/buildbot/chromeos/kahlee-paladin/1465

In it's SetupBoard stage https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fkahlee-paladin%2F1465%2F%2B%2Frecipes%2Fsteps%2FSetupBoard%2F0%2Fstdout I see which binhost is is trying to use...

08:29:06: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10436.0.0-rc2/packages/Packages /tmp/tmpJhvvEk.tmp
08:29:08: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/kahlee/paladin-R66-10436.0.0-rc2/packages/Packages /tmp/tmpMV2lV5.tmp
08:29:09: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/caroline/chrome-R66-10436.0.0-rc1/packages/Packages /tmp/tmpi25pAh.tmp

That's this amd64-generic and this kahlee binhost package listing being copied temp temp files on disk (not clear to me where these temp files get read from).

Picking an arbitrary package from BuildPackages that was build from source 

[ebuild  N     ] media-libs/fontconfig-2.11.1-r22:1.0::chromiumos to /build/kahlee/ USE="subpixel_rendering touchview -cros_host -doc -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB

I see:
It is present in the amd64 package listing but there's no tarball for it in  the binhost tree. Are we failing to upload needed binary packages from amd64-generic paladin perhaps?
It is present also in the kahlee package listing AND its tarball is present in the binhost tree. But perhaps this is being ignored because we are first searching in amd64 generic binhost and failing to use prebuilt if we are missing a tarball?