I originally blamed Hshi's Nyan related overlay changes, but they would not have affected all boards on the canaries.

The ghostscript ebuild was updated about 2 weeks ago. The lcms ebuild, over a year ago. Looking further into git history.

Strange indeed.

In the failing log (e.g. lumpy https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-release/builds/4336/steps/BuildPackages%20%5Bafdo_use%5D/logs/stdio), when "calculating deps", we see

line 133: [ebuild  N     ] media-libs/lcms-2.6-r1:2 to /build/lumpy/ USE="threads zlib -doc -jpeg -static-libs {-test} -tiff" ABI_X86="(64) -32 (-x32)" 0 KiB

line 620: [ebuild  N     ] app-text/ghostscript-gpl-9.19-r6::chromiumos to /build/lumpy/ USE="crosfonts cups internal -X -dbus -djvu -gtk -idn -static-libs -tiff" LINGUAS="-de -ja -ko -zh_CN -zh_TW" 0 KiB

line 3298-3300:
ghostscript-gpl-9.19-r6: checking for system lcms2 library... checking for _cmsCreateMutex in -llcms2... no
ghostscript-gpl-9.19-r6: configure: error: lcms2 not found, or too old
ghostscript-gpl-9.19-r6: configure: error: Recursive call to configure (for auxiliary tools) script failed

So how could it be not found or too old? The ghostscript ebuild says

COMMON_DEPEND="
...
	>=media-libs/lcms-2.6:2

I'm not an expert but it seems lcms-2.6-r1:2 would not be considered "too old"

This CL looks related, but not sufficient to cause the problem.

https://chromium-review.googlesource.com/#/c/391136/

When I try to emerge ghostscript-glp locally it seems to work ok on my workstation. 

checking for local png library source... no
checking for png_create_write_struct in -lpng... yes
checking png.h usability... yes
checking png.h presence... yes
checking for png.h... yes
checking for local lcms2 library source... no
checking for system lcms2 library... checking for _cmsCreateMutex in -llcms2... yes
checking lcms2.h usability... yes
checking lcms2.h presence... yes
checking for lcms2.h... yes

The version of lcms is 2.6 on sourceforge. We haven't bumped rev for a year and the library definitely has exported a function called "_cmsCreateMutex".

Also, the failing log shows lcms-2.6-r1 was in fact successfully emerged before building ghostscript-gpl.

I was able to reproduce locally with:

cros clean --clobber
./build_packages --board lumpy --nousepkg

I'm testing now to see if I can reproduce without --nousepkg

This builds without problems:

cros clean --clobber
./build_packages --board lumpy

What difference does the "--nousepkg" make?

My nyan overlay changes were suspected because they were the only chumped changes, but they only affected overlay-nyan-* and overlay-variant-nyan-*.

The only thing that could remotely affect the build systems as a whole was the update to chroot version hooks (https://chromium-review.googlesource.com/#/c/398520/) but even that should only affect nyan boards.

Could there be external toolchain or package updates?

--nousepkg for setup_board to rebuild compiler and headers (llvm, linux-headers, gcc-libs)

The command in ghostscript that fails is 
AC_CHECK_LIB(lcms2, _cmsCreateMutex, [AC_CHECK_HEADERS([lcms2.h],  ...

and fails when linking against lcms2 lib, looking for symbol _cmsCreateMutex().

Could not reproduce with
./setup_board --board cave --force --nousepkg, although it cleans /build/cave.
./build_packages --board cave --nousepkg

Re#11 can confirm, does not reproduce with lumpy if I clean (setup_board --force) first.

Maybe if we force regen configs on all failing boards it would go away?

Some CQ builders also started failing.


NOTE: The Commit Queue will retry your change automatically.

The following build(s) failed:

nyan-paladin: The BuildPackages stage failed: Packages failed in ./build_packages: app-text/ghostscript-gpl in https://uberchromegw.corp.google.com/i/chromeos/builders/nyan-paladin/builds/10870

nyan-full-compile-paladin: The BuildPackages stage failed: Packages failed in ./build_packages: app-text/ghostscript-gpl in https://uberchromegw.corp.google.com/i/chromeos/builders/nyan-full-compile-paladin/builds/7221

falco-full-compile-paladin: The BuildPackages stage failed: Packages failed in ./build_packages: app-text/ghostscript-gpl in https://uberchromegw.corp.google.com/i/chromeos/builders/falco-full-compile-paladin/builds/7224

Also seen on guado_moblab-paladin and reef-paladin

https://chromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/4018

https://chromegw.corp.google.com/i/chromeos/builders/reef-paladin/builds/408

Should we add --force --nousepkg to setup_board for all the builders ? That would possibly solve this problem on all the boards.

It seems that currently, there is no --force / --nousepkg when paladin builder runs setup_board.
e.g.
 
22:09:58: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpQIyTIP' 'USE=chrome_internal' 'FEATURES=separatedebug' -- ./setup_board '--board=reef' '--accept_licenses=@CHROMEOS' --skip_chroot_upgrade in /b/cbuild/internal_master
INFO    : Selecting profile: /mnt/host/source/src/private-overlays/overlay-reef-private/profiles/base for /build/reef

in https://chromegw.corp.google.com/i/chromeos/builders/reef-paladin/builds/408/steps/SetupBoard/logs/stdio

To rule out the possibility that CLs related to nyan are the cause, I will submit a tryjob.

I found the issue:
- you can repro with a clean cbuild, minilayout is fine,
- or if you remove lcms from the chroot with sudo emerge -C lcms:
indeed, ghostscript configure looks at lcms2 in /usr not /build/<board>/usr

Brian has chased cross compiling issue earlier (https://chromium-review.googlesource.com/352635), I think we need to go further.

Deleted: config.log 16.9 KB

config.log 16.9 KB View Download

The log with the issue is actually configaux.log

Deleted: configaux.log 50.0 KB

configaux.log 50.0 KB View Download

Actually ghostcript may be right to check the host: we fails in the config aux, which builds the host tools needed compile ghostscript on the target.

I uploaded a CL that fixes the problem by adding lcsm to virtual/target-sdk, but that's not a great solution: once ghostscript changes, we would need to revisit that CL (CL/399645)

cbuildbot at https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6799

It failed with different reason.

ghostscript-gpl-9.19-r6: In file included from ./openjpeg/src/lib/openjp2/j2k.c:43:0:
ghostscript-gpl-9.19-r6: ./openjpeg/src/lib/openjp2/opj_includes.h:112:6: warning: "__STDC_VERSION__" is not defined [-Wundef]
ghostscript-gpl-9.19-r6:  #if (__STDC_VERSION__ != 199901L)
ghostscript-gpl-9.19-r6:       ^
ghostscript-gpl-9.19-r6:  * ERROR: app-text/ghostscript-gpl-9.19-r6::chromiumos failed (compile phase):
ghostscript-gpl-9.19-r6:  *   emake failed

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6799/steps/BuildPackages%20%5Bafdo_use%5D/logs/stdio

scratch #20.
The error is :
ghostscript-gpl-9.19-r6: make: *** [base/unix-aux.mak:75: obj/aux/echogs] Error 1
ghostscript-gpl-9.19-r6: /usr/x86_64-pc-linux-gnu/binutils-bin/2.25.51/ld.bfd.real: cannot find -lpaper

drinkcat has updated the CL https://chromium-review.googlesource.com/#/c/399645/4 to add libpaper

drinkcat's updated CL seems to fix the build :

Still building ghostscript-gpl-9.19-r6 (4m55.5s). Logs in /tmp/ghostscript-gpl-9.19-r6-q7S7gR
Completed app-text/ghostscript-gpl-9.19-r6 (in 5m37.1s)

 in https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6800/steps/BuildPackages%20%5Bafdo_use%5D/logs/stdio

Tryjob with the revert of https://chromium-review.googlesource.com/#/c/391136/  https://chromium-review.googlesource.com/#/c/398941/ also passed build stage of ghostscript-gpl.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6802

The tryjob with fix https://chromium-review.googlesource.com/#/c/399645

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6800

Let's wait for 6800 to finish. If it is good we can chump in https://chromium-review.googlesource.com/#/c/399645.
If there is other failure in 6800, and 6802 is good, we can chump in https://chromium-review.googlesource.com/#/c/398941/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/44f1564239a1a73da24ca8f8526725494c3c3e32

commit 44f1564239a1a73da24ca8f8526725494c3c3e32
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Mon Oct 17 05:25:27 2016

target-chromium-os-sdk: Add lcms/libpaper needed by ghostscript tools.

ghostscript needs lcms/libpaper on the host to compile host tools.

BUG= chromium:656313 
TEST=cbuildbot --remote -g <id> cave-release

Change-Id: I1396a6ab0f2759981893b5314e9b1bd1fc2ada9f
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/399645
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
Trybot-Ready: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/44f1564239a1a73da24ca8f8526725494c3c3e32/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1.ebuild
[rename] https://crrev.com/44f1564239a1a73da24ca8f8526725494c3c3e32/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1-r67.ebuild

I expect this is more fallout of this change:

https://chromium-review.googlesource.com/#/c/391136/

Very similar to this bug:

https://bugs.chromium.org/p/chromium/issues/detail?id=654890

I expect we were implicitly relying on the SDK cups build pulling in a few host tools, but this went away when we disabled icedtea-bin's USE=cups.

I also believe previous similar errors are masked by the fact that many of the builders don't actually do clean SDK setups -- they reuse tools from previous builds, where we used to pull in these libraries. So we hit these issues first only on builders like chromiumos-sdk, where a full build is done. But this doesn't catch Ghostscript issues, because it's only the chromeos-overlay now... So now we're hitting regular builder errors now, once they refresh their SDKs (? -- I don't know when/how this happens actually).

Side note: is there no way for an ebuild to declare its host dependencies in a way that doesn't require messing with the target-chromium-os-sdk ebuild? I guess plain $DEPEND doesn't do that, since it would still build a $BOARD-targeted version.

(Extra side note: I've still got this open upstream to get ghostscript building more correctly for cross-compilation. If you have any extra thoughts, feel free to let me know:
http://bugs.ghostscript.com/show_bug.cgi?id=696508
IMO, it's really ugly right now, but at least it's closer to actually building a proper cross-compiled binary now.)

yes, that's why things have only semi-started to fail -- we don't actively unmerge packages that have been dropped from the sdk target

currently, there isn't a way for an ebuild to declare build-time deps that the sdk would pull in.  there's been talk in Gentoo for a while, but it's always stalled for various reasons.  the virtual/target-sdk is the workaround for that.

it seems weird that the build still wants libpaper even though the aux script was run with --without-libpaper explicitly.

since the bots are working again, let's close this out.  if we want to pursue cleanup, we can prob just start a new bug.  or just not worry about it for now.

@27: Yeah, that is weird. Probably (yet another) bug with Ghostscript's build system. I may take another look at that sometime, or at least mention it on the aforementioned upstream bug.