Failing example of release builder: https://uberchromegw.corp.google.com/i/chromeos/builders/guado-release/builds/1140

Passing example of pre-cq:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/34372/

Release builder builds like:
[ebuild  N     ] sys-apps/huddly-updater-0.0.1-r20::chromiumos to /build/guado/ USE="-cros-debug -cros_host -profiling" 0 KiB

Pre-cq builder builds like:
[ebuild  N     ] sys-apps/huddly-updater-0.0.1-r22::chromiumos to /build/guado/ USE="cros-debug -cros_host -profiling" 0 KiB

The key difference is the cros-debug USE flag. 

The release builder configuration removes this flag explicitly:
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/chromeos_config.py?l=1403

The paladins mostly build with cros-debug, we have one representative that does not: https://uberchromegw.corp.google.com/i/chromeos/builders/x86-mario-nowithdebug-paladin

So in a generic case where we broke the -cros-debug version we would catch it on the mario paladin, however we would not in all possible cases (like this one). 

I suspect this is a WontFix, and we just have this particular kind of bug that could fall into this hole, we just need to be careful with cros-debug related aspects of ebuilds (e.g. it is not necessarily worth building both possibilites in pre-cq and cq). For this particular failure case, vapier's change at https://chromium-review.googlesource.com/#/c/511004/ will help to plug it by default, and going forward we will at least be building guado in the pre-cq for the impacted repo and in general on all repos when the paladin instantiates on the next master waterfall restart.

Thanks for the resolution, Bernie. I agree to the analysis and the conclusion of WontFix. 


This is a discussion: This is the part I failed to fully grasp the idea behind "the fix" under prep, and I feel the use of NDEBUG and the release builder configuration are functioning, but inconsistently designed. Here is why:

b/725187 was to address the release build. To fix that, CL 510825 introduced cros-workon_src_configure(). This in turn, appears to call cros-debug-add-NDEBUG(), which is nothing but "use cros-debug || append-cppflags -DNDEBUG". 
Then finally release builds remove the cros-debug by "useflags=append_useflags(['-cros-debug'])".

So in logical hierarchy, release build did not even, to my view, need to base itself on cros-debug, and thus it did not even have to require "cros-debug-add-NDEBUG()". 


Another issue I see is the use of NDEBUG to cause a linker failure.
https://cs.corp.google.com/chromeos_public/src/aosp/external/libchrome/base/logging.h?type=cs&q=BaseInitLoggingImpl_built_without_NDEBUG+file:%5Esrc/+package:%5Echromeos_public$&l=203

Together with the release build configuration, this makes the linker failure is invisible at early stages. I wonder the use of NDEBUG and the release build configuration are designed consistently.

Adding derat@ for insights and advice.

Please discuss at will.

the linker failure is there by design -- it's how we make sure the runtime programs & libraries are in sync.  i don't know what you mean by "invisible at early stages" ... there really aren't any stages earlier than the build step.

Just to provide some history that everyone may already know:

cros-debug was added long ago to prevent us from building a shared library with NDEBUG but then linking against it from an executable that had been built (and that #include-d the library's headers) without NDEBUG, or vice versa.

Some headers include or exclude additional members in struct definitions when NDEBUG is (un)set, and if code doesn't agree internally about the size of objects, you'll get all sorts of hard-to-track-down segfaults.

yeah i think that's only tribal knowledge.  we discuss NDEBUG lightly here:
  http://dev.chromium.org/chromium-os/packages/libchrome

but prob worth adding a few more sentences on the background under the "Internal Details" section.

filed  issue 725931  to help catch this in the CQ

Thanks, I added a brief description of cros-debug to that doc.