Adding Zhifeng. I have the impression _BSD_SOURCE has been in use for a long time - the change is morely around treating the warning as a error.

BTW, this is the part of the log I'm referring to:
jetstream-platform-support-3-r14: make: Entering directory '/build/whirlwind/tmp/portage/virtual/jetstream-platform-support-3-r14/work/jetstream-platform-support-3/liblightcontrol'
...
jetstream-platform-support-3-r14: armv7a-cros-linux-gnueabi-gcc -Wall -Werror -fPIC -std=c99 -D_BSD_SOURCE   -c -o sp5.o sp5.c

jetstream-platform-support-3-r14:  # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
jetstream-platform-support-3-r14:    ^
jetstream-platform-support-3-r14: cc1: all warnings being treated as errors

#1458 build completed
#1459 build failed with the same error.
#1460 build failed with the same error.

Now to track down what really changed between those builds. :/

Sorry, I don't know why I thought that the following build passed.  Looks like this really needs attention.  Thank you.

I can't reproduce this locally. I repo sync'd and Whirlwind is building fine for me. Can you "clobber" the builder and restart it?

Ningning, can you help with this?  Thanks!

Issue 678534 has been merged into this issue.

If you want to clobber the build, you can just need to stop the build and force it with clobber checked on the UI. 
The current run long lasts 40 mins, if you want, I can stop and force a build now. The release-master is using Buildbucket to schedule slaves now, so if the build didn't finish in time, next release-master build will cancel/stop it anyway. semenzato@, please decide.

Thank you Ningning.

Grant, this is failing repeatedly, and canary builders build everything from scratch.  Why will clobbering the build help this?

Ningning do you think it can help?

I am starting a build on my workstation, see if I can repro.

My build didn't repro either.

Is it the case that the cached build for a repo is hiding this?

Ningning, do you know how to invalidate or bypass the cache?

I think we don't know what kind cache is affecting this. Clobbering might be a try here.

I just kicked off a "force build" w/clobber checked.

These builders seem to still be failing on this error :(.

Grant, which builders did you attempt a clobber on?

Whirlwind-paladin seems ok (notwithstanding the HWTest failures), but whirlwind-release (and gale-release) are still broken.

If we can confirm clobber did not fix it, and we are certain it is a problem with the builder's cached state, we can bring in more expertise on doing a more full wipe of the builder (dgarrett@).

@bhthompson I did a "force build" and checked the clobber button that was listed here:
   https://uberchromegw.corp.google.com/i/chromeos/builders/whirlwind-release/

adding dgarrett since it's clearly outside of semenzato and my "level of expertise".

BTW, arkham builds are nearly identical but don't see this failure:
   https://uberchromegw.corp.google.com/i/chromeos/builders/arkham-release

raising priority since this is blocking us from releasing

grundler, that was the right way to do a clobber build.

However, since the builder is a GCE instance, I can totally reinstantiate it pretty easily.

@dgarrett - perfect - thanks. Please "reinstantiate" ASAP and kick off a new build. Until I can repro at my desk, I'm suspicious of any caches used by builders or build tools. Maybe that's what I need to do: clobber all local caches and retest. :/

But I've reinstantiated, and the builder is back online. Force a build if that's the right thing to do.

PS:
"cros clean --clobber" will wipe pretty much everything but source from a local sandbox.

Thanks for the command - I didn't know that. But it doesn't like me:
(cr) ((8fc7254...)) grundler@firesword ~/trunk/src/scripts $ cros clean --clobber
14:44:23: ERROR: cros: please run outside the chroot

It likes you, just run it outside the chroot. Since part of what it kills is the chroot, that's a reasonable requirement.

gale-paladin is failing the same way and it started failing around the same time. This suggests some common infrastructure changed. Is there a log of toolchain changes?

First failure on Jan 4th:
https://uberchromegw.corp.google.com/i/chromeos/builders/gale-paladin?numbuilds=100

Logs for most failure (Jan 10th):
https://uberchromegw.corp.google.com/i/chromeos/builders/gale-paladin/builds/1334/steps/BuildPackages/logs/stdio

...
liblightcontrol-0.0.1-r8: /build/gale/usr/include/features.h:148:3: error: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-Werror=cpp]
liblightcontrol-0.0.1-r8:  # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
liblightcontrol-0.0.1-r8:    ^
liblightcontrol-0.0.1-r8: In file included from /build/gale/usr/include/stdint.h:25:0,
liblightcontrol-0.0.1-r8:                  from /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.9.x/include/stdint.h:9,
liblightcontrol-0.0.1-r8:                  from gale_common.h:9,
liblightcontrol-0.0.1-r8:                  from gale.c:5:
liblightcontrol-0.0.1-r8: /build/gale/usr/include/features.h:148:3: error: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-Werror=cpp]
liblightcontrol-0.0.1-r8:  # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
liblightcontrol-0.0.1-r8:    ^
liblightcontrol-0.0.1-r8: cc1: all warnings being treated as errors
liblightcontrol-0.0.1-r8: make: *** [<builtin>: liblightcontrol.o] Error 1
...

And after running "cros clean --clobber", I can now reproduce the failure (for gale build in this case):

liblightcontrol-0.0.1-r8: /build/gale/usr/include/features.h:148:3: error: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-Werror=cpp]
liblightcontrol-0.0.1-r8:  # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
liblightcontrol-0.0.1-r8:    ^
liblightcontrol-0.0.1-r8: cc1: all warnings being treated as errors
liblightcontrol-0.0.1-r8: make: *** [<builtin>: liblightcontrol.o] Error 1

So my next question is: how did this get committed? :/  But maybe it's a bit earlier for post-mortem discussion. Let me see if I can fix this.

(cr) ((8fc7254...)) grundler@firesword ~/trunk/src/scripts $ equery-gale b features.h
 * Searching for features.h ... 
chromeos-base/libchrome-395517-r4 (/usr/include/base-395517/base/allocator/features.h)

Ugh. This is a hairball. ejcaruso is trying to update libchrome but can't due to dependencies by libweave on a specific version of libchrome. :(

Given the libchrome version is "hard coded" in the libweave ebuild, I expect this failure started with a toolchain or other infrastructure change. Let me talk to eric about getting this patched ASAP and roll libchrome version.

features.h is a red herring - sorry.

arkham: liblightcontrol/Makfile does NOT have "-D_BSD_SOURCES"
whirlwind: does have
gale:      also has

testing a patch now.

Can someone ASAP review and submit the follow two CLs:
   https://chrome-internal-review.googlesource.com/315007
   https://chrome-internal-review.googlesource.com/315031

It's past my lunch time and I'm getting grumpy.

Don,
The respective liblightcontrol files that I'm changing above haven't changed for months. That's why I didn't look there first.

Can you please follow up why the whirlwind/gale builds started failing on Jan 4?

My best guess is toolchain changed to actually enforce errors OR eclass stuff changed to enforce errors. (perhaps just not ignore them?)

TIA!

whirlwind/gale started failing hwtesting because of a shard issue in the lab ( crbug.com/679410 ).

Don, perhaps the shard issue triggered a clobber which first exposed the build error?

Luis,
Do you happen to know where the toolchain history is for the arm builds? I'm trying to determine what changed on or before Jan 4 which led to whirlwind/gale builders repeatedly failing with the build errors copied into comments above.

+yunlian,

we upgraded glibc around that time. 

Yunlian, is this related?

Yes, we fixed a similar issue via this CL
https://chromium-review.googlesource.com/#/c/349275/

So we should do similar change for this package.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-gale-private/+/08e9ac970b305b1e4bd057e75740bee93c912a8a

commit 08e9ac970b305b1e4bd057e75740bee93c912a8a
Author: Grant Grundler <grundler@google.com>
Date: Tue Jan 10 19:47:29 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-whirlwind-private/+/6a3344496251d70980795836dd107954763db8eb

commit 6a3344496251d70980795836dd107954763db8eb
Author: Grant Grundler <grundler@google.com>
Date: Tue Jan 10 20:41:29 2017

Hisham can break out the champaign now :)

Both green:
https://uberchromegw.corp.google.com/i/chromeos/builders/whirlwind-release/builds/1479
https://uberchromegw.corp.google.com/i/chromeos/builders/gale-release/builds/858

Yunlian, why didn't this fix get applied to other packages that needed it before upgrading glibc?

And how does adding DEFAULT_SOURCE fix the warning about _BSD_SOURCE? (TIA)

Lastly, is the glibc upgrade also responsible for making sure debug libs and regular libs are in sync? (Adding jdiez in case he knows the bug number)

http://man7.org/linux/man-pages/man7/feature_test_macros.7.html answered some of my questions about _DEFAULT_SOURCE and _BSD_SOURCE.

My impression is by using -std=c99 flag on the gcc command line, _DEFAULT_SOURCE is not needed. Is that correct?

My only concern is we have now have some *potential* changes in behavior "where standards conflict" since _BSD_SOURCE is no longer defined. My gut feeling is liblightcontrol is unlikely to be doing anything that cares about BSD vs POSIX behavior. But is there a tool that will scan code for such function calls?

Such a scanning tool would make it trivial to remove _BSD_SOURCE from code that doesn't need it.

+vapier

In general, we should use _DEFAULT_SOURCE to replace _BSD_SOURCE. For this particular package, I think using -std=c99 is fine. I am not aware of any tools that can do this scanning. For ChromeOS, we only found one problem besides this one.
We will test jetstream boards for out next toolchain upgrades.

yes, you can replace _BSD_SOURCE with _DEFAULT_SOURCE if you don't care about older versions of glibc.  if you want to maximize compatibility, you can define both at the same time and glibc won't warn.