amd64-generic-fuzzer turns red because of the same issue:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8928127837022629888

This fixes the particular error:
https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/1358746/2/fuzz/build.mk

I am seeing another error locally:
chromeos-ec-9999: In file included from fuzz/mem_hash_tree.cc:5:
chromeos-ec-9999: In file included from ./fuzz/mem_hash_tree.h:14:
chromeos-ec-9999: In file included from ./board/host/dcrypto.h:24:
chromeos-ec-9999: In file included from ./chip/g/dcrypto/dcrypto.h:16:
chromeos-ec-9999: In file included from ./chip/g/dcrypto/internal.h:13:
chromeos-ec-9999: include/util.h:67:1: error: attribute declaration must precede definition
chromeos-ec-9999:       [-Werror,-Wignored-attributes]
chromeos-ec-9999: __stdlib_compat int atoi(const char *nptr);
chromeos-ec-9999: ^
chromeos-ec-9999: include/common.h:224:40: note: expanded from macro '__stdlib_compat'
chromeos-ec-9999: #define __stdlib_compat __attribute__((visibility("hidden")))
chromeos-ec-9999:                                        ^
chromeos-ec-9999: /usr/include/stdlib.h:361:8: note: previous definition is here
chromeos-ec-9999: __NTH (atoi (const char *__nptr))
chromeos-ec-9999:        ^
chromeos-ec-9999:   CC      RO/common/queue.o
chromeos-ec-9999: 1 error generated.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/5f4f729f2fde645c377b6e3011dab82b1585a469

commit 5f4f729f2fde645c377b6e3011dab82b1585a469
Author: Allen Webb <allenwebb@google.com>
Date: Tue Dec 04 08:11:31 2018

cr50_fuzz: Disable this fuzz target until chromeos-ec is fixed.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: Ib2ffc7035d0f6912834709bb8a5f6b3c11a2c67a
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1359652
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/5f4f729f2fde645c377b6e3011dab82b1585a469/fuzz/build.mk

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/cc31a630160817c5ab332e35b0d10a0b8cd772a8

commit cc31a630160817c5ab332e35b0d10a0b8cd772a8
Author: Allen Webb <allenwebb@google.com>
Date: Tue Dec 04 22:00:04 2018

cr50_fuzz: Fix missing dependencies.

This fixes a dependency problem that was introduced in CL:1184107.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: Ib4795d6a716fe3fcb7a88bf6a165f96ffe10640a
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1358746
Reviewed-by: Mattias Nissler <mnissler@chromium.org>

[modify] https://crrev.com/cc31a630160817c5ab332e35b0d10a0b8cd772a8/fuzz/build.mk

Thanks for the fix. Can we close this?

I was waiting on https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/1361680 to land before marking this fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/1caaa593b1abee713a6d5b30ac3c20a6713ddf76

commit 1caaa593b1abee713a6d5b30ac3c20a6713ddf76
Author: Allen Webb <allenwebb@google.com>
Date: Wed Dec 05 01:40:11 2018

cr50_fuzz: Re-enable this fuzz target now that chromeos-ec is fixed.

'mem_hash_tree.h' was missing '#define HIDE_EC_STDLIB' before
'dcrypto.h'. This problem was only apparent when -O2 is set.

CQ-DEPEND=CL:1358746
BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: I19d00c165764f80cfa385fb3bed64efc67bfc3f9
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1361680
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/1caaa593b1abee713a6d5b30ac3c20a6713ddf76/fuzz/build.mk
[modify] https://crrev.com/1caaa593b1abee713a6d5b30ac3c20a6713ddf76/fuzz/mem_hash_tree.h

Hi Allen,

The fuzzer builders are broken again, Can you ptal? 

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8928014596761141392


chromeos-ec-0.0.1-r5248: clang++ -O2 -pipe -march=x86-64 -msse3 -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -DPROTOBUF_INLINE_NOT_IN_HEADERS=0 -DPROTOBUF_INLINE_NOT_IN_HEADERS=0 -DOUTDIR=build/host/cr50_fuzz/ -DCHIP=host -DBOARD_TASKFILE=ec.tasklist -DBOARD=host -DCORE=host -DPROJECT=cr50_fuzz -DCHIP_VARIANT= -DCHIP_FAMILY= -DBOARD_HOST -DCHIP_HOST -DCORE_HOST -DCHIP_VARIANT_ -DCHIP_FAMILY_ -DFINAL_OUTDIR=build/host/cr50_fuzz  -Iinclude  -Icore/host/include  -Icore/host  -Ichip/host  -Iboard/host  -Iboard/host  -Icommon  -Ifuzz  -Ipower  -Itest  -Icts/common  -Icts/  -Ibuild/host/cr50_fuzz/gen  -Iprivate  -Icommon  -Icommon/vboot  -Idriver  -Idriver/bc12  -Idriver/temp_sensor  -Idriver/charger  -Idriver/tcpm  -Idriver/battery  -Idriver/led  -Idriver/ppc  -Ibuild/host/cr50_fuzz  -Ifuzz  -Itest  -I. -DTEST_BUILD -DTEST_TASKFILE=cr50_fuzz.tasklist  -DEMU_BUILD  -DTEST_cr50_fuzz -DTEST_CR50_FUZZ -fsanitize=address -fsanitize=fuzzer-no-link -DTEST_FUZZ    -DSECTION_IS_ -DSECTION=  -DHAS_TASK_HOOKS -DHAS_TASK_HOSTCMD -DHAS_TASK_CONSOLE -I/build/amd64-generic/tmp/portage/chromeos-base/chromeos-ec-0.0.1-r5248/work/chromeos-ec-0.0.1/platform/ec/chip/g -I/build/amd64-generic/tmp/portage/chromeos-base/chromeos-ec-0.0.1-r5248/work/chromeos-ec-0.0.1/third_party/cryptoc/include  -c -o build/host/cr50_fuzz/gen/fuzz/cr50_fuzz.pb.o build/host/cr50_fuzz/gen/fuzz/cr50_fuzz.pb.cc
chromeos-ec-0.0.1-r5248:   PROTOC  gen/fuzz/pinweaver/pinweaver.pb.cc
chromeos-ec-0.0.1-r5248: clang++ -O2 -pipe -march=x86-64 -msse3 -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -DPROTOBUF_INLINE_NOT_IN_HEADERS=0 -DPROTOBUF_INLINE_NOT_IN_HEADERS=0 -DOUTDIR=build/host/cr50_fuzz/ -DCHIP=host -DBOARD_TASKFILE=ec.tasklist -DBOARD=host -DCORE=host -DPROJECT=cr50_fuzz -DCHIP_VARIANT= -DCHIP_FAMILY= -DBOARD_HOST -DCHIP_HOST -DCORE_HOST -DCHIP_VARIANT_ -DCHIP_FAMILY_ -DFINAL_OUTDIR=build/host/cr50_fuzz  -Iinclude  -Icore/host/include  -Icore/host  -Ichip/host  -Iboard/host  -Iboard/host  -Icommon  -Ifuzz  -Ipower  -Itest  -Icts/common  -Icts/  -Ibuild/host/cr50_fuzz/gen  -Iprivate  -Icommon  -Icommon/vboot  -Idriver  -Idriver/bc12  -Idriver/temp_sensor  -Idriver/charger  -Idriver/tcpm  -Idriver/battery  -Idriver/led  -Idriver/ppc  -Ibuild/host/cr50_fuzz  -Ifuzz  -Itest  -I. -DTEST_BUILD -DTEST_TASKFILE=cr50_fuzz.tasklist  -DEMU_BUILD  -DTEST_cr50_fuzz -DTEST_CR50_FUZZ -fsanitize=address -fsanitize=fuzzer-no-link -DTEST_FUZZ    -DSECTION_IS_ -DSECTION=  -DHAS_TASK_HOOKS -DHAS_TASK_HOSTCMD -DHAS_TASK_CONSOLE -I/build/amd64-generic/tmp/portage/chromeos-base/chromeos-ec-0.0.1-r5248/work/chromeos-ec-0.0.1/platform/ec/chip/g -I/build/amd64-generic/tmp/portage/chromeos-base/chromeos-ec-0.0.1-r5248/work/chromeos-ec-0.0.1/third_party/cryptoc/include  -c -o build/host/cr50_fuzz/gen/fuzz/pinweaver/pinweaver.pb.o build/host/cr50_fuzz/gen/fuzz/pinweaver/pinweaver.pb.cc
chromeos-ec-0.0.1-r5248:   EXE     cr50_fuzz.exe
chromeos-ec-0.0.1-r5248: /usr/x86_64-pc-linux-gnu/binutils-bin/2.27.0/ld.bfd.real: cannot find -lprotobuf-mutator-libfuzzer
chromeos-ec-0.0.1-r5248: /usr/x86_64-pc-linux-gnu/binutils-bin/2.27.0/ld.bfd.real: cannot find -lprotobuf-mutator

I think the solution is to pass "--sysroot=/build/$BOARD" argument when linking,
portage should also be setting $SYSROOT variable when building which can be used as well.

Another issue  I see (not related to breakage) is "-fsanitize=address" is being passed for ubsan builds. Can you tweak EC builds so that "-fsanitize=<ubsan_flags>"  are passed (look for current set of ubsan flags in cros-sanitizers.eclass).

amd64-generic-fuzzer is also failing. PTAL.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927969297004140688

I have already been looking at it. There is a ongoing discussion about the best way to handle chromeos-ec which isn't really setup to cross compile targets using anything except the core-boot sdk.

See: https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/1363674/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/4d06ddb87852e1e755e36acffd35e7f05b263714

commit 4d06ddb87852e1e755e36acffd35e7f05b263714
Author: Allen Webb <allenwebb@google.com>
Date: Thu Dec 06 14:40:49 2018

cr50_fuzz: Disable this fuzz target until chromeos-ec is fixed.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: Ib2ffc7035d0f6912834709bb8a5f6b3c11a2c67a
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1359652
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/1365641
Reviewed-by: Jett Rink <jettrink@chromium.org>
Commit-Queue: Jett Rink <jettrink@chromium.org>
Tested-by: Jett Rink <jettrink@chromium.org>
Trybot-Ready: Jett Rink <jettrink@chromium.org>

[modify] https://crrev.com/4d06ddb87852e1e755e36acffd35e7f05b263714/fuzz/build.mk

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/1d8ea65f7e4db0326427c7b03ac91a20dd86be03

commit 1d8ea65f7e4db0326427c7b03ac91a20dd86be03
Author: Allen Webb <allenwebb@google.com>
Date: Thu Dec 06 14:40:51 2018

cr50_fuzz: Fix missing dependencies.

This fixes a dependency problem that was introduced in CL:1184107.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: Ib4795d6a716fe3fcb7a88bf6a165f96ffe10640a
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1358746
Reviewed-by: Mattias Nissler <mnissler@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/1365644
Reviewed-by: Jett Rink <jettrink@chromium.org>
Commit-Queue: Jett Rink <jettrink@chromium.org>
Tested-by: Jett Rink <jettrink@chromium.org>
Trybot-Ready: Jett Rink <jettrink@chromium.org>

[modify] https://crrev.com/1d8ea65f7e4db0326427c7b03ac91a20dd86be03/fuzz/build.mk

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/fe63e5b773f90c2600e8277e802b1954591209a1

commit fe63e5b773f90c2600e8277e802b1954591209a1
Author: Allen Webb <allenwebb@google.com>
Date: Thu Dec 06 14:40:52 2018

cr50_fuzz: Re-enable this fuzz target now that chromeos-ec is fixed.

'mem_hash_tree.h' was missing '#define HIDE_EC_STDLIB' before
'dcrypto.h'. This problem was only apparent when -O2 is set.

CQ-DEPEND=CL:1358746
BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: I19d00c165764f80cfa385fb3bed64efc67bfc3f9
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1361680
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/1365645
Reviewed-by: Jett Rink <jettrink@chromium.org>
Commit-Queue: Jett Rink <jettrink@chromium.org>
Tested-by: Jett Rink <jettrink@chromium.org>
Trybot-Ready: Jett Rink <jettrink@chromium.org>

[modify] https://crrev.com/fe63e5b773f90c2600e8277e802b1954591209a1/fuzz/build.mk
[modify] https://crrev.com/fe63e5b773f90c2600e8277e802b1954591209a1/fuzz/mem_hash_tree.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/de73abca39bb953411cb0906deb7b60a0f064f73

commit de73abca39bb953411cb0906deb7b60a0f064f73
Author: Allen Webb <allenwebb@google.com>
Date: Fri Dec 07 06:06:48 2018

Makefile.toolchain: Set sysroot correctly when linking.

This fixes cross compilation by setting --sysroot=${SYSROOT} when
applicable so that targets that depend on outside libraries link
correctly.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec &&
  (cd ../platform/ec && unset BOARD && make -j buildall)

Change-Id: I6b0d3554d91460a9bca58e800c9e1cea9c3caf7a
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1363674
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/de73abca39bb953411cb0906deb7b60a0f064f73/Makefile.toolchain

Now the fail is because of a new issue:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927810754697424096

chromeos-ec-0.0.1-r5255:   EXE     cr50_fuzz.exe
chromeos-ec-0.0.1-r5255: build/host/cr50_fuzz/libec.a(libec.a.2.o): In function `nvmem_clear_cache':
chromeos-ec-0.0.1-r5255: /build/amd64-generic/tmp/portage/chromeos-base/chromeos-ec-0.0.1-r5255/work/chromeos-ec-0.0.1/platform/ec/common/nvmem.c:620: undefined reference to `nvmem_wipe_cache'
chromeos-ec-0.0.1-r5255: clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
chromeos-ec-0.0.1-r5255: make[1]: *** [Makefile.rules:438: build/host/cr50_fuzz/cr50_fuzz.exe] Error 1
chromeos-ec-0.0.1-r5255: make: *** [Makefile.rules:344: host-cr50_fuzz] Error 2

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/7f325f2e91482aee604de71f19f6669762594818

commit 7f325f2e91482aee604de71f19f6669762594818
Author: Allen Webb <allenwebb@google.com>
Date: Fri Dec 07 18:34:25 2018

cr50_fuzz: Disable this target. It is breaking the chromeos-ec build.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: I15ac87b14a0f28a62e257bb155f1862753053eb4
Reviewed-on: https://chromium-review.googlesource.com/c/1368010
Tested-by: Allen Webb <allenwebb@google.com>
Trybot-Ready: Allen Webb <allenwebb@google.com>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/7f325f2e91482aee604de71f19f6669762594818/fuzz/build.mk

I put up another CL to disable cr50_fuzz until I can resolve the root problem. I am wondering how tractable it is to keep this target building without precq checks on other CLs that go in.

I will probably need to do the work to get the fuzz targets included in the buildall part of the EC workflow to help developers catch this kind of problem before the CLs get merged.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/24d29fb0f5579a18846f2729d40102da9ce0b1b9

commit 24d29fb0f5579a18846f2729d40102da9ce0b1b9
Author: Allen Webb <allenwebb@google.com>
Date: Tue Dec 11 21:00:58 2018

Makefiles: Add support for MSAN and UBSAN.

This adds support for building fuzz targets with memory and undefined
behavior sanitizers.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec &&
    (cd ../platform/ec && unset BOARD && make -j buildall
    buildfuzztests)

Change-Id: Ic7f4c1d7fcc1f6347f091b98567167066787cb9c
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1364326
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/24d29fb0f5579a18846f2729d40102da9ce0b1b9/Makefile.toolchain
[modify] https://crrev.com/24d29fb0f5579a18846f2729d40102da9ce0b1b9/Makefile
[modify] https://crrev.com/24d29fb0f5579a18846f2729d40102da9ce0b1b9/Makefile.rules

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/e079131341dc671b12483ef94bc064153e5217f8

commit e079131341dc671b12483ef94bc064153e5217f8
Author: Allen Webb <allenwebb@google.com>
Date: Wed Dec 12 03:33:52 2018

chromeos-ec: Hook up sanitizer USE flags with the ec build.

Previously this package only ever built with address sanitization.

BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: If98c805a59fc94220c73959b629a0a1962ac9f12
Reviewed-on: https://chromium-review.googlesource.com/1363723
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/e079131341dc671b12483ef94bc064153e5217f8/chromeos-base/chromeos-ec/chromeos-ec-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/eb890d8891c23706bf1be934b4d73655b768826a

commit eb890d8891c23706bf1be934b4d73655b768826a
Author: Allen Webb <allenwebb@google.com>
Date: Sat Dec 15 06:40:31 2018

Makefiles: Set fuzz targets to use the correct CROSS_COMPILE.

This addresses a cross compilation bug for fuzzing targets where
CROSS_COMPILE was always ''.

BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec &&
    (cd ../platform/ec && unset BOARD && make -j buildall)

Change-Id: I1c0b99f4ecef1e6ddec489568ccb13a8e8f5fb85
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1363541
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Nicolas Boichat <drinkcat@chromium.org>

[modify] https://crrev.com/eb890d8891c23706bf1be934b4d73655b768826a/Makefile.toolchain
[modify] https://crrev.com/eb890d8891c23706bf1be934b4d73655b768826a/Makefile.rules

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/934c1103a2b06b3e16d24ed15bd2592f131d2b90

commit 934c1103a2b06b3e16d24ed15bd2592f131d2b90
Author: Allen Webb <allenwebb@google.com>
Date: Thu Jan 03 02:14:04 2019

cr50_fuzz: Re-enable this fuzz target now that chromeos-ec is fixed.

A new symbol was added to third_party/tpm2 that needed a mock
implementation in cr50_fuzz to fix compilation.

CQ-DEPEND=CL:1370404,CL:1377383
BRANCH=None
BUG= chromium:911310 
TEST=USE="ubsan asan fuzzer" ./build_packages \
  --board=amd64-generic --skip_chroot_upgrade chromeos-ec

Change-Id: I3e2546829646e89361bfa2a8b4084c95b346f78f
Signed-off-by: Allen Webb <allenwebb@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1370747
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/934c1103a2b06b3e16d24ed15bd2592f131d2b90/fuzz/build.mk