Could these changes be related to this failure?
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1257591
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1257592

My CLs don't touch tcpdump package.

has the toolchain rolled in a way that'd cause this ?  the flurry of assembler related CLs is a bit suspicious if the toolchain hasn't been updated.

Toolchain was updated last week.

Note that tael-full is green: https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=tael-full&buildBranch=master. 

Just to verify that my CLs are not at fault, The try job with both my CLs is green for tael-full.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933688629721099584

The package itself has problems. When cross-compiling, it used system headers.

 armv7a-cros-linux-gnueabihf-clang -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -O2 -DHAVE_CONFIG_H  -DHAVE_CAP_NG_H -I./missing  -D_U_="__attribute__((unused))" -I. -I/usr/include  -I./missing -DINET6 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -O2 -c ./addrtoname.c

It uses -I/usr/include when cross compiling.
Upstream 4.9.2 still has this problem.

benchan@ is the owner of tcpdump. Ben, tcpdump is currently breaking tael-paladin.

Note that not every tael-paladin build fails. We have failed #1354 and #1357 with this build error, and green #1355 and #1356 in between.

I have a suspicion that tcpdump succeeds when building for first time (Case for full builds) but fails to build when rebuilding (case for paladin builds) maybe because its configure stage found an existing installation and decided to add more flags?

ANd in both failed CQ runs I see https://crrev.com/c/1257591 and https://crrev.com/c/1257592. While the successful runs were made w/o them. So, one of them must be at fault here.

@6: How do you figure that? My local builds show this:

x86_64-cros-linux-gnu-clang -pipe -march=x86-64 -msse3 -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -O2 -DHAVE_CONFIG_H  -DHAVE_CAP_NG_H -I./missing  -D_U_="__attribute__((unused))" -I. -I/build/betty/usr/include  -I./missing -DINET6 -pipe -march=x86-64 -msse3 -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -O2 -c ./print-sunatm.c

which has the correct /build/$BOARD/... include path

As Yunlian already pointed out, the problem is in tcpdump's build flags. 
It is using -I/usr/include instead of -I/build/tael/usr/include. 

Regarding #12, I suggest building on tael board and then try to build+rebuild tcpdump. Flags on x86/arm can often vary.

Well, this package hasn't changed in forever, so it seems like we should be able to mitigate the damage somehow... apronin is investigating reverts.

But it looks like this bit is yielding bad include flags for ARM systems. (I can't even build tcpdump on my few-day-old chroot on ARM -- so this package must have been broken for a while...):

dnl
dnl Do whatever AC_LBL_C_INIT work is necessary before using AC_PROG_CC.
dnl
dnl It appears that newer versions of autoconf (2.64 and later) will,
dnl if you use AC_TRY_COMPILE in a macro, stick AC_PROG_CC at the
dnl beginning of the macro, even if the macro itself calls AC_PROG_CC.
dnl See the "Prerequisite Macros" and "Expanded Before Required" sections
dnl in the Autoconf documentation.
dnl
dnl This causes a steaming heap of fail in our case, as we were, in
dnl AC_LBL_C_INIT, doing the tests we now do in AC_LBL_C_INIT_BEFORE_CC,
dnl calling AC_PROG_CC, and then doing the tests we now do in
dnl AC_LBL_C_INIT.  Now, we run AC_LBL_C_INIT_BEFORE_CC, AC_PROG_CC,
dnl and AC_LBL_C_INIT at the top level.
dnl
AC_DEFUN(AC_LBL_C_INIT_BEFORE_CC,
[
    AC_BEFORE([$0], [AC_LBL_C_INIT])
    AC_BEFORE([$0], [AC_PROG_CC])
    AC_BEFORE([$0], [AC_LBL_FIXINCLUDES])
    AC_BEFORE([$0], [AC_LBL_DEVEL])
    AC_ARG_WITH(gcc, [  --without-gcc           don't use gcc])
    $1=""
    if test "${srcdir}" != "." ; then
            $1="-I$srcdir"
    fi
    if test "${CFLAGS+set}" = set; then
            LBL_CFLAGS="$CFLAGS"
    fi
    if test -z "$CC" ; then
            case "$host_os" in

            bsdi*)
                    AC_CHECK_PROG(SHLICC2, shlicc2, yes, no)
                    if test $SHLICC2 = yes ; then
                            CC=shlicc2
                            export CC
                    fi
                    ;;
            esac
    fi
    if test -z "$CC" -a "$with_gcc" = no ; then
            CC=cc
            export CC
    fi
])

I can't build tcpdump locally with an old chroot.


Given that tael-full works, I suppose a temporary fix for tael-paladin would be to clear the builder chroot and rebuild all packages from source.

See this Cl as an example to clear prebuilts.
https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/312171

Ignore #14. The error is further down. For some reason, ARM builds end up using the base pcap-config, whose --cflags points you at /usr/include instead of ${SYSROOT}/usr/include:

tcpdump-4.5.1-r1: checking for armv7a-cros-linux-gnueabihf-pcap-config... no
tcpdump-4.5.1-r1: checking for pcap-config... /usr/bin/pcap-config
tcpdump-4.5.1-r1: configure: WARNING: using cross tools not prefixed with host triplet

that might be due to stale binpkgs from the gnueabi->gnueabihf transition ?

i wonder if we force a bump of libpcap if we'll get the right wrapper script again

I just sent https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1258578 that might fix it by clearing the prebuilts.

@17: yep, when I force this symlink

$ pwd
/build/scarlet/build/bin
$ sudo ln -s armv7a-cros-linux-gnueabi-pcap-config armv7a-cros-linux-gnueabihf-pcap-config

my local build works:

checking for armv7a-cros-linux-gnueabihf-pcap-config... /build/scarlet/build/bin/armv7a-cros-linux-gnueabihf-pcap-config

(Not a package problem. Problem with ABI switchover.)

For the reference my ToT tael-paladin trybot failed: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933664546529140816

The paladin trybot with the two CLs reverted was fine: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8933664471269751392

Same with tael-compile-only-pre-cq.
Failed (still running but the error is there) ToT: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933664545779004512

Succeeded with reverted CLs: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8933664469899173472

And failed again in CQ with the same error.

So, unless we have an alt fix, regardless of what package is at fault, and what just triggers the bug by changing something, have to revert to keep CQ green :(

Yunlian, I thought you already cleared prebuilts when doing gnueabi->gnueabihf transition, or did you switch tael's CHOST after that?

Trying https://crrev.com/c/1258578
on the same two trybots as alt fix:
  https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933660629997823824
  https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933660628755265504

yunlian, can you clear the prebuilts again globally?

Per Mike's suggestion, potential revbump fix:

https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/1259622

Tryjobs (paladin and compile-only-pre-cq):

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933660091050629376
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933660066195764832

Both my tryjobs have finished building tcpdump successfully. Is that chump-worthy?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/67ef3ec1e94f52cb728284ffb88f96e8f90bc111

commit 67ef3ec1e94f52cb728284ffb88f96e8f90bc111
Author: Brian Norris <briannorris@chromium.org>
Date: Wed Oct 03 20:10:18 2018

net-libs/libpcap: revbump

Some builders aren't getting this rebuilt, and so it doesn't reflect the
change in ARM ABI. This can cause downstream package build failures.

BUG= chromium:891620 
TEST=tael-paladin tryjob

Change-Id: I46708d6374599d4307cd4f7986b7217393e56698
Signed-off-by: Brian Norris <briannorris@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/1259622
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[add] https://crrev.com/67ef3ec1e94f52cb728284ffb88f96e8f90bc111/net-libs/libpcap/libpcap-1.5.3-r1.ebuild

I think I'll call this Fixed.

Dunno if there's follow-up work to see what's really up with prebuilts (and how to clear them properly in the future).

Sheriff follow-up: unmark tael-paladin EXPERIMENTAL once we're sure it's picking up a good fix.

Thanks everyone!