Looks like something uprevved the breakpad ebuild at 18:08.

commit ec6567deb04860549a1b5ad676568084aa2faa82
Author: chrome-bot <chrome-bot@chromium.org>
Date:   Thu Feb 16 18:08:23 2017 -0800

    Marking set of ebuilds as stable
    
    Marking 9999 ebuild for chromeos-base/google-breakpad as stable.
    
    Marking 9999 ebuild for chromeos-base/mobmonitor as stable.
    
    Marking 9999 ebuild for chromeos-base/ec-devutils as stable.
    
    Marking 9999 ebuild for chromeos-base/chromeos-ec as stable.
    
    Marking 9999 ebuild for chromeos-base/chromite as stable.
    
    Marking 9999 ebuild for chromeos-base/ec-utils as stable.
    
    Marking 9999 ebuild for chromeos-base/ec-utils-test as stable.
    
    Change-Id: I9885306663cae2824b8119c78996257c1457d287

Here is an example of a failed build:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/no_vmtest_pre_cq/builds/6010

google-breakpad-2017.02.13.175506-r113: armv7a-cros-linux-gnueabi-clang++ -DHAVE_CONFIG_H -I. -I/build/daisy_spring/tmp/portage/chromeos-base/google-breakpad-2017.02.13.175506-r113/work/google-breakpad-2017.02.13.175506 -I./src  -I/build/daisy_spring/tmp/portage/chromeos-base/google-breakpad-2017.02.13.175506-r113/work/google-breakpad-2017.02.13.175506/src   -Werror -Werror=missing-braces -Werror=non-virtual-dtor -Werror=overloaded-virtual -Werror=reorder -Werror=sign-compare -Werror=unused-variable -Werror=vla -fPIC -O2 -O2 -O2 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -g       -clang-syntax -g -clang-syntax -std=c++11 -c -o src/processor/microdump_processor.o /build/daisy_spring/tmp/portage/chromeos-base/google-breakpad-2017.02.13.175506-r113/work/google-breakpad-2017.02.13.175506/src/processor/microdump_processor.cc
google-breakpad-2017.02.13.175506-r113: In file included from ../src/processor/minidump_dump.cc:39:
google-breakpad-2017.02.13.175506-r113: ../src/google_breakpad/processor/minidump.h:853:8: error: 'Read' overrides a member function but is not marked 'override' [-Werror,-Winconsistent-missing-override]
google-breakpad-2017.02.13.175506-r113:   bool Read(uint32_t expected_size_);
google-breakpad-2017.02.13.175506-r113:        ^
google-breakpad-2017.02.13.175506-r113: ../src/google_breakpad/processor/minidump.h:153:16: note: overridden virtual function is here
google-breakpad-2017.02.13.175506-r113:   virtual bool Read(uint32_t expected_size) = 0;
google-breakpad-2017.02.13.175506-r113:                ^



I guess -Werror flags started getting added to clang maybe in
https://chromium-review.googlesource.com/c/441864/

i've rolled the branch back now

The problematic change has been reverted, but the PreCQ is expected to remain broken until the next CQ uprev (which can happen, even for some CQ failures).

Tree failure due to -Werror is now back, now in Plaadins:

https://luci-milo.appspot.com/buildbot/chromeos/kevin-paladin/261
https://luci-milo.appspot.com/buildbot/chromeos/elm-paladin/1909
https://luci-milo.appspot.com/buildbot/chromeos/oak-paladin/6465

google-breakpad-2017.02.20.180314-r115: In file included from ../src/client/linux/handler/exception_handler.cc:66:
google-breakpad-2017.02.20.180314-r115: In file included from ../src/client/linux/handler/exception_handler.h:42:
google-breakpad-2017.02.20.180314-r115: In file included from ../src/client/linux/minidump_writer/minidump_writer.h:41:
google-breakpad-2017.02.20.180314-r115: In file included from ../src/client/linux/minidump_writer/linux_dumper.h:51:
google-breakpad-2017.02.20.180314-r115: In file included from ../src/client/linux/dump_writer_common/thread_info.h:37:
google-breakpad-2017.02.20.180314-r115: In file included from ../src/common/memory.h:50:
google-breakpad-2017.02.20.180314-r115: ../src/third_party/lss/linux_syscall_support.h:2613:31: error: deprecated instruction in IT block [-Werror,-Winline-asm]
google-breakpad-2017.02.20.180314-r115:                              "moveq %0,%1\n"
google-breakpad-2017.02.20.180314-r115:                               ^
google-breakpad-2017.02.20.180314-r115: <inline asm>:6:1: note: instantiated into assembly here
google-breakpad-2017.02.20.180314-r115: moveq r6,#-22
google-breakpad-2017.02.20.180314-r115: ^
google-breakpad-2017.02.20.180314-r115: 1 error generated.

Is it because https://chromium-review.googlesource.com/c/444385/ is chumped today? Can we re-revert these?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/google-breakpad/+/bf799046e75d00f611eba6190388a6c79078e8a2

commit bf799046e75d00f611eba6190388a6c79078e8a2
Author: Ryo Hashimoto <hashimoto@google.com>
Date: Tue Feb 21 07:25:45 2017

Stop applying -Werror

Our aarch64 build fails with -Werror.
This change should be reverted once the upstream gets fixed.

BUG= chromium:693313 
TEST=emerge-elm google-breakpad

Change-Id: Ib674905e8c3eb612c1973bdd0ef101a5832cdc9c
Reviewed-on: https://chromium-review.googlesource.com/445556
Reviewed-by: Kazuhiro Inaba <kinaba@chromium.org>
Tested-by: Ryo Hashimoto <hashimoto@chromium.org>
Trybot-Ready: Ryo Hashimoto <hashimoto@chromium.org>

[modify] https://crrev.com/bf799046e75d00f611eba6190388a6c79078e8a2/configure.ac
[modify] https://crrev.com/bf799046e75d00f611eba6190388a6c79078e8a2/configure

I've landed https://chromium-review.googlesource.com/445556 to fix 64-bit ARM builds.
Please verify with all architectures when doing breakpad roll next time.

the error doesn't come up on all ARM systems, only armv8 ones.  so testing against other ARM wouldn't catch this bug.

i've opened https://bugs.chromium.org/p/linux-syscall-support/issues/detail?id=18 to track the issue.

By "all architectures", I meant please also verify with aarch64 boards (e.g. elm).

Two things;
 - why didn't it get caught and auto-uprev'd ? 
 - it seemed nobody on my sheriff rotation knew how to roll changes back in breakpad.  Can this be documented?

> - why didn't it get caught and auto-uprev'd ? 
The problem is that it seems no bot for aarch64 boards is included in the uprev process and CQ.

> - it seemed nobody on my sheriff rotation knew how to roll changes back in breakpad.  Can this be documented?
IIUC only people with chromiumos-breakpad-owners gerrit group membership can perform roll which requires push capability.
https://chromium-review.googlesource.com/?polygerrit=0#/admin/projects/chromiumos/platform/google-breakpad,access

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/452d1fcc8faf8f3bcb9c1394e53ed5f42c798c7a

commit 452d1fcc8faf8f3bcb9c1394e53ed5f42c798c7a
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Jul 03 18:00:51 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/manifest/+/6803e7219065962a8c30ac986e035553902c1094

commit 6803e7219065962a8c30ac986e035553902c1094
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Jul 03 18:00:51 2017

breakpad: update lss deps

This has fixed ARM assembly.

BUG= chromium:693313 
TEST=build still works

Change-Id: Ib4acbeae87f783a0713d7c0fd60ba17f44de8c52
Reviewed-on: https://chromium-review.googlesource.com/449912
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>

[modify] https://crrev.com/6803e7219065962a8c30ac986e035553902c1094/full.xml