groeck@, can you paste the link to the unit test failures? did the unit tests failed because of infra issues?

I see kahlee-paladin has been passing except some abort caused by CQ self-destruction.
https://uberchromegw.corp.google.com/i/chromeos/builders/kahlee-paladin

Found an example:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/62449/steps/UnitTest/logs/stdio



quipper-0.0.1-r2055: [libprotobuf FATAL ../../protobuf-3.3.0/src/google/protobuf/message_lite.cc:71] CHECK failed: (bytes_produced_by_serialization) == (byte_size_before_serialization): Byte size calculation and serialization were inconsistent.  This may indicate a bug in protocol buffers or it may be caused by concurrent modification of quipper.PerfDataProto.
quipper-0.0.1-r2055: Error: /var/cache/portage/chromeos-base/quipper/out/Default/perf_recorder_test: failed with signal SIGIOT|SIGABRT(6)
quipper-0.0.1-r2055:  * ERROR: chromeos-base/quipper-0.0.1-r2055::chromiumos failed (test phase):




libbrillo-0.0.1-r970: [ERROR:dbus_method_invoker.h(117)] CallMethodAndBlockWithTimeout(...): Domain=dbus, Code=org.freedesktop.DBus.Error.Failed, Message=Failed to call D-Bus method: org.test.Object.TestInterface.TestMethod3
libbrillo-0.0.1-r970: ../../../../../../../tmp/portage/chromeos-base/libbrillo-0.0.1-r970/work/libbrillo-0.0.1/platform2/libbrillo/brillo/dbus/dbus_method_invoker_unittest.cc:126: Failure
libbrillo-0.0.1-r970: Expected: (nullptr) != (response.get()), actual: 8-byte object <00-00 00-00 00-00 00-00> vs NULL
libbrillo-0.0.1-r970: [FATAL:dbus_method_invoker.h(228)] Check failed: message. Unable to extract parameters from a NULL message.
libbrillo-0.0.1-r970: /usr/lib64/libbase-core-395517.so(base::debug::StackTrace::StackTrace()+0x13) [0x7f9dee8fbdd3]
libbrillo-0.0.1-r970: 
libbrillo-0.0.1-r970: Error: /var/cache/portage/chromeos-base/libbrillo/out/Default/libbrillo-395517_unittests: failed with signal SIGIOT|SIGABRT(6)
libbrillo-0.0.1-r970:  * ERROR: chromeos-base/libbrillo-0.0.1-r970::chromiumos failed (test phase):
libbrillo-0.0.1-r970:  *   (no error message)





And I found some passed kahlee-pre-cq:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/62483
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/62514

and some failed kahlee-pre-cq:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/62473

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/62449


The unittests in kahlee-pre-cq are flaky, we should either find the owner from libbrillo and quipper to fix the unit tests, or remove the unit tests if necessary.

mysql> select id, buildbucket_id, status from buildTable where build_config='kahlee-pre-cq' order by id desc limit 15;
+---------+---------------------+--------+
| id      | buildbucket_id      | status |
+---------+---------------------+--------+
| 1958878 | 8965337783442382464 | pass   |
| 1958401 | 8965347412157315936 | pass   |
| 1958305 | 8965350133319068256 | fail   |
| 1957982 | 8965356267548223920 | fail   |
| 1957603 | 8965364210016453216 | fail   |
| 1957299 | 8965369315276431472 | pass   |
| 1957184 | 8965373200453561504 | pass   |
| 1956769 | 8965380703564618464 | fail   |
| 1956722 | 8965381676059009056 | pass   |
| 1954769 | 8965435768005023104 | fail   |
| 1954740 | 8965437503828380544 | pass   |
| 1953183 | 8965467688820681664 | fail   |
| 1953154 | 8965468688681391808 | pass   |
| 1952888 | 8965478634065346816 | pass   |
| 1951414 | 8965522560900175648 | pass   |
+---------+---------------------+--------+


The failure rate is high, raising the priority

sheriffs@, can you help to find the owners for libbrillo and quipper 

Re OP, this is not in the default set of pre-cqs

It _is_ in the pre-cq list for kernel 4.4 (+snanda) which may widen its impact.
https://cs.corp.google.com/chromeos_public/src/third_party/kernel/v4.4/COMMIT-QUEUE.ini?type=cs&q=kahlee-pre-cq&sq=package:chromeos&l=14

Finally, this is not an infra bug.

Re #6: 
I'm not sure we need to be alarmed. We don't know how many of those are bad CLs.
I see one series of three failures. This should have triggered a ToT pre-cq build, which would have passed.

There isn't much we can do in infra in response to flakiness of this kind, but we can make it more visible sometimes...
This case is particularly hard because pre-cqs fail mostly because of bad CLs (that's their purpose) and we don't run enough tot-pre-cqs to infer flakiness from them.

kahlee uses the 4.12 kernel, via chipset-stnyridge:

chipset-stnyridge/profiles/base/make.defaults:USE="${USE} kernel-4_12"

Maybe drop from COMMIT-QUEUE.ini for chromeos-4.4 and add to chromeos-4.12 ?

quipper has a team we can file a bug for.  not sure if they use crbug.com or something else, but we can throw something over their direction.

libbrillo is owned by the CrOS team ;).

#4: I thought I did. The link I provided points to various failed CLs, which in turn point to failed builds. Here are some.

https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/62449
https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/62397
https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/61612
https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/62204
https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/61951

CL:728366 for dropping kahlee-pre-cq from COMMIT-QUEUE.ini in chromeos-4.4.

I'll take care of COMMIT-QUEUE.ini, but who should own the unit test flakiness ? Any suggestions for compoment(s) ?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d222098e5241ecc88e1c939fc884b1bbcc65b2bd

commit d222098e5241ecc88e1c939fc884b1bbcc65b2bd
Author: Guenter Roeck <groeck@chromium.org>
Date: Fri Oct 20 04:15:26 2017

COMMIT-QUEUE.ini: Drop kahlee-pre-cq

kahlee uses chromeos-4.12. Its unit tests are flaky, causing unnecessary
pre-cq failures. Let's drop it from chromeos-4.4.

BUG= chromium:776369 
TEST=Run pre-cq on chromeos-4.4 patches

Change-Id: Ib09799cf46adea3506c1b22048156988b3d5e046
Signed-off-by: Guenter Roeck <groeck@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/728366
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Justin TerAvest <teravest@chromium.org>

[modify] https://crrev.com/d222098e5241ecc88e1c939fc884b1bbcc65b2bd/COMMIT-QUEUE.ini

I see a failure on the CQ overnight on a different board (grunt) that looks very similar to this one: https://luci-milo.appspot.com/buildbot/chromeos/grunt-paladin/2668

@20 - the grunt-paladin quipper failure is tracked in  crbug.com/879206 

kahlee builder was removed this bug is techinical obsolete.