Trying to see if https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1241532 is relevant.

I think the query is based on this:
https://cs.chromium.org/chromium/src/third_party/chromite/lib/builder_status_lib.py?q=chromite/lib/builder_status_lib.py&sq=package:chromium&g=0&l=106&rcl=ffb9516fca03509b5fe23d337e8ce8f7941b7089

But still not sure if "adding index for id" can cause this.

CIDBConnection.GetBuildStatuses can compose invalid query when passed an empty set.
https://cs.chromium.org/chromium/src/third_party/chromite/lib/cidb.py?q=chromite/lib/cidb.py&sq=package:chromium&g=0&pv=1&rcl=0464dbfe8e216882669df371cebd5c81879444f2&l=1192

In the trace, the set was got by CIDBConnection.GetBuildMessages.
https://cs.chromium.org/chromium/src/third_party/chromite/lib/builder_status_lib.py?q=builder_status&sq=package:chromium&g=0&rcl=ffb9516fca03509b5fe23d337e8ce8f7941b7089&l=106

Let's take master-paladin/19728 build for an example:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933926949639185056

Its metadata.json contains "self_destructed_build": true
https://storage.cloud.google.com/chromeos-image-archive/master-paladin/R71-11115.0.0-rc1/metadata.json

That's why GetSlavesAbortedBySelfDestructedMaster() was called.

Then it's strange why it could not find any self-destructed slaves. self-destructed messages are inserted in build_status.py:

        logging.warning('This build will self-destruct given the results of '
                        'relevant change triages.')

        if should_self_destruct_with_success:
          logging.info('This build will self-destruct with success.')

        self.metadata.UpdateWithDict({
            constants.SELF_DESTRUCTED_BUILD: True,
            constants.SELF_DESTRUCTED_WITH_SUCCESS_BUILD:
            should_self_destruct_with_success})

        fields = {
            'build_config': self.config.name,
            'self_destructed_with_success': should_self_destruct_with_success}
        metrics.Counter(constants.MON_CQ_SELF_DESTRUCTION_COUNT).increment(
            fields=fields)

        # For every uncompleted build, the master build will insert an
        # ignored_reason message into the buildMessageTable.
        for build in uncompleted_important_builds:
          if build in self.all_cidb_status_dict:
            self.db.InsertBuildMessage(
                self.master_build_id,
                message_type=constants.MESSAGE_TYPE_IGNORED_REASON,
                message_subtype=constants.MESSAGE_SUBTYPE_SELF_DESTRUCTION,
                message_value=str(self.all_cidb_status_dict[build].build_id))
        builder_status_lib.CancelBuilds(uncompleted_build_buildbucket_ids,
                                        self.buildbucket_client,
                                        self.dry_run,
                                        self.config)

The first warning message can be found in the log, the code path was surely hit. I'm not sure what went wrong from current logs.

In master-paladin 19729,
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933910306518919888
coral-paladin and eve-paladin are both failing by the same error as Issue 890446: "AC power is not plugged in".
That issues says it's happening periodically. So I think this is still the case.

crrev.com/c/1253483 is intended to suppress the runtime error, but the question in #3 still needs to be addressed.

Did the fix in https://crrev.com/c/1243145 allow this to be reported correctly? If so, this might fix the CQ hang problem we've seen from time to time ( https://crbug.com/883421 ).

Excellent!

 Issue 877342  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/bdf3daa3a79e6e4c6506c0022ae8512ec696b9ff

commit bdf3daa3a79e6e4c6506c0022ae8512ec696b9ff
Author: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Date: Tue Oct 02 12:20:01 2018

Avoid CommitQueueCompletion failure by ProgrammingError.

This is a tantative fix for  crbug.com/890651 .
We haven't got the root cause yet, but at least when passing an
empty passed to GetBuildStatuses, it causes runtime error by an invalid
SQL query.

Bug:890651
Change-Id: I7ef3ffe57a2ac6fb4aff98585adcfb47fb9c635c
Reviewed-on: https://chromium-review.googlesource.com/1253483
Commit-Ready: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/bdf3daa3a79e6e4c6506c0022ae8512ec696b9ff/lib/builder_status_lib.py

Need to cherry-pick both:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1253483 (Avoid CommitQueueCompletion failure by ProgrammingError.)
and
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1253962 (
cidb: Include branch info on inserting builds.)

back to active release branches. I'll manage that today.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/b199fb1faccf7c84c8fd10a4146137d440717d0f

commit b199fb1faccf7c84c8fd10a4146137d440717d0f
Author: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Date: Wed Oct 03 15:12:49 2018

Avoid CommitQueueCompletion failure by ProgrammingError.

This is a tantative fix for  crbug.com/890651 .
We haven't got the root cause yet, but at least when passing an
empty passed to GetBuildStatuses, it causes runtime error by an invalid
SQL query.

Bug:890651
Change-Id: I7ef3ffe57a2ac6fb4aff98585adcfb47fb9c635c
Reviewed-on: https://chromium-review.googlesource.com/1253483
Commit-Ready: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
(cherry picked from commit bdf3daa3a79e6e4c6506c0022ae8512ec696b9ff)
Reviewed-on: https://chromium-review.googlesource.com/c/1258528
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Commit-Queue: Jason Clinton <jclinton@chromium.org>
Trybot-Ready: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/b199fb1faccf7c84c8fd10a4146137d440717d0f/lib/builder_status_lib.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/d0400f54e1d1860c42aab40acb26f0aac9bb5fc1

commit d0400f54e1d1860c42aab40acb26f0aac9bb5fc1
Author: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Date: Wed Oct 03 15:16:09 2018

Avoid CommitQueueCompletion failure by ProgrammingError.

This is a tantative fix for  crbug.com/890651 .
We haven't got the root cause yet, but at least when passing an
empty passed to GetBuildStatuses, it causes runtime error by an invalid
SQL query.

Bug:890651
Change-Id: I7ef3ffe57a2ac6fb4aff98585adcfb47fb9c635c
Reviewed-on: https://chromium-review.googlesource.com/1253483
Commit-Ready: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
(cherry picked from commit bdf3daa3a79e6e4c6506c0022ae8512ec696b9ff)
Reviewed-on: https://chromium-review.googlesource.com/c/1258526
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Commit-Queue: Jason Clinton <jclinton@chromium.org>
Trybot-Ready: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/d0400f54e1d1860c42aab40acb26f0aac9bb5fc1/lib/builder_status_lib.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f29c89f69ad31cb9c1a100e1036a253fb38cd481

commit f29c89f69ad31cb9c1a100e1036a253fb38cd481
Author: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Date: Wed Oct 03 15:16:10 2018

Avoid CommitQueueCompletion failure by ProgrammingError.

This is a tantative fix for  crbug.com/890651 .
We haven't got the root cause yet, but at least when passing an
empty passed to GetBuildStatuses, it causes runtime error by an invalid
SQL query.

Bug:890651
Change-Id: I7ef3ffe57a2ac6fb4aff98585adcfb47fb9c635c
Reviewed-on: https://chromium-review.googlesource.com/1253483
Commit-Ready: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
(cherry picked from commit bdf3daa3a79e6e4c6506c0022ae8512ec696b9ff)
Reviewed-on: https://chromium-review.googlesource.com/c/1258527
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Commit-Queue: Jason Clinton <jclinton@chromium.org>
Trybot-Ready: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/f29c89f69ad31cb9c1a100e1036a253fb38cd481/lib/builder_status_lib.py

All cherry-picks complete. Thanks for the help. Closing this out.