It keeps failing. But commit queue is still trying to merge new CLs.
So pre-cq failure actually does not block commit queue.

Also happens on nyan_freon-paladin
 https://uberchromegw.corp.google.com/i/chromeos/builders/nyan_freon-paladin/builds/8846

I guess that builder should be removed ?

My apologies. I must have misunderstood how things work, because I assumed CL https://chromium-review.googlesource.com/#/c/400030/ would have taken care of removal of builders.

How can we remove a builder? Is this a manual process?

We need to restart the waterfall to get it to go away entirely I suspect.

We happen to be at a good time, the canaries are just finishing up for the next round at 10 am so we have a window to perform a restart without having significant outages. 

I can go ahead and initiate the pin and restart.

The nyan_freon-paladin builder has been banished from the waterfall. 

The pre-cq and https://chromium-review.googlesource.com/#/c/399908/ is kind of weird though, since the CL removes the COMMIT-QUEUE.ini I would have expected it to get the generic mix of pre-cq builders, however I guess the pre-cq operates off of the prior state of the COMMIT-QUEUE.ini file this is expected and we may need to just chump the CL.

I don't think there is much else we can do in this bug though.

The pre-cq launcher shouldn't crash just because a particular requested build config doesn't exist.

nxia@ can you take a look? Should be a matter of a simple try-catch.

Relevant logs:
@@@STEP_LINK@testing | board-overlays | hshi:399908:d35e6193@https://chromium-review.googlesource.com/#/c/399908/@@@
09:56:19: INFO: Get service account /creds/service_accounts/service-account-chromeos.json
09:56:19: INFO: RunCommand: cbuildbot --remote --timeout 14400 nyan_freon-pre-cq -g 399908 --use-buildbucket in /b/cbuild/internal_master

@@@STEP_FAILURE@@@
09:56:19: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: cbuildbot --remote --timeout 14400 nyan_freon-pre-cq -g 399908 --use-buildbucket
09:56:19: ERROR: No such configuraton target: "nyan_freon-pre-cq".
09:56:19: ERROR: One or more invalid configuration targets specified. You can check the available configs by running `cbuildbot --list --all`

cwd=/b/cbuild/internal_master
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/failures_lib.py", line 172, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1912, in PerformStage
    builder_run=self._run)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1460, in AcquirePool
    if pool.AcquireChanges(gerrit_query, ready_fn, change_filter):
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1360, in AcquireChanges
    self, self.candidates, self.non_manifest_changes)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1833, in ProcessChanges
    self.LaunchTrybot(plan, configs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1476, in LaunchTrybot
    cmd, cwd=self._build_root, capture_output=True)
  File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 624, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: cbuildbot --remote --timeout 14400 nyan_freon-pre-cq -g 399908 --use-buildbucket
09:56:19: ERROR: No such configuraton target: "nyan_freon-pre-cq".
09:56:19: ERROR: One or more invalid configuration targets specified. You can check the available configs by running `cbuildbot --list --all`

Pre-cq-launcher can catch this exception and continue. wonder what's the right step to remove a board-overlay. should first remove the board-overlay from the master, then remove the board config from the config file?

Instead of catching the SystemExit exception, going to check the configs in the pre-cq-launcher, skip the CL the pre-cq-configs do not exist. 

Fixed at https://chromium-review.googlesource.com/#/c/401498/