The x86 PFQ builders are:

x86-alex-tot-chrome-pfq-informational
x86-generic-chromium-pfq

If these do not have chrome-os or chromium-os paladin coverage, we should definitely remove them from the PFQ.

Also:

x86-generic-tot-chromium-pfq-informational

(But oddly that builder is not currently failing due to issue 729608)

just to be clear, killing 32-bit boards in the waterfall doesn't mean we want to kill all the cbuildbot/trybot configs

Ah.... I thought we did. Mostly so that toolchain support could be shutdown.

toolchain support can't be shutdown because we still have USE=32bit_au boards, and i'm pretty sure our firmware builds use them too.  look at all the i686 tuples in toolchain.conf files.

it seems we still have some 32-bit uses for the toolchain. I will start a conversation about this because we are currently not testing it.

and just to be clear. From the toolchain team point of view, we don't care about those builders and you can remove them.

Killed x86-alex-tot-chrome-pfq-informational here:

https://chrome-internal-review.googlesource.com/c/392889/

Is this now fixed?

ChromeOS Infra P1 Bugscrub.

P1 Bugs in this component should be important enough to get weekly status updates.

Is this already fixed?  -> Fixed
Is this no longer relevant? -> Archived or WontFix
Is this not a P1, based on go/chromeos-infra-bug-slo rubric? -> lower priority.
Is this a Feature Request rather than a bug? Type -> Feature
Is this missing important information or scope needed to decide how to proceed? -> Ask question on bug, possibly reassign.
Does this bug have the wrong owner? -> reassign.

Bugs that remain in this state next week will be downgraded to P2.

Not yet: 

https://build.chromium.org/p/chromiumos.chromium/waterfall:
x86-generic-tot-chromium-pfq-informational

https://build.chromium.org/p/chromiumos/waterfall:
x86-generic-paladin
x86-generic-incremental
x86-generic-full

Issue 748176 has been merged into this issue.

x86-generic-tot-chromium-pfq-informational is failing and generated bugs. What can I do to help get rid of it?

Issue 748225 has been merged into this issue.

Issue 748241 has been merged into this issue.

> x86-generic-tot-chromium-pfq-informational is failing and generated bugs. What can I do to help get rid of it?

Hmmm...  According to cbuildbot/waterfall_layout_dump.txt in the
chromite sources, that builder isn't on the waterfall.  If I've
read it right, that means when we restart the builder this evening
(see bug 747506), then it'll be gone some time this evening.

That said, there are various other x86-generic builders still on
the waterfall (mostly tryservers):

    $ egrep '(x86-generic|==)' cbuildbot/waterfall_layout_dump.txt
    == chromeos ==
    == chromeos.infra ==
    == chromeos_release ==
    == chromiumos ==
    x86-generic-full (baremetal)
    x86-generic-incremental (baremetal)
    == chromiumos.tryserver ==
    x86-generic-compile-only-pre-cq (gce_beefy)
    x86-generic-no-vmtest-pre-cq (gce_beefy)
    x86-generic-pre-cq (baremetal)
    x86-generic-unittest-stress (gce_beefy)

Issue 748237 has been merged into this issue.

That dump file is only reliable for the waterfalls that are configured from chromite (yes, every waterfall is special in it's own way). The Chrome informational waterfalls are all hand configured in buildbot, and so not in the dump file.

Looks like that build is failing because it's assigned buildslave is offline. "build252-m2". A bug to the deputy and/or trooper can either get it repaired, or removed from the waterfall, as you prefer.

We want it removed. We want all 32 bit boards removed, isn't that what this issue is tracking?

Paul, as deputy, this routes to you.

so, remove x86-generic-* builders from the chromite config.py file?

> so, remove x86-generic-* builders from the chromite config.py file?

Pretty much.  Also help orchestrate a waterfall restart as necessary.

Probably also help clean up unexpected failures.  :-)

 Issue 748708  has been merged into this issue.

For the informational waterfall, you have to remove them from the buildbot waterfall configuration, not chromite. Troopers are the first stop for this.

 Issue 746448  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7358a14e362f0070b87bfe99db6479aaf42723e0

commit 7358a14e362f0070b87bfe99db6479aaf42723e0
Author: Paul Hobbs <phobbs@google.com>
Date: Mon Aug 14 11:01:58 2017

chromeos_config: remove x86-generic-*

Remove all x86-generic-* boards.

BUG= chromium:729645 
TEST=chromeos_config_unittest

Change-Id: I83bf8130a4571113126a48156a805ab15c01aacf
Reviewed-on: https://chromium-review.googlesource.com/588081
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/build_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/artifact_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/test_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/report_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/lkgm_manager_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/manifest_version_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/lib/results_lib_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/lib/tree_status_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/remote_try_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/lib/const/waterfall.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/builders/simple_builders_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/sync_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/chromeos_config_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/completion_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/stages/generic_stages_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/cbuildbot_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/lib/config_lib_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/scripts/cbuildbot_unittest.py
[modify] https://crrev.com/7358a14e362f0070b87bfe99db6479aaf42723e0/cbuildbot/commands_unittest.py

Checking in; the following builders are still present:

https://build.chromium.org/p/chromiumos/waterfall:
x86-generic-incremental
x86-generic-full

We need a waterfall restart to propagate this.

Does this also include updating the pre-CQ, or should I file a separate bug for that? https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/627665 failed because x86-mario-pre-cq, x86-alex-pre-cq, and x86-zgb-pre-cq are running and don't have a chromeos-chrome prebuilt.

sep bug.  removing active builders from the waterfall isn't the same as deleting their cbuildbot configs for use with trybots (which is basically all the precq is).

I think in the comment 36 case, the solution is to remove those deprecated overlays, along with their COMMIT-QUEUE.ini files, given they don't appear to build properly now.

Re #37: Posted https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/629420

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/ce307916022c33baf10b69e0ebf12f6d902ac70a

commit ce307916022c33baf10b69e0ebf12f6d902ac70a
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Aug 23 19:44:54 2017

Drop x86-* pre-cq triggers.

BUG= chromium:729645 
TEST=None

Change-Id: I7c8a63c493f7ea87393063a5a254da29ed8b6b02
Reviewed-on: https://chromium-review.googlesource.com/629420
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>

[delete] https://crrev.com/9a04ea577f803339f9bef24c10cbd468e7466427/overlay-x86-alex/COMMIT-QUEUE.ini
[delete] https://crrev.com/9a04ea577f803339f9bef24c10cbd468e7466427/overlay-x86-mario/COMMIT-QUEUE.ini
[delete] https://crrev.com/9a04ea577f803339f9bef24c10cbd468e7466427/overlay-x86-zgb/COMMIT-QUEUE.ini
[delete] https://crrev.com/9a04ea577f803339f9bef24c10cbd468e7466427/overlay-x86-generic/COMMIT-QUEUE.ini

More removals, hooray! However there are still two x86 builders in this waterfall:

https://build.chromium.org/p/chromiumos/waterfall
x86-generic-incremental
x86-generic-full

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/744557375c5aec0877eae0cdfefd4676de53618a

commit 744557375c5aec0877eae0cdfefd4676de53618a
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Sep 15 03:14:11 2017

chromeos_config: Remove x86-generic-incremental

Removed x86-generic-incremental builder, and x86-generic-full from the waterfall
external list.

BUG= chromium:729645 
TEST=chromeos_config_unittest

Change-Id: Id74134da73c8fe3b17c15e38e82c232df1fd0367
Reviewed-on: https://chromium-review.googlesource.com/666235
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/744557375c5aec0877eae0cdfefd4676de53618a/cbuildbot/config_dump.json
[modify] https://crrev.com/744557375c5aec0877eae0cdfefd4676de53618a/cbuildbot/waterfall_layout_dump.txt
[modify] https://crrev.com/744557375c5aec0877eae0cdfefd4676de53618a/cbuildbot/chromeos_config.py

What's left here? A waterfall restart?

Yep. In this case, the restart is cosmetic (removes the two red build columns).

No x86-* left on the waterfall!