Bot was added 12 days ago by Ting-Yuan: https://chromium-review.googlesource.com/#/c/422414/

Perhaps the timeout needs to be adjusted.

I believe the bot is not blocking for the PFQ yet.

Setzer Release is also failing with the same build
https://uberchromegw.corp.google.com/i/chromeos/builders/setzer-release/builds/797

This bot seems quite flaky. It failed every one or two days. 

These two build slaves are slow; They never built in time.
https://uberchromegw.corp.google.com/i/chromeos/buildslaves/cros-beefy343-c2
https://uberchromegw.corp.google.com/i/chromeos/buildslaves/build200-m2

Let's increase the timeout:
https://chromium-review.googlesource.com/#/c/435557/

Because the additional USE=clang, chell-chrome-pfq is substantially longer in the two stages:

1. SetupBoard: 36m21s v.s. 2m32s.
2. BuildPackages: 1h45m v.s. 1h8m

The gap in SetupBoard will disappear after USE=clang becomes default. I've been working on a patch to improve 2. but it could still be longer than lumpy-chrome-pfq. 

The specs of build slaves looks identical. The two mentioned in comment 6 only have 32 cores.

memory total: 125.87 GB
os family: Debian
os version: 12.04
processor count: 40
processor type: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
product name: PowerEdge R620

Will the switch to clang affect all boards that much? Or is this because prebuilts aren't available?

I believe that is because the prebuilts are not available.

Which raises the question.... what will the impact of switching to clang be? If not slower, will it make our builds faster?

> I believe that is because the prebuilts are not available.

Is it possible to spin up a paladin to create the necessary pre-builts?
Or does that create problems?

> Which raises the question.... what will the impact of switching to clang be?
> If not slower, will it make our builds faster?

Ideally clang should have an edge than gcc on build time. However, due to the way we configure it ( crbug.com/661019 ), it is slower on start-up. This CL (https://chromium-review.googlesource.com/435472) should fix the problem.

> Is it possible to spin up a paladin to create the necessary pre-builts?

Here is the comparison of BuildPackages on lumpy-chrome-pfq and chell-chrome-pfq. Note that most of the slowdowns of common packages on chell are hundreds seconds compared to couples of seconds on lumpy. It looks like the differences came from prebuilts.

The other non-trivial difference is that chell has more packages.

=========== Common ===========
time_diff lumpy chell package_name
-60.9 78.3 17.4 chromeos-base/autotest-tests-audio-0.0.1
-50.0 78.3 28.3 chromeos-base/autotest-tests-ltp-0.0.1
-44.2 61.0 16.8 chromeos-base/autotest-tests-graphics-0.0.1
-29.2 47.3 18.1 chromeos-base/autotest-deps-glmark2-0.0.1
-27.7 49.5 21.8 chromeos-base/autotest-tests-cellular-0.0.1
-25.2 67.2 42.0 chromeos-base/autotest-tests-security-0.0.1
-20.3 30.5 10.2 chromeos-base/autotest-tests-touchpad-0.0.1
-14.0 39.1 25.1 chromeos-base/autotest-tests-p2p-0.0.1
-10.2 30.4 20.2 chromeos-base/autotest-tests-power-0.0.1
-8.2 10.2 2.0 virtual/target-chromium-os-factory-shim-1
-7.3 21.1 13.8 net-analyzer/tcpdump-4.5.1
-7.2 13.7 6.5 x11-libs/libX11-1.6.0
-6.3 13.0 6.7 chromeos-base/verity-0.0.1
-6.3 11.7 5.4 app-laptop/laptop-mode-tools-1.59
-5.9 7.9 2.0 chromeos-base/chromeos-installshim-0.0.1
-5.9 8.0 2.1 chromeos-base/factory-deps-1
-5.8 8.1 2.3 app-admin/perl-cleaner-2.16
-5.7 12.9 7.2 dev-python/contextlib2-0.4.0
-5.6 11.7 6.1 dev-python/pyxattr-0.5.3
-5.4 13.0 7.6 media-libs/libv4l-1.6.0
-5.2 11.0 5.8 sys-devel/automake-1.13.4
-5.0 12.8 7.8 x11-proto/xf86bigfontproto-1.2.0
-4.9 11.2 6.3 chromeos-base/factory_installer-0.0.1
-4.8 12.4 7.6 app-text/docbook-xml-dtd-4.5
-4.7 9.4 4.7 sys-apps/coreboot-utils-0.0.1
-4.5 6.6 2.1 virtual/target-os-factory-shim-1
-4.4 6.5 2.1 virtual/target-chromium-os-dev-1
-4.3 20.8 16.5 sci-geosciences/gpsd-3.4
-4.1 9.3 5.2 dev-util/ninja-1.7.1
-4.1 9.4 5.3 x11-proto/dri2proto-2.6
-4.1 9.0 4.9 x11-drivers/opengles-headers-0.0.1
-3.9 9.3 5.4 media-libs/ladspa-sdk-1.13
-3.8 9.8 6.0 sys-devel/flex-2.5.39
-3.7 7.1 3.4 chromeos-base/shill-test-scripts-0.0.1
-3.7 12.7 9.0 app-misc/tmux-1.9a
-3.6 13.3 9.7 chromeos-base/touchbot-1.0
-3.6 12.5 8.9 virtual/update-policy-1
-3.4 11.0 7.6 x11-misc/read-edid-1.4.2
..........
71.9 2.8 74.7 net-libs/libmbim-1.13.0
75.0 12.0 87.0 chromeos-base/factory-mini-0.0.1
79.6 23.2 102.8 media-libs/dlm-0.0.1
81.1 8.7 89.8 sys-boot/chromeos-bmpblk-1.0.1
84.6 9.9 94.5 net-print/hplip-3.16.3
89.8 3.2 93.0 x11-libs/libdrm-2.4.70
96.4 14.4 110.8 chromeos-base/pepper-flash-24.0.0.201
98.4 2.7 101.1 chromeos-base/libbrillo-0.0.1
103.5 20.4 123.9 chromeos-base/autotest-server-tests-0.0.2
104.2 2.6 106.8 chromeos-base/quipper-0.0.1
105.9 9.3 115.2 x11-apps/intel-gpu-tools-1.15
127.4 2.2 129.6 chromeos-base/chromeos-fonts-0.0.1
127.8 6.7 134.5 net-misc/modemmanager-next-1.5.0
129.3 10.7 140.0 chromeos-base/chromeos-assets-0.0.2
144.7 2.8 147.5 chromeos-base/google-breakpad-2016.12.12.161622
168.0 13.7 181.7 sys-boot/coreboot-0.0.1
170.8 2.3 173.1 dev-libs/dbus-c++-0.0.2
177.0 8.3 185.3 app-crypt/trousers-0.3.3
178.1 8.6 186.7 chromeos-base/autotest-0.0.2
180.3 7.2 187.5 sys-fs/udev-225
181.2 7.7 188.9 media-libs/libmtp-0.0.1
183.8 4.0 187.8 chromeos-base/libscrypt-1.1.6
196.1 3.2 199.3 media-libs/jbig2dec-0.11
204.7 23.1 227.8 dev-libs/boost-1.55.0
211.1 2.5 213.6 sys-process/psmisc-22.20
223.0 278.4 501.4 net-misc/strongswan-5.5.0
231.1 6.9 238.0 net-misc/tlsdate-0.0.5
232.2 5.7 237.9 net-print/cups-filters-1.8.2
269.5 2.2 271.7 sys-apps/upstart-1.2
272.8 2.6 275.4 sys-process/procps-3.3.9
316.0 4.2 320.2 media-libs/mesa-17.1.0
341.5 9.0 350.5 sys-boot/chromeos-mrc-0.0.1
435.3 2.7 438.0 sys-apps/findutils-4.4.2
435.8 10.7 446.5 media-gfx/deqp-2017.01.20
446.5 10.1 456.6 media-gfx/imagemagick-6.7.8.7
692.6 3.0 695.6 app-text/ghostscript-gpl-9.19
867.5 3.7 871.2 app-arch/tar-1.29
915.2 5.5 920.7 sys-apps/coreutils-8.24
1143.6 37.8 1181.4 sys-devel/llvm-3.8.1

=========== lumpy only ===========
.....................
10.0 virtual/chromeos-board-default-apps-1
10.3 dev-cpp/gmock32-1.7.0
17.4 sys-apps/daisydog-0.0.1
56.5 chromeos-base/autotest-tests-wimax-0.0.1
145.8 chromeos-base/crosh-extension-0.8.35.0
197.9 sys-kernel/chromeos-kernel-3_8-3.8.11
3570.1 chromeos-base/chromeos-chrome-58.0.2999.0_rc

=========== chell only ===========
30.6 virtual/lpe-support-2
31.4 chromeos-base/chromeos-bsp-chipset-skl-private-0.0.1
32.9 virtual/chromeos-board-default-apps-3
34.8 chromeos-base/autotest-deps-candycrush-0.1
35.5 sys-libs/libsepol-2.4
35.7 chromeos-base/ec-utils-test-0.0.1
36.5 sys-libs/libselinux-2.4
36.7 chromeos-base/autotest-deps-antutu-6.1.4
37.0 chromeos-base/arc-camera-service-0.0.1
37.8 sys-boot/nhlt-blobs-0.0.1
38.0 media-libs/arc-cros-gralloc-0.0.1
40.1 chromeos-base/arc-networkd-0.0.1
44.9 chromeos-base/arc-codec-chipset-skl-0.0.1
45.6 chromeos-base/chromeos-firmware-chell-0.0.1
46.2 virtual/arc-codec-1.5
46.2 virtual/awk-1
50.0 sys-apps/restorecon-2.4
51.3 sys-boot/chromeos-bootimage-0.0.2
54.0 media-libs/adi-spro-support-0.0.1
57.7 chromeos-base/autotest-deps-powerloadtest-1.1
60.0 sys-boot/chromeos-seabios-0.0.1
67.2 chromeos-base/libmojo-395517
67.8 sys-boot/libpayload-0.0.1
68.7 sys-boot/depthcharge-0.0.1
70.4 media-libs/skl-hotword-support-0.0.1
73.4 chromeos-base/chromeos-cheets-scripts-0.0.1
77.9 chromeos-base/chromeos-ec-0.0.1
93.1 net-misc/bridge-utils-1.5
93.3 dev-util/android-tools-5.1.1_p13
103.2 sys-boot/coreboot-private-files-baseboard-glados-0.0.1
112.4 net-libs/libndp-1.6
115.1 chromeos-base/android-container-3688587
140.3 sys-power/dptf-8.2.11202
189.7 chromeos-base/autotest-deps-gfxbench-4.0.10
209.5 dev-libs/libconfig-1.5
285.4 sys-kernel/chromeos-kernel-3_18-3.18
678.5 media-libs/arc-mesa-12.1.0_pre
694.3 app-arch/cpio-2.11
4181.9 chromeos-base/chromeos-chrome-58.0.2998.0_rc

It looks like to me that the difference comes from prebuilts and will automatically disappear in a few days after making clang default. Shall we increase the timeouts for the transition and tune them back later? It's probably safer to increase the timeouts for all builders.

These are 5 rounds of emerge-chell chromeos-chrome. With CL435557, clang is 20% faster than gcc on a machine with 48 cores.

GCC                 CLANG

real  59m6.884s     real  46m41.878s
user  1725m55.205s  user  1381m26.155s
sys   236m10.588s   sys   119m3.234s

real  59m35.396s    real  47m36.645s
user  1724m27.779s  user  1381m59.670s
sys   230m36.910s   sys   119m56.977s

real  59m4.086s     real  46m41.123s
user  1725m24.401s  user  1383m4.388s
sys   234m45.593s   sys   118m53.745s

real  58m59.861s    real  47m7.542s
user  1723m2.621s   user  1382m26.704s   
sys   228m44.352s   sys   119m32.277s

real  60m22.226s    real  47m1.475s
user  1724m35.345s  user  1383m46.556s
sys   235m25.407s   sys   119m22.441s

Sorry, the CL is https://chromium-review.googlesource.com/#/c/435472/

I'm closing this since chell-chrome-pfq hasn't timed out for a week.