we are planning a compiler roll for next week. This will block the roll. Please diagnose as soon as posible. 

We can move it to a beefier machine.

It's currently running on GCE.

What is known to work VM, or Baremetal?

i don't think we ran into timeouts before it was on GCE, but it's hard to say that's the cause.  it is highly suspicious :).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e1cb8dda3e17de8d47cf90e8ba44e60d7d0c4ade

commit e1cb8dda3e17de8d47cf90e8ba44e60d7d0c4ade
Author: Don Garrett <dgarrett@google.com>
Date: Tue Mar 15 23:42:02 2016

chromiumos-sdk: Switch to a physical builder.

This build config is having performance problems, so switch it to a
baremetal builder to see if things improve.

BUG= chromium:595127 
TEST=run_tests

Change-Id: I8f7adf187cc574ccb0b636dffcfa6c8074287419
Reviewed-on: https://chromium-review.googlesource.com/332277
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[modify] https://crrev.com/e1cb8dda3e17de8d47cf90e8ba44e60d7d0c4ade/cbuildbot/config_dump.json
[modify] https://crrev.com/e1cb8dda3e17de8d47cf90e8ba44e60d7d0c4ade/cbuildbot/chromeos_config.py

I don't know if the problem is solved, but a physical builder is being used. The first build on the new machine is now in progress.

Moving back to metal seems to have fixed this.

We can try moving it back to GCE later, but if we do, we'll use a GCE_BEEFY instance.

thanks!

yeah, as wrong as it feels to say, we're "just" back down to like 11hrs per run now

Bulk verified