Another possible problem is mojo. 

The first failed build: 
https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%281%29/builds/29817

Suspicious CLs:
* Mojo C++ bindings: switch WebKit mojom targets to use STL/WTF types (https://codereview.chromium.org/2522553003)
* Make indexed_db.mojom.BlobInfo.size a signed integer. (https://codereview.chromium.org/2517343002)
* Introduce Display Compositor mojo interface. Use InProcessContextProvider. (https://codereview.chromium.org/2481263002)

Based on logs, it looks like Mojo C++ bindings: switch WebKit mojom targets to use STL/WTF types (https://codereview.chromium.org/2522553003) is the culprit. 

  Providing:
    capability: gpu containing interfaces:
      device::mojom::PowerMonitor
    capability: plugin containing interfaces:
      device::mojom::PowerMonitor
    capability: renderer containing interfaces:
      blink::mojom::BackgroundSyncService
      blink::mojom::BroadcastChannelProvider
      blink::mojom::BudgetService
      blink::mojom::Hyphenation
      blink::mojom::MimeRegistry
      blink::mojom::NotificationService
      blink::mojom::OffscreenCanvasCompositorFrameSinkProvider
      blink::mojom::OffscreenCanvasSurface
      blink::mojom::PermissionService
      blink::mojom::ShapeDetection
      blink::mojom::WebSocket
      content::mojom::MemoryCoordinatorHandle
      content::mojom::ServiceWorkerDispatcherHost
      content::mojom::StoragePartitionService
      content::mojom::URLLoaderFactory
      content::mojom::VideoCaptureHost
      device::BatteryMonitor
      device::mojom::LightSensor
      device::mojom::MotionSensor
      device::mojom::OrientationSensor
      device::mojom::PowerMonitor
      device::mojom::TimeZoneMonitor
      media::mojom::ImageCapture
      memory_coordinator::mojom::MemoryCoordinatorHandle
      payments::mojom::PaymentAppManager
    capability: service_manager:service_factory containing interfaces:
      service_manager::mojom::ServiceFactory


I will try to revert the CL.

The revert is impossible because of conflicts. 

https://codereview.chromium.org/2525243003/

yzshen@: Please take a look ASAP.

Status update:

It seems this exception happened before my CL:
For example:
https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%281%29/builds/29763/steps/mash_browser_tests/logs/stdio

And the log doesn't seems related to my CL.

But I will keep looking....

Ohh I see the problem. SubmitCompositorFrame's SurfaceFactory has a base::Unretained callback to GpuCompositorFrameSink::DidReceiveCompositorFrameAck. I'll fix it.

Woohoo! Thanks Fady! :)

https://codereview.chromium.org/2527203002/ CL in CQ.

fsamuel@: Great thanks! This old problem seemed unsolvable. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/029f79a574a4e032561ac44a72746c95814487dc

commit 029f79a574a4e032561ac44a72746c95814487dc
Author: fsamuel <fsamuel@chromium.org>
Date: Thu Nov 24 16:38:09 2016

Mus: GpuCompositorFrameSink uses WeakPtrFactory

A Surface can outlive the GpuCompositorFrameSink that created it.
Submitting a CompositorFrame to a surface means that the draw callback
could get called after GpuCompositorFrameSink is destroyed. By binding
the draw callback to a weak ptr we ensure we don't call it if
GpuCompositorFrameSink has been destroyed.

BUG= 668420 
TBR=yzshen@chromium.org, sadrul@chromium.org

Review-Url: https://codereview.chromium.org/2527203002
Cr-Commit-Position: refs/heads/master@{#434347}

[modify] https://crrev.com/029f79a574a4e032561ac44a72746c95814487dc/services/ui/surfaces/gpu_compositor_frame_sink.cc
[modify] https://crrev.com/029f79a574a4e032561ac44a72746c95814487dc/services/ui/surfaces/gpu_compositor_frame_sink.h

Linux ChromiumOS Tests (1) is still backed up with jobs waiting for 24 hours. r433783 (in  http://crbug.com/666481#c19 ) added a timeout, but it seems not to timeout..

Maybe if we can get the bot past r434347 it will go somewhere. But it's currently stuck at r434272.

It has 1 buildslave, which says it's offline - "vm911-m1 offline".

Deleted: Screen Shot 2016-11-25 at 9.35.55 am.png 586 KB

	Screen Shot 2016-11-25 at 9.35.55 am.png 586 KB View Download

Looks like the builder has processed the backlog over the weekend.