mmoss@ - It looks like the roll at https://chromium-review.googlesource.com/c/chromium/src/+/1279972 might be related to this ... could you please take a look?

Clicking through that CL (which I have nothing to do with, other than having been auto-TBR'd on it), you can see it only brought in Clank changes. It seems pretty unlikely to be the cause of failures with cros tests.

The first failing build is https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/21564

Which picks up an LKGM bump. Strange that that would have caused this. I'll try to find out how.

Here's the relevant part of the step's stdout:

  2018/10/14 05:45:20 Running ui.ChromeLogin
  2018/10/14 05:45:20 Waiting for org.chromium.SessionManager D-Bus service
  2018/10/14 05:45:20 Restarting ui job
  2018/10/14 05:45:21 Waiting for org.chromium.SessionManager D-Bus service
  2018/10/14 05:45:21 Asking session_manager to enable Chrome testing
  2018/10/14 05:45:21 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
  2018/10/14 05:45:22 Checking cryptohomed service
  2018/10/14 05:45:22 Removing cryptohome for testuser@gmail.com
  2018/10/14 05:45:22 Finding OOBE DevTools target
  2018/10/14 05:45:22 Connecting to Chrome at ws://127.0.0.1:46637/devtools/page/43529482699364135767A4C9B47B6A33
  2018/10/14 05:45:23 Waiting for OOBE
  2018/10/14 05:45:29 Logging in as user "testuser@gmail.com"
  2018/10/14 05:45:29 Waiting for cryptohome for user "testuser@gmail.com"
  2018/10/14 05:45:37 Waiting for OOBE to be dismissed
  2018/10/14 05:45:39 Waiting for SessionStateChanged "started" D-Bus signal from session_manager
  2018/10/14 05:45:39 Got SessionStateChanged signal
  2018/10/14 05:45:39 Creating new page with URL http://127.0.0.1:38725
  2018/10/14 05:45:40 Connecting to Chrome at ws://127.0.0.1:46637/devtools/page/AEB01DD9DD457F59950411D7DE0B4661
  2018/10/14 05:45:41 Got content ""
  2018/10/14 05:45:41 Error: [chrome_login.go:102] Expected page content "Hooray, it worked!", got ""
  2018/10/14 05:45:41 Finished ui.ChromeLogin

This isn't a failure that I've ever seen before, but it looks like it also started on Chrome OS builders in R72-11157.0.0.

Based on the failure and the time when it regressed, I'd guess that it's a race introduced by these changes:

https://crrev.com/c/1278508
https://crrev.com/c/1278509

I'll revert and investigate.

Revert is at https://crrev.com/c/1278509. It'll take a while to make its way to the Chrome CQ, so I'll also explicitly exclude this test there.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f97494fc08ad233b7df9c946dd6cb708b72140c0

commit f97494fc08ad233b7df9c946dd6cb708b72140c0
Author: Dan Erat <derat@chromium.org>
Date: Mon Oct 15 17:31:18 2018

Revert "tast-tests: Use Chrome.NewConn argument to open an initial page."

This reverts commit 990cce828cd4ec6cea4bb1d697f32b2d91aace56.

Reason for revert: Probably causing  https://crbug.com/895337 .

Original change's description:
> tast-tests: Use Chrome.NewConn argument to open an initial page.
> 
> Also adds document.readyState polling to make sure the load is
> complete.
> 
> BUG=None
> TEST=tast run DUT ui.ChromeLogin
> 
> Change-Id: Ie7ef14cf4bab993554c8d9a332beaa971e2e11cb
> Reviewed-on: https://chromium-review.googlesource.com/1278509
> Commit-Ready: Shuhei Takahashi <nya@chromium.org>
> Tested-by: Shuhei Takahashi <nya@chromium.org>
> Reviewed-by: Dan Erat <derat@chromium.org>

BUG= chromium:895337 

Change-Id: Iebdec54a8a423844da034f0bb7ba72f76deddf45
Reviewed-on: https://chromium-review.googlesource.com/c/1280926
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/f97494fc08ad233b7df9c946dd6cb708b72140c0/src/chromiumos/tast/local/bundles/cros/ui/chrome_login.go

I've uploaded https://crrev.com/c/1281050 to disable this on the Chrome side.

Thanks dan! Unfortunately, the sanity test is also failing since it runs ui.ChromeLogin in it as well. It's a bit harder to disable it from that suite, so I'll just revert the latest lkgm bump. That should get us back to using VM images w/o the changes in #4... And hopefully the next time it gets bumped, it'll pick up the revert in #5.

Thanks! I'll abandon the Chrome change from #7, in that case.

lkgm revert at https://chromium-review.googlesource.com/c/chromium/src/+/1281085

CQ looks good and the waterfall bot's rolling green:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/21599

Hopefully the next lkgm bump picks up the revert in #6, otherwise we may have to revert it again.

 Issue 895215  has been merged into this issue.

Sorry for the breakage and thanks for prompt fixes.

Let's close this bug since the flakiness was resolved by the revert.