I could not repro with the internal simple chrome build and sentry-release/R60-9589.0.0.

I will install 9592 now.

No repro either with 9592. It looks like caroline is turning red right now.

Setting up a lumpy now.

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/lumpy-tot-chrome-pfq-informational/builds/22625

Lumpy reproduces already on stock lumpy-release/R60-9591.0, even though the informational builder was green with 9591.

The first few times after a boot the test passes. Wait a little, and the test will start failing (unable to login, Chrome goes back to OOBE) each time until next reboot. Chances are this problem was in the build for a while, even though wmatrix says it is green
https://wmatrix.googleplex.com/unfiltered?hide_missing=True&releases=tot&tests=login_CryptohomeIncognito

20:02:09 INFO | Websocket at port 23 not yet available: [Errno 111] Connection refused
20:02:14 INFO | Websocket at port 23 not yet available: [Errno 111] Connection refused
[...]
20:11:15 INFO | Invoking Oobe.guestLoginForTesting

Letting it idle for a few minutes we never see the "Invoking Oobe.guestLoginForTesting" spew.

I think this is coming from here
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/oobe.py?type=cs&q=guestLoginForTesting+package:%5Echromium$&l=55

Identical incognito login problems with security_ProfilePermissions

Seems to be connected to ui spew
[5097:5164:0526/212939.233244:ERROR:socket_posix.cc(137)] bind() returned an error, errno=13: Permission denied
[5097:5164:0526/212939.233360:ERROR:socket_posix.cc(137)] bind() returned an error, errno=13: Permission denied
[5097:5164:0526/212939.233398:ERROR:devtools_http_handler.cc(226)] Cannot start http server for devtools. Stop devtools.

Parameters
ERROR:socket_posix.cc(137)] bind() returned an error, errno=13, socket_fd_=107, addr=0x7f8c9366f508, len=16: Permission denied
ERROR:socket_posix.cc(137)] bind() returned an error, errno=13, socket_fd_=107, addr=0x7f8c9366f508, len=28: Permission denied
ERROR:socket_posix.cc(137)] bind() returned an error, errno=13, socket_fd_=112, addr=0x7f7c245e9508, len=16: Permission denied
ERROR:socket_posix.cc(137)] bind() returned an error, errno=13, socket_fd_=112, addr=0x7f7c245e9508, len=28: Permission denied

I assume the problem started in the VM with the telemetry roll below. But it is likely it just exposed an existing problem.
https://chromium-review.googlesource.com/517285
https://codereview.chromium.org/2904193002

lumpy-release/R60-9500.0.0 is solid, not problems. So this must be a more recent regression. I find the telemetry patch series in  issue 720317  interesting. Trying newer images.

lumpy-release/R60-9550.0.0 is good.

I think there are 2 regressions:
1)  Making VMTest fail immediately between 9591.0.0 and 9592.0.0 (plus new Chrome) as discussed in #9.
2) Making test on lumpy fail when waiting a few minutes with pure release images between 9589.0.0 (probably good) and 9590.0.0 (verified bad).

You need to find way to put cros test on chromium CQ, otherwise we just keep chasing fires here.

Welp. Where Ned is right he is right.

I need to pass this to the next gardener. I know next to nothing about login.

I'm inclined to revert [1] mid-day tomorrow (even if it isn't the root cause) simply to get the tree green, especially given that the patch is a small code cleanup. Reverting locally fixes the failing tests. I haven't yet been able to determine the cause of the failures.

1: https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+/02e58de92d78a5d8455a5973ef43b6e7c3cca48b

Ok to revert. But please help me find which assumption in CrOS test that CL is breaking so I can reland it. Reducing the overhead between Telemetry & browser in general is good for perf testing.

Revert didn't apply cleanly so I put together a small patch that does the same thing at [1].

1: https://codereview.chromium.org/2913213002

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e3db9988b2c7e8b41f75b9d326edad68cdb0b184

commit e3db9988b2c7e8b41f75b9d326edad68cdb0b184
Author: jdufault <jdufault@chromium.org>
Date: Wed May 31 23:18:44 2017

Roll src/third_party/catapult/ 24245dde6..e6ad06342 (3 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/24245dde6fc8..e6ad06342efb

$ git log 24245dde6..e6ad06342 --date=short --no-merges --format='%ad %ae %s'
2017-05-31 dtu [pinpoint] Move Change.Validate() into Change.FromDict().
2017-05-31 jdufault cros: Fix bots by using a different websocket to get system info.
2017-05-31 rnephew [Telemetry] Pass browser finder options to story expectations methods to determine disabling.

Created with:
  roll-dep src/third_party/catapult

TBR=nednguyen@chromium.org
BUG=726917

Review-Url: https://codereview.chromium.org/2920583002
Cr-Commit-Position: refs/heads/master@{#476085}

[modify] https://crrev.com/e3db9988b2c7e8b41f75b9d326edad68cdb0b184/DEPS

Ned - steps to reproduce this bug:

Launch a cros VM using the instructions in https://go/cros-vm

From your catapult checkout:
telemetry/bin/run_tests --browser=cros-chrome-guest --remote=localhost --remote-ssh-port=9222 testBrowserCreation

I will wait for the test to green again before working on validating this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f471b73c30b2b9bc7f2684c850e30299d0abc366

commit f471b73c30b2b9bc7f2684c850e30299d0abc366
Author: Jacob Dufault <jdufault@google.com>
Date: Wed May 31 23:55:47 2017

Roll src/third_party/catapult/ 24245dde6..e6ad06342 (3 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/24245dde6fc8..e6ad06342efb

$ git log 24245dde6..e6ad06342 --date=short --no-merges --format='%ad %ae %s'
2017-05-31 dtu [pinpoint] Move Change.Validate() into Change.FromDict().
2017-05-31 jdufault cros: Fix bots by using a different websocket to get system info.
2017-05-31 rnephew [Telemetry] Pass browser finder options to story expectations methods to determine disabling.

Created with:
  roll-dep src/third_party/catapult

TBR=nednguyen@chromium.org
BUG=726917

Review-Url: https://codereview.chromium.org/2920583002
Cr-Original-Commit-Position: refs/heads/master@{#476085}
Review-Url: https://codereview.chromium.org/2921533002 .
Cr-Commit-Position: refs/branch-heads/3116@{#3}
Cr-Branched-From: 64a9be37ccf1e29e9e9cc302da2297f2d3661b5d-refs/heads/master@{#475721}

[modify] https://crrev.com/f471b73c30b2b9bc7f2684c850e30299d0abc366/DEPS

Tests are now green - passing to nednguyen@ to eventually re-land.