@ned, do you have an idea how telemetry is managed on these bots? It behaves as if telemetry was not rolled...

achuith@ is the one who know how Telemetry is managed in cros infra

@achuith: I recently changed the way telemetry attaches to the /devtools/browser web socket, it is no longer discoverable at /devtools/browser and the client needs to know exact browser target id in order to get engaged for debugging at /devtools/browser/<id>.

It looks like I should have changed something in cros_browser_backend.py to make it read browser target id first as it does for chrome. Or I could have made /devtools/browser unconditionally discoverable when in cros (make it work the old way). Could you look at my original change in telemetry at https://codereview.chromium.org/2989413002 and recomment a cros-specific fix?

 Issue 753104  has been merged into this issue.

I went ahead and made things work the old way for CrOS at https://chromium-review.googlesource.com/c/604734. But it is important that remote debugging is never enabled in production / outside of build bots or controlled environment.

Pavel has a fix here: https://codereview.chromium.org/2997603002/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f05cf3bcebea7a5b5815337ba39e427d76dd03f7

commit f05cf3bcebea7a5b5815337ba39e427d76dd03f7
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Wed Aug 09 06:59:41 2017

Roll src/third_party/catapult/ 49fffef98..4dd91f2ea (8 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/49fffef9824d..4dd91f2eab39

$ git log 49fffef98..4dd91f2ea --date=short --no-merges --format='%ad %ae %s'
2017-08-08 phsilva Revert of Revision Info into GenericSet (patchset #3 id:40001 of https://codereview.chromium.org/2990293002/ )
2017-08-08 phsilva Revert of Plumb SparseDiagnostics by Name (patchset #7 id:120001 of https://codereview.chromium.org/2989143002/ )
2017-08-08 pfeldman Fix telemetry for CrOS with non-default browser target name
2017-08-08 phsilva Plumb SparseDiagnostics by Name
2017-08-08 phsilva Revision Info into GenericSet
2017-08-08 phsilva Update ownership information in the bug report
2017-08-08 loloangela Fix errors related to global-statement
2017-08-08 simonhatch Pinpoint - Reset jobInfo when navigating away

Created with:
  roll-dep src/third_party/catapult
BUG= 753116 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I3343d5da9c3f5497f5e6d91a6daffde3f14cd697
Reviewed-on: https://chromium-review.googlesource.com/607400
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#492873}
[modify] https://crrev.com/f05cf3bcebea7a5b5815337ba39e427d76dd03f7/DEPS

Chrome PFQ is dead for 2 days due to "Timed out while waiting 240s for HasBrowserFinishedLaunching.":
https://uberchromegw.corp.google.com/i/chromeos/builders/tricky-chrome-pfq

Does it share the same cause as this one?

I believe this is mostly fixed, there's a lingering issue with the incognito browser: https://bugs.chromium.org/p/chromium/issues/detail?id=753876

Kazuhiro-san - the cause is this bug, yes.

Issue 753725 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd76a42ad31b5d436dd8523e79ed942cef4edeeb

commit cd76a42ad31b5d436dd8523e79ed942cef4edeeb
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Thu Aug 10 02:04:26 2017

Roll src/third_party/catapult/ 656df4e2f..bd05965e4 (4 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/656df4e2fcc8..bd05965e4557

$ git log 656df4e2f..bd05965e4 --date=short --no-merges --format='%ad %ae %s'
2017-08-09 pfeldman Re-create devtools client upon browser restart (cros guest session did a browser restart without recreating the dev tools backend).
2017-08-09 benjhayden TBMv2: Add HistogramGroupings for DeviceInfo.
2017-08-09 phsilva Revision Info into GenericSet
2017-08-09 htwiggsmith Implement get_diagnostics

Created with:
  roll-dep src/third_party/catapult
BUG= 753116 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: Ie04a1e72ca4f0f2ad3a37415c069b19e372aa7ea
Reviewed-on: https://chromium-review.googlesource.com/609483
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#493224}
[modify] https://crrev.com/cd76a42ad31b5d436dd8523e79ed942cef4edeeb/DEPS

 Issue 753070  has been merged into this issue.

I'm still seeing the 240s timeout on my caroline with Chrome r493501.

login_OwnershipTaken and cheets_ContainerSmokeTest succeed with xbuddy://remote/caroline/R62-9831.0.0/test (the latest Chrome OS image which uses Chrome r491883 according to goldeneye), but once I deploy self-built Chrome r493501 (yesterday's tot, no local mods), they always fail with the following timeout:

---------------------------------------------------------------------------------------------
/tmp/test_that_results_NQIjU_/results-1-login_OwnershipTaken                      [  FAILED  ]
/tmp/test_that_results_NQIjU_/results-1-login_OwnershipTaken                        FAIL: Unhandled BrowserConnectionGoneException: Timed out while waiting 240s for HasBrowserFinishedLaunching.
/tmp/test_that_results_NQIjU_/results-1-login_OwnershipTaken/login_OwnershipTaken [  FAILED  ]
/tmp/test_that_results_NQIjU_/results-1-login_OwnershipTaken/login_OwnershipTaken   FAIL: Unhandled BrowserConnectionGoneException: Timed out while waiting 240s for HasBrowserFinishedLaunching.
/tmp/test_that_results_NQIjU_/results-1-login_OwnershipTaken/login_OwnershipTaken   08/11 14:45:53.888 ERROR|           browser:0068| Failed with BrowserConnectionGoneException while starting the browser backend.
---------------------------------------------------------------------------------------------
Total PASS: 0/2 (0%)

-------------------------------------------------------------------------------------------------------
/tmp/test_that_results_AjLIx3/results-1-cheets_ContainerSmokeTest                           [  FAILED  ]
/tmp/test_that_results_AjLIx3/results-1-cheets_ContainerSmokeTest                             ERROR: Unhandled BrowserConnectionGoneException: Timed out while waiting 240s for HasBrowserFinishedLaunching.
/tmp/test_that_results_AjLIx3/results-1-cheets_ContainerSmokeTest/cheets_ContainerSmokeTest [  FAILED  ]
/tmp/test_that_results_AjLIx3/results-1-cheets_ContainerSmokeTest/cheets_ContainerSmokeTest   ERROR: Unhandled BrowserConnectionGoneException: Timed out while waiting 240s for HasBrowserFinishedLaunching.
/tmp/test_that_results_AjLIx3/results-1-cheets_ContainerSmokeTest/cheets_ContainerSmokeTest   08/11 13:00:36.041 ERROR|           browser:0068| Failed with BrowserConnectionGoneException while starting the browser backend.
-------------------------------------------------------------------------------------------------------
Total PASS: 0/2 (0%)

Running gclient sync again (just to make sure my third_party/catapult/ is up to date) before building Chrome didn't fix the issue. Can I reopen this?

I'm seeing connection refused in the logs now (ie, https://luci-milo.appspot.com/buildbot/chromeos.chrome/peach_pit-tot-chrome-pfq-informational/7317):

/devtools/browser/a28c2390-f6a3-49be-8550-240782b67bb0
08/10 20:10:05.092 INFO |cros_browser_backe:0075| Discovered ephemeral port 53523
08/10 20:10:05.096 INFO |cros_browser_backe:0076| Browser target: /devtools/browser/a28c2390-f6a3-49be-8550-240782b67bb0
08/10 20:10:05.101 INFO |devtools_client_ba:0062| Websocket at port 53523 not yet available: [Errno 111] Connection refused
08/10 20:10:10.197 DEBUG|    cros_interface:0363| ListProcesses(<predicate>)->[200 processes]
08/10 20:10:10.200 ERROR|           browser:0068| Failed with BrowserConnectionGoneException while starting the browser backend.

Jake - the bots appear to be green at the moment. Is this still an issue?

The bot linked in #17 is still failing.

I think this is a different issue, with chrome starting under mash. Looks like chrome is crashing:
https://storage.cloud.google.com/chromeos-autotest-results/134279479-chromeos-test/chromeos6-row2-rack10-host6/desktopui_MashLogin/sysinfo/var/log/messages?_ga=1.46805856.655977216.1501535104


2017-08-11T03:06:04.091261+00:00 INFO session_manager[11954]: [INFO:browser_job.cc(140)] Running child /opt/google/chrome/chrome --ppapi-flash-path=/opt/google/chrome/pepper/libpepflashplayer.so --ppapi-flash-version=26.0.0.137 --ui-prioritize-in-gpu-process --use-gl=egl --gpu-sandbox-failures-fatal=yes --gpu-sandbox-start-early --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --max-unused-resource-memory-usage-percentage=5 --system-developer-mode --login-profile=user --has-chromeos-keyboard --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/default_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/default_small.jpg --child-wallpaper-large=/usr/share/chromeos-assets/wallpaper/child_large.jpg --child-wallpaper-small=/usr/share/chromeos-assets/wallpaper/child_small.jpg --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --enable-consumer-kiosk --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --login-manager --gpu-no-complete-info-collection --mash --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --enable-gpu-benchmarking --disable-background-networking --disable-component-extensions-with-background-pages --disable-default-apps --disable-search-geolocation-disclosure --enable-smooth-scrolling --enable-threaded-compositing --remote-debugging-port=0 --start-maximized --ash-disable-system-sounds --allow-failed-policy-fetch-for-test --oobe-skip-postlogin --disable-logging-redirect --disable-gaia-services --testing-channel=NamedTestingInterface:/tmp/.org.chromium.Chromium.Iwu1Tc/.org.chromium.Chromium.CWm4Df --vmodule=tablet_power_button_controller=1,*chromeos/login/*=1,auto_enrollment_controller=1,*plugin*=2,*zygote*=1,*/ui/ozone/*=1,*/ui/display/manager/chromeos/*=1,power_button_observer=2,webui_login_view=2,lock_state_controller=2,webui_screen_locker=2,screen_locker=2,*/chromeos/net/*=2,*/chromeos/login/*=2,chrome_browser_main_posix=2 
2017-08-11T03:06:04.092961+00:00 INFO session_manager[11954]: [INFO:session_manager_service.cc(217)] Browser is 12024
2017-08-11T03:06:05.480137+00:00 WARNING crash_reporter[12190]: Could not load the device policy file.
2017-08-11T03:06:05.480395+00:00 WARNING crash_reporter[12190]: [user] Received crash notification for chrome[12036] sig 11, user 1000 (developer build - not testing - always dumping)
2017-08-11T03:06:05.484015+00:00 INFO crash_reporter[12190]: State of crashed process [12036]: S (sleeping)
2017-08-11T03:06:05.486975+00:00 INFO metrics_daemon[1887]: [INFO:metrics_daemon.cc(404)] Got org.chromium.CrashReporter.UserCrash D-Bus signal
2017-08-11T03:06:05.809413+00:00 INFO kernel: [  506.036933] exynos-dp 145b0000.display-port-controller: EDID data does not include any extensions.
2017-08-11T03:06:05.834407+00:00 ERR kernel: [  506.061227] exynos-dp 145b0000.display-port-controller: EDID Read success!
2017-08-11T03:06:05.834446+00:00 INFO kernel: [  506.063331] exynos-dp 145b0000.display-port-controller: Link Training Clock Recovery success
2017-08-11T03:06:05.834453+00:00 INFO kernel: [  506.065013] exynos-dp 145b0000.display-port-controller: Link Training success!
2017-08-11T03:06:10.079242+00:00 INFO crash_reporter[12190]: Stored minidump to /var/spool/crash/chrome.20170810.200605.12036.dmp
2017-08-11T03:06:10.080144+00:00 INFO crash_reporter[12190]: Leaving core file at /var/spool/crash/chrome.20170810.200605.12036.core due to developer image
2017-08-11T03:06:10.154934+00:00 INFO session_manager[11954]: [INFO:child_exit_handler.cc(73)] 12083 is not a managed job.
2017-08-11T03:06:11.296101+00:00 INFO session_manager[11954]: [INFO:child_exit_handler.cc(77)] Handling 12024 exit.
2017-08-11T03:06:11.296141+00:00 ERR session_manager[11954]: [ERROR:child_exit_handler.cc(79)]   Exited with exit code 1
2017-08-11T03:06:11.296178+00:00 INFO session_manager[11954]: [INFO:session_manager_service.cc(267)] Exiting process is chrome.
2017-08-11T03:06:11.296208+00:00 INFO session_manager[11954]: [INFO:browser_job.cc(149)] Terminating process group: Ensuring browser processes are gone.
2017-08-11T03:06:11.296239+00:00 INFO session_manager[11954]: [INFO:system_utils_impl.cc(111)] Sending 9 to -12024 as 1000
2017-08-11T03:06:11.296294+00:00 INFO session_manager[11954]: [INFO:system_utils_impl.cc(111)] Sending 9 to 12024 as 0
2017-08-11T03:06:11.296373+00:00 INFO session_manager[11954]: [INFO:browser_job.cc(173)] Cleaned up child 12024

Filed https://bugs.chromium.org/p/chromium/issues/detail?id=755237