I tried TOT catapult and it seems ok

3/30 22:12:14.612 ERROR|           browser:0065| Failure while starting browser backend.
Traceback (most recent call last):
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 58, in __init__
    self._browser_backend.Start()
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 140, in Start
    self._WaitForBrowserToComeUp()
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 158, in _WaitForBrowserToComeUp
    raise exceptions.BrowserGoneException(self.browser, e)
BrowserGoneException: Timed out while waiting 240s for HasBrowserFinishedLaunching.



This looks like it could be a bug in chrome.

Some of the PFQ builders started failing yesterday with what may be the same issue.

crbug.com/706939

I didn't manage to reproduce it yesterday, but was going to try again today.

I saw that issue; maybe it's related. 

I just tried TOT chrome with TOT catapult and that worked fine too.

Still failing.

These tests pass on my wolf DUT (TOT chrome + TOT catapult).

These failures seem to occur only on VMs.

chrome doesn't come up on VMs with latest ChromeOS.

This appears to be a ChromeOS bug, as TOT chrome is fine with older ChromeOS from Mar 25.

Attached /var/log/messages. chrome appears to be in a reboot loop, and there are a lot of cryptohomed messages.

Deleted: messages.txt 393 KB

messages.txt 393 KB View Download

First failing build in the PFQ:
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/9907

Smaller list of changes:
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/amd64-generic-tot-chromium-pfq-informational/builds/11857

Following CLs are suspect:
https://chromium-review.googlesource.com/c/459068/
Make compiler-rt ebuild suitable for cross compiling in crossdev.

https://chromium-review.googlesource.com/c/463586/
Remove doins for LibCrosService.

https://chromium-review.googlesource.com/c/461317/
system_api: remove unused debugd D-Bus method name

https://chromium-review.googlesource.com/c/456716/
UPSTREAM: hostapd: Add LCI request

https://chromium-review.googlesource.com/c/454107/
CHROMIUM: arm: dts: Gale: Reduce drive strength for SDIO_CLK.

https://chromium-review.googlesource.com/c/462378/
power: Log when tagged udev devices are added or removed.

https://chromium-review.googlesource.com/c/461219/
debugd: remove GetDebugLogs method

https://chromium-review.googlesource.com/c/459598/
CHROMIUM: iwl7000: mvm: SAR profiles should go from 1 to 4

https://chromium-review.googlesource.com/c/461320/
Move buildbucket_lib to lib/

Chrome logs are useful when debugging issues like this one. :-P

---

[2993:3034:0330/175451.231055:ERROR:bus.cc(584)] Failed to get the ownership of org.chromium.LibCrosService: Connection ":1.68" is not allowed to own the service "org.chromium.LibCrosService" due to security policies in the configuration file
The subsystem type is not supported yet
The subsystem type is not supported yet
libEGL warning: MESA-LOADER: failed to retrieve device information

libEGL warning: DRI2: failed to open vgem (search paths /usr/lib64/dri)
[3052:3052:0330/175451.338794:ERROR:sandbox_linux.cc(343)] InitializeSandbox() called with multiple threads in process gpu-process. This error can be safely ignored in VMTests.
libva info: VA-API version 0.39.2
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[2993:2993:0330/175451.363459:FATAL:cros_dbus_service.cc(76)] Failed to own: org.chromium.LibCrosService
#0 0x5cb9d5ef4e6c <unknown>
#1 0x5cb9d5f091fd <unknown>
#2 0x5cb9d687af66 <unknown>
#3 0x5cb9d4ada5bc <unknown>
#4 0x5cb9d5f7ccf5 <unknown>
#5 0x5cb9d5f0ef92 <unknown>
#6 0x5cb9d5f0f24b <unknown>
#7 0x5cb9d5f0f62d <unknown>
#8 0x5cb9d5f10e49 <unknown>
#9 0x5cb9d5f0ed0b <unknown>
#10 0x5cb9d5f2e5e0 <unknown>
#11 0x5cb9d5ba8fa2 <unknown>
#12 0x5cb9d47e4824 <unknown>
#13 0x5cb9d47e77f2 <unknown>
#14 0x5cb9d47dfc4c <unknown>
#15 0x5cb9d5b480b6 <unknown>
#16 0x5cb9d6ed9fa2 <unknown>
#17 0x5cb9d5b47041 <unknown>
#18 0x5cb9d42e80d0 <unknown>
#19 0x7c0536f85816 __libc_start_main
#20 0x5cb9d42e7f09 <unknown>

Received signal 6
#0 0x5cb9d5ef4e6c <unknown>
#1 0x5cb9d5ef49e1 <unknown>
#2 0x7c053833d580 <unknown>
#3 0x7c0536f98eb2 gsignal
#4 0x7c0536f9acd6 abort
#5 0x5cb9d5ef3aa5 <unknown>
#6 0x5cb9d5f094ff <unknown>
#7 0x5cb9d687af66 <unknown>
#8 0x5cb9d4ada5bc <unknown>
#9 0x5cb9d5f7ccf5 <unknown>
#10 0x5cb9d5f0ef92 <unknown>
#11 0x5cb9d5f0f24b <unknown>
#12 0x5cb9d5f0f62d <unknown>
#13 0x5cb9d5f10e49 <unknown>
#14 0x5cb9d5f0ed0b <unknown>
#15 0x5cb9d5f2e5e0 <unknown>
#16 0x5cb9d5ba8fa2 <unknown>
#17 0x5cb9d47e4824 <unknown>
#18 0x5cb9d47e77f2 <unknown>
#19 0x5cb9d47dfc4c <unknown>
#20 0x5cb9d5b480b6 <unknown>
#21 0x5cb9d6ed9fa2 <unknown>
#22 0x5cb9d5b47041 <unknown>
#23 0x5cb9d42e80d0 <unknown>
#24 0x7c0536f85816 __libc_start_main
#25 0x5cb9d42e7f09 <unknown>
  r8: ffffb06edfd751f8  r9: ffffb06edfd751e8 r10: 0000000000000008 r11: 0000000000000206
 r12: 00007ffc22b27820 r13: 00005cb9dbe31d10 r14: 00007ffc22b27590 r15: 00007ffc22b27580
  di: 0000000000000bb1  si: 0000000000000bb1  bp: 00007ffc22b27100  bx: 0000000000000000
  dx: 0000000000000006  ax: 0000000000000000  cx: 00007c0536f98eb2  sp: 00007ffc22b26fd8
  ip: 00007c0536f98eb2 efl: 0000000000000206 cgf: 0000000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.

---

So, looks like it's https://chromium-review.googlesource.com/c/463586/.

I'm confused about why this is failing, because:

- Chrome is copying these configs to its output dir: https://codereview.chromium.org/2764513002
- chromite is installing these configs to /opt/google/chrome/dbus: https://chromium-review.googlesource.com/457260
- /opt/google/chrome/dbus is searched by dbus-daemon: https://chromium-review.googlesource.com/461330

Thanks for digging into this, Dan! Any chance we can revert this change? All the informational builders rely on VM tests, as well as a number of waterfall builders.

Yeah, I think that reverting is safe (since in the worst case we'd be loading two copies of the same "allow" directives).

I'll revert.

Revert is at https://chromium-review.googlesource.com/c/466510/; feel free to chump it if it needs to go in quickly (or ask me to do it).

If it's easy for you to grab from one of the failing devices, I (and Justin too, I suspect) would love to see a directory listing of /etc/dbus-1/system.d and /opt/google/chrome/dbus.

If we have https://chromium-review.googlesource.com/c/463586/ then we must also have all the earlier OS changes that make dbus-daemon load configs from /opt/google/chrome/dbus.

That implies to me that the deploy_chrome part of this isn't working, but https://codereview.chromium.org/2764513002 and https://chromium-review.googlesource.com/c/457260/ both went in back on March 20.

using the buildbot log:
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational/builds/11489

look at the cbuildbot Artifacts link under the cbuildbot Report stage:
https://storage.cloud.google.com/chromeos-image-archive/x86-generic-tot-chromium-pfq-informational/R59-9416.0.0-b11489/index.html

that should have all the input images before the VM launched.

I'll chump the CL - thanks Dan.

Issue 707469 has been merged into this issue.

The revert fixed the bot failure, so closing.