I see the following @ https://stainless.corp.google.com/browse/chromeos-autotest-results/277214501-chromeos-test/ - looks like issue 921479

(from chrome.20190115.081947.5026.log)

2019-01-15T16:19:46.282769+00:00 INFO session_manager[653]: [INFO:session_manager_impl.cc(1637)] Started Android container with PID 5398
2019-01-15T16:19:46.694831+00:00 INFO session_manager[653]: [INFO:policy_key.cc(49)] No policy key on disk at /home/root/9d03ce79361d610cc015b6787e9010503dd78ae7/session_manager/policy/key
2019-01-15T16:19:46.725271+00:00 ERR session_manager[653]: [ERROR:nss_util.cc(145)] Not checking key because size is 0
2019-01-15T16:19:46.725450+00:00 WARNING session_manager[653]: [WARNING:device_policy_service.cc(504)] Could not verify that owner key belongs to this user.
2019-01-15T16:19:46.725533+00:00 ERR session_manager[653]: [ERROR:dbus_util.cc(13)] CreateError(...): Domain=dbus, Code=org.chromium.SessionManagerInterface.PubkeySetIllegal, Message=Could not verify that owner key belongs to this user.
2019-01-15T16:19:46.728899+00:00 INFO session_manager[653]: [INFO:upstart_signal_emitter.cc(35)] Emitting start-user-session Upstart signal
2019-01-15T16:19:46.733359+00:00 INFO session_manager[653]: [INFO:session_manager_impl.cc(658)] Starting user session
2019-01-15T16:19:46.733548+00:00 INFO session_manager[653]: [INFO:session_manager_impl.cc(666)] Emitting D-Bus signal SessionStateChanged:started
2019-01-15T16:19:46.745999+00:00 INFO session_manager[653]: [INFO:key_generator.cc(55)] Generating key at /home/user/9d03ce79361d610cc015b6787e9010503dd78ae7/key.pub using nssdb under /home/user/9d03ce79361d610cc015b6787e9010503dd78ae7

(from chrome.20190115.081947.5026.dmp.txt)

Operating system: Linux
                  0.0.0 Linux 4.4.169-15634-g6da702e55181 #1 SMP PREEMPT Tue Jan 15 06:22:53 PST 2019 aarch64
CPU: arm
     ARMv1 ARM part(0x4100d0b0) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
     6 CPUs

GPU: UNKNOWN

Crash reason:  SIGSEGV
Crash address: 0x0
Process uptime: not available

Thread 0 (crashed)
 0  chrome!chromeos::ProxyResolutionServiceProvider::ResolveProxy(dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>) [proxy_resolution_service_provider.cc : 135 + 0x0]
     r0 = 0x0a45b7a0    r1 = 0x01652b19    r2 = 0x01652ae9    r3 = 0x02755b75
     r4 = 0x02755545    r5 = 0x0748a638    r6 = 0x07c659b0    r7 = 0xffb62448
     r8 = 0x00000001    r9 = 0x07c2f880   r10 = 0x00000000   r12 = 0x00000000
     fp = 0x07ba4520    sp = 0xffb62310    lr = 0x01643c04    pc = 0x0275529e
    Found by: given as instruction pointer in context
 1  chrome!base::internal::Invoker<base::internal::BindState<void (ash::DisplayServiceProvider::*)(dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>), base::WeakPtr<ash::DisplayServiceProvider> >, void (dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>)>::Run(base::internal::BindStateBase*, dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>&&) [bind_internal.h : 516 + 0x1]
     r4 = 0xef123ac0    r5 = 0x00000000    r6 = 0x0c6a0938    r7 = 0xffb62460
     r8 = 0xffb624a0    r9 = 0x0b921493   r10 = 0xef123ac0    fp = 0x00000000
     sp = 0xffb62450    pc = 0x02752c6d
    Found by: call frame info
 2  chrome!dbus::ExportedObject::RunMethod(base::RepeatingCallback<void (dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>)>, std::__1::unique_ptr<dbus::MethodCall, std::__1::default_delete<dbus::MethodCall> >, base::TimeTicks) [callback.h : 129 + 0x1]
     r4 = 0x07be8450    r5 = 0x0417c779    r6 = 0x0c6a0938    r7 = 0xffb62490
     r8 = 0xffb624a0    r9 = 0x0b921493   r10 = 0xef123ac0    fp = 0x00000000
     sp = 0xffb62468    pc = 0x0417bcb3
    Found by: call frame info
 3  chrome!base::internal::Invoker<base::internal::BindState<void (dbus::ExportedObject::*)(base::RepeatingCallback<void (dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>)>, std::__1::unique_ptr<dbus::MethodCall, std::__1::default_delete<dbus::MethodCall> >, base::TimeTicks), scoped_refptr<dbus::ExportedObject>, base::RepeatingCallback<void (dbus::MethodCall*, base::RepeatingCallback<void (std::__1::unique_ptr<dbus::Response, std::__1::default_delete<dbus::Response> >)>)>, std::__1::unique_ptr<dbus::MethodCall, std::__1::default_delete<dbus::MethodCall> >, base::TimeTicks>, void ()>::RunOnce(base::internal::BindStateBase*) [bind_internal.h : 516 + 0x3]
     r4 = 0x0417bc2d    r5 = 0xef123ac0    r6 = 0x00000000    r7 = 0xffb624b8
     r8 = 0x00000000    r9 = 0x07b1d9c0   r10 = 0xffb62610    fp = 0xef123ac0
     sp = 0xffb62498    pc = 0x0417c5cd
    Found by: call frame info
 4  chrome!base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) [callback.h : 99 + 0x3]
     r4 = 0x077dd150    r5 = 0x0a5a7570    r6 = 0x0736cc1c    r7 = 0xffb62548
     r8 = 0x00000000    r9 = 0x07b1d9c0   r10 = 0xffb62610    fp = 0xef123ac0
     sp = 0xffb624c0    pc = 0x0359c2fd
    Found by: call frame info
 5  chrome!base::MessageLoopImpl::RunTask(base::PendingTask*) [message_loop_impl.cc : 353 + 0x5]
     r4 = 0x07cbb914    r5 = 0x0760ff28    r6 = 0x00eb322d    r7 = 0xffb625e0
     r8 = 0x00000928    r9 = 0x07b1d9c0   r10 = 0x07cbb914    fp = 0xffb62610
     sp = 0xffb62550    pc = 0x0359b4cf
    Found by: call frame info
 6  chrome!base::MessageLoopImpl::DoWork() [message_loop_impl.cc : 364 + 0x5]
     r4 = 0x07b1d9c0    r5 = 0x07b1da24    r6 = 0xffb62638    r7 = 0xffb62750
     r8 = 0xffb626b8    r9 = 0x00eba675   r10 = 0x074baf64    fp = 0x00000019
     sp = 0xffb625e8    pc = 0x0359bbc1
    Found by: call frame info
 7  chrome!base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) [message_pump_libevent.cc : 210 + 0x3]
     r4 = 0x07b3e600    r5 = 0x07b1d9c4    r6 = 0x00000000    r7 = 0xffb627a8
     r8 = 0x07562090    r9 = 0x00eba675   r10 = 0xef123ac0    fp = 0x00000001
     sp = 0xffb62758    pc = 0x0361b9db
    Found by: call frame info
 8  chrome!base::RunLoop::Run() [run_loop.cc : 150 + 0x23]
     r4 = 0x075620b0    r5 = 0x00eba676    r6 = 0x08a4c344    r7 = 0xffb62820
     r8 = 0x00000000    r9 = 0x07b1d9cc   r10 = 0xef123ac0    fp = 0x07c8afe0
     sp = 0xffb627b0    pc = 0x035b5ed9
    Found by: call frame info
 9  chrome!ChromeBrowserMainParts::MainMessageLoopRun(int*) [chrome_browser_main.cc : 1859 + 0x3]
     r4 = 0x077dbf38    r5 = 0x077dbf40    r6 = 0x0736cc30    r7 = 0xffb62888
     r8 = 0xef123ac0    r9 = 0x07b26400   r10 = 0xef123ac0    fp = 0x077dec18
     sp = 0xffb62828    pc = 0x0329c4e1
    Found by: call frame info
10  chrome!content::BrowserMainLoop::RunMainMessageLoopParts() [browser_main_loop.cc : 1000 + 0x1]
     r4 = 0x0736cc30    r5 = 0x077dea18    r6 = 0x07bd33e0    r7 = 0xffb62900
     r8 = 0xef123ac0    r9 = 0x07b26400   r10 = 0x07562b48    fp = 0x077dec18
     sp = 0xffb62890    pc = 0x01e7716f
    Found by: call frame info
11  chrome!content::BrowserMainRunnerImpl::Run() [browser_main_runner_impl.cc : 165 + 0x3]
     r4 = 0x07bd33e0    r5 = 0x077dea18    r6 = 0x07bd33e0    r7 = 0xffb62910
     r8 = 0xffb629c4    r9 = 0xef123ac0   r10 = 0x07562b48    fp = 0x077dec18
     sp = 0xffb62908    pc = 0x01e79f1d
    Found by: call frame info
12  chrome!content::BrowserMain(content::MainFunctionParams const&) [browser_main.cc : 47 + 0x5]
     r4 = 0x077dea18    r5 = 0x077dea18    r6 = 0x07bd33e0    r7 = 0xffb62990
     r8 = 0xffb629c4    r9 = 0xef123ac0   r10 = 0x07562b48    fp = 0x077dec18
     sp = 0xffb62918    pc = 0x01e722af
    Found by: call frame info
13  chrome!content::ContentMainRunnerImpl::Run(bool) [content_main_runner_impl.cc : 545 + 0x3]
     r4 = 0xffffffff    r5 = 0x00000000    r6 = 0x00000000    r7 = 0xffb62ae8
     r8 = 0x07b3bd20    r9 = 0x00000000   r10 = 0x07b2efa0    fp = 0xef123ac0
     sp = 0xffb62998    pc = 0x03291795
    Found by: call frame info
14  chrome!service_manager::Main(service_manager::MainParams const&) [main.cc : 461 + 0x15]
     r4 = 0x07b22760    r5 = 0x073c5ce8    r6 = 0xffffffff    r7 = 0xffb62ca8
     r8 = 0x00000003    r9 = 0xffb62cbc   r10 = 0xef123ac0    fp = 0x00000000
     sp = 0xffb62af0    pc = 0x032961cf
    Found by: call frame info
15  chrome!content::ContentMain(content::ContentMainParams const&) [content_main.cc : 19 + 0x3]
     r4 = 0xffb62d00    r5 = 0xffb62cf8    r6 = 0xef123ac0    r7 = 0xffb62cf0
     r8 = 0xffb62cbc    r9 = 0x00000000   r10 = 0x0775f30c    fp = 0x00000000
     sp = 0xffb62cb0    pc = 0x0329007b
    Found by: call frame info
16  chrome!ChromeMain [chrome_main.cc : 102 + 0x3]
     r4 = 0xffb62ec4    r5 = 0x0000003e    r6 = 0x07b22760    r7 = 0xffb62d68
     r8 = 0xef123ac0    r9 = 0x00000000   r10 = 0x0775f30c    fp = 0x00000000
     sp = 0xffb62cf8    pc = 0x0163780d
    Found by: call frame info
17  libc-2.27.so!__libc_start_main [libc-start.c : 308 + 0xb]
     r4 = 0xffb62d88    r5 = 0x00000000    r6 = 0x00000000    r7 = 0x00000000
     r8 = 0x00000000    r9 = 0x00000000   r10 = 0x0775f30c    fp = 0x00000000
     sp = 0xffb62d70    pc = 0xee58b0a3
    Found by: call frame info
18  chrome!_start + 0x32
     r4 = 0x00000000    r5 = 0x00000000    r6 = 0x014b0001    r7 = 0x00000000
     r8 = 0x00000000    r9 = 0x00000000   r10 = 0x0775f30c    fp = 0x00000000
     sp = 0xffb62eb8    pc = 0x014b0035
    Found by: call frame info
19  chrome!__libc_csu_init [elf-init.c : 87 + 0xf]
     sp = 0xffb62ebc    pc = 0x0162320d
    Found by: stack scanning
20  0xffb640e3
     r3 = 0xef106b2d    r4 = 0xffb62ec4    r5 = 0xffb6403c    r6 = 0xffb64056
     r7 = 0xffb64098    r8 = 0xffb640b9    r9 = 0xffb640d8    sp = 0xffb62edc
     pc = 0xffb640e5
    Found by: call frame info

Yes, I made a comment on issue 921479 .. I will see if I can bisect locally.

It seems the same crash is causing failures on tricky-chrome-pfq and veyron_minnie-chrome-pfq.

I cannot repro issue 921479 at ToT on eve. I don't have any of the devices mentioned in that issue (Daisy, Kip, Reks), nor any of the ones associated with the failing builders (caroline, kevin, tricky, veyron_minnie).

Let me try one more thing.

I assume this (and the related bugs) was caused by crrev.com/c/1399297 (7112246b3d), which I landed at 1:30pm yesterday, and reverted in crrev.com/c/1411072 (4dfaf6c9c1) at 9:30pm. Are these failures in that window?

Thanks for letting us know. I don't think the PFQ picked up the revert yet. It synced to version 73.0.3672.0 which doesn't include the revert.

The range between the last good build and the failing build includes only the bad CL: https://chromium.googlesource.com/chromium/src/+log/73.0.3671.0..73.0.3672.0?pretty=fuller&n=10000

+Bernie, I have a question: The current PFQ run is doomed, because the builder is syncing chrome to 73.0.3672.0 which doesn't contain the revert mentioned #6. What to do? Merge the revert back into that branch? Or push a new tag that the PFQ may pick?

+bhthompson@ for #8.

Also +dgagnon@

The way one would do this would be to talk to the Chrome TPM and ask if it is ok to pick the change into the prior day's Chrome branch, then we trigger a new Chrome version on that branch (using https://chromepmo.appspot.com/trigger) which in turn triggers a new PFQ run with the new version (e.g. 3672.1).

However at this point in the day, if the change is landed on ToT Chrome, there will be a Chrome (e.g. 3673) in another few hours, even if we triggered now the PFQ would not be done before the next Chrome exists.

cc'ing non-PST constable

Can this be closed now?

The revert in question should have made it to the PFQ.

Sounds good to close - Chrome PFQ had a successful run as well that picked up the revert mentioned in comment #6