Error during browser startup: android.content.pm.PackageManager$NameNotFoundException: com.google.android.gms

Logs: go/chrome-androidlogs1/5/624556

broke this morning between these two commits: 
pass: 392099634d1ca439a1365569f558a5661af17a0d
fail: 13be608a5c62ca243fb665e8d19a37c379e82bbe

Oh that's not good. Webview can't assume there is gms on the device (chrome can, sort of..). Full stack:

4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: [BuildInfo.java:74] GMS package is not found: android.content.pm.PackageManager$NameNotFoundException: com.google.android.gms
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: android.content.pm.PackageManager$NameNotFoundException: com.google.android.gms
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at android.app.ApplicationPackageManager.getPackageInfo(ApplicationPackageManager.java:137)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.base.BuildInfo.getGMSVersionCode(BuildInfo.java:71)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.content.browser.BrowserStartupController.nativeSetCommandLineFlags(Native Method)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.content.browser.BrowserStartupController.access$700(BrowserStartupController.java:40)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.content.browser.BrowserStartupController$4.run(BrowserStartupController.java:316)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.content.browser.BrowserStartupController.prepareToStartBrowserProcess(BrowserStartupController.java:329)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.content.browser.BrowserStartupController.startBrowserProcessesSync(BrowserStartupController.java:211)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.android_webview.AwBrowserProcess$1.run(AwBrowserProcess.java:88)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.base.ThreadUtils.runOnUiThreadBlocking(ThreadUtils.java:67)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.android_webview.AwBrowserProcess.start(AwBrowserProcess.java:78)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at com.android.webview.chromium.WebViewChromiumFactoryProvider.startChromiumLocked(WebViewChromiumFactoryProvider.java:288)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at com.android.webview.chromium.WebViewChromiumFactoryProvider.ensureChromiumStartedLocked(WebViewChromiumFactoryProvider.java:229)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at com.android.webview.chromium.WebViewChromiumFactoryProvider.startYourEngines(WebViewChromiumFactoryProvider.java:320)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at com.android.webview.chromium.WebViewChromium.init(WebViewChromium.java:223)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at android.webkit.WebView.<init>(WebView.java:606)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at android.webkit.WebView.<init>(WebView.java:542)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at android.webkit.WebView.<init>(WebView.java:525)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at android.webkit.WebView.<init>(WebView.java:512)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at android.webkit.WebView.<init>(WebView.java:502)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at org.chromium.webview_shell.WebViewThreadTestActivity$2.run(WebViewThreadTestActivity.java:137)
4bab5:  06-29 15:58:21.949  3805  3831 D cr_BuildInfo: 	at java.lang.Thread.run(Thread.java:818)

Issue 624717 has been merged into this issue.

This bot is not failing the same set of tests every time and they are not always failing with the same call stack (e.g. testLoadDataNonUiThread doesn't fail in createWebViewOnNonUiThreadbut instead fails in loadDataWebViewNonUiThread.

I ran these tests locally with an earlier local version of ToT (from yesterday). After adding the commit https://codereview.chromium.org/2109043002
locally the following tests start failing
org.chromium.webview_shell.test.WebViewThreadTest#testLoadDataNonUiThread
org.chromium.webview_shell.test.WebViewThreadTest#testLoadUrlNonUiThread

so that could be the culprit..?

you sure? https://codereview.chromium.org/2109043002 doesn't seem to have anything to do with gmscore

The gms core log message is not a crash, it's just a debug log print from a try/catch block (https://cs.chromium.org/chromium/src/base/android/java/src/org/chromium/base/BuildInfo.java?rcl=0&l=74) so that is not necessarily the cause of the failures.

Oh you are right. Then was my duping in #6 correct then?

I'm still guessing that this and issue 624717 are the same, and neither has anything to do with gms (that was a red herring due to log spam).

https://codereview.chromium.org/2109043002 is in my tiny 3-CL blamelist as well, so I would also suspect of it.

Additionally see https://bugs.chromium.org/p/chromium/issues/detail?id=624717#c11 for instructions to reproduce the error and a smaller (and probably more useful) logcat.

It's an ANR, ie the main thread is deadlocks. This would be the UI thread sending a synchronous IPC to the renderer compositor thread. So I guess renderer ipc/mojo init could affect that.

ben/rockot: Could we revert first and ask questions later?

  RELADDR   FUNCTION                                                                                            FILE:LINE
  000175e4  syscall+28                                                                                          /system/lib/libc.so
  0003f27b  __pthread_cond_timedwait_relative(pthread_cond_internal_t*, pthread_mutex_t*, timespec const*)+56   /system/lib/libc.so
  000afcdd  Wait                                                                                                /android/chromium/src/base/synchronization/condition_variable_posix.cc:69
  000b0c77  WaitMany                                                                                            /android/chromium/src/base/synchronization/waitable_event_posix.cc:277
  00021e85  WaitForReply                                                                                        /android/chromium/src/ipc/ipc_sync_channel.cc:524
  00021cd7  Send                                                                                                /android/chromium/src/ipc/ipc_sync_channel.cc:505
  006f23f1  Send                                                                                                /android/chromium/src/content/browser/renderer_host/render_process_host_impl.cc:1711
  006feff5  Send                                                                                                /android/chromium/src/content/browser/renderer_host/render_widget_host_impl.cc:494
  007a1dab  DemandDrawHw                                                                                        /android/chromium/src/content/browser/android/synchronous_compositor_host.cc:111
  00062fcb  OnDrawHardware                                                                                      /android/chromium/src/android_webview/browser/browser_view_renderer.cc:244
  0007bb59  OnDraw                                                                                              /android/chromium/src/android_webview/native/aw_contents.cc:948
  0007ba7f  Java_org_chromium_android_1webview_AwContents_nativeOnDraw                                          /android/chromium/src/out/Default/gen/android_webview/native/native_jni/android_webview/jni/AwContents_jni.h:239
  0003ab6d  Java_org_chromium_android_1webview_AwContents_nativeOnDraw__JLandroid_graphics_Canvas_2ZIIIIII+176  /data/app/com.google.android.webview-2/oat/arm/base.odex
  at org.chromium.android_webview.AwContents.nativeOnDraw(Native method)
  at org.chromium.android_webview.AwContents.access$4700(AwContents.java:98)
  at org.chromium.android_webview.AwContents$AwViewMethodsImpl.onDraw(AwContents.java:2976)
  at org.chromium.android_webview.AwContents.onDraw(AwContents.java:1307)
  at com.android.webview.chromium.WebViewChromium.onDraw(WebViewChromium.java:1696)
  at android.webkit.WebView.onDraw(WebView.java:2486)
  at android.view.View.draw(View.java:16184)
  at android.view.View.updateDisplayListIfDirty(View.java:15180)
  at android.view.View.draw(View.java:15954)
  at android.view.ViewGroup.drawChild(ViewGroup.java:3609)
  at android.view.ViewGroup.dispatchDraw(ViewGroup.java:3399)
  at android.view.View.updateDisplayListIfDirty(View.java:15175)
  at android.view.View.draw(View.java:15954)
  at android.view.ViewGroup.drawChild(ViewGroup.java:3609)
  at android.view.ViewGroup.dispatchDraw(ViewGroup.java:3399)
  at android.view.View.updateDisplayListIfDirty(View.java:15175)
  at android.view.View.draw(View.java:15954)
  at android.view.ViewGroup.drawChild(ViewGroup.java:3609)
  at android.view.ViewGroup.dispatchDraw(ViewGroup.java:3399)
  at android.view.View.updateDisplayListIfDirty(View.java:15175)
  at android.view.View.draw(View.java:15954)
  at android.view.ViewGroup.drawChild(ViewGroup.java:3609)
  at android.view.ViewGroup.dispatchDraw(ViewGroup.java:3399)
  at android.view.View.draw(View.java:16187)
  at com.android.internal.policy.PhoneWindow$DecorView.draw(PhoneWindow.java:2690)
  at android.view.View.updateDisplayListIfDirty(View.java:15180)
  at android.view.ThreadedRenderer.updateViewTreeDisplayList(ThreadedRenderer.java:281)
  at android.view.ThreadedRenderer.updateRootDisplayList(ThreadedRenderer.java:287)
  at android.view.ThreadedRenderer.draw(ThreadedRenderer.java:322)
  at android.view.ViewRootImpl.draw(ViewRootImpl.java:2615)
  at android.view.ViewRootImpl.performDraw(ViewRootImpl.java:2434)
  at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:2067)
  at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:1107)
  at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:6013)
  at android.view.Choreographer$CallbackRecord.run(Choreographer.java:858)
  at android.view.Choreographer.doCallbacks(Choreographer.java:670)
  at android.view.Choreographer.doFrame(Choreographer.java:606)
  at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:844)
  at android.os.Handler.handleCallback(Handler.java:739)
  at android.os.Handler.dispatchMessage(Handler.java:95)
  at android.os.Looper.loop(Looper.java:148)
  at android.app.ActivityThread.main(ActivityThread.java:5417)
  at java.lang.reflect.Method.invoke!(Native method)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)

This is apparently timing dependent. webview in gmail loads fine, but the test shell these tests are using hits it pretty consistently, or at least I hit it on first try

I don't think this needs to be RVG

Attaching traces.txt, which is the thread stack dump of every single thread.
And also stack.txt, which is the native part of those stacks symbolized, but it also strips out other things like thread name and whatnot, so have to look back and forth to match them up a bit

Deleted: traces.txt 279 KB

traces.txt 279 KB View Download

Deleted: stack.txt 183 KB

stack.txt 183 KB View Download

Revert in CQ

Chrome_IOThread is in epoll, so looks like just waiting for work

jvm didn't dump Chrome_ChildIOT or Compositor threads because they weren't attached to the jvm :/

revert landed: https://chromium.googlesource.com/chromium/src/+/b44da081cde654f070a29ececea36b33214a7c5e