New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 911678 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug-Regression

Blocking:
issue crashpad:30



Sign in to add a comment

ContextLost_WebGLBlockedAfterJSNavigation times out on Nexus 5 and Nexus 6

Project Member Reported by ynovikov@chromium.org, Dec 4

Issue description

Started in https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20FYI%20Release%20%28Nexus%205%29/12165
and
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20FYI%20Release%20%28Nexus%206%29/14677

ContextLost_WebGLUnblockedAfterUserInitiatedReload in context_lost_tests is also failing, but it was Flaky, so not sure if related.

Regression range is http://crrev.com/613320..613337, there are many commits in there that could have slowed browser startup, from my limited understanding.

  _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:155
    self.RunActualGpuTest(url, *args)
  RunActualGpuTest at content/test/gpu/gpu_tests/context_lost_integration_test.py:107
    getattr(self, test_name)(test_path)
  _ContextLost_WebGLBlockedAfterJSNavigation at content/test/gpu/gpu_tests/context_lost_integration_test.py:298
    timeout=wait_timeout)
  traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:245
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
  traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:307
    self.GetCurrentConsoleOutputBuffer())
  traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  Inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:41
    inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
  traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  Inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:38
    return func(inspector_backend, *args, **kwargs)
  GetCurrentConsoleOutputBuffer at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:196
    return self._console.GetCurrentConsoleOutputBuffer(timeout)
  GetCurrentConsoleOutputBuffer at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_console.py:38
    self._inspector_websocket.DispatchNotifications(timeout)
  DispatchNotifications at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:156
    self._Receive(timeout)
  _Receive at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:181
    raise WebSocketException(err)
TimeoutException: Timeout after 60s while waiting for JavaScript:window.initFinished

Ken, could you please triage this further?
 
Blocking: crashpad:30
Cc: mark@chromium.org kbr@chromium.org yfried...@chromium.org jperaza@chromium.org tobiasjs@chromium.org
Components: Internals>CrashReporting
ContextLost_WebGLBlockedAfterJSNavigation crashes the GPU process and waits for the page to detect the context loss event before attempting to navigate to a new page:
https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/context_lost_integration_test.py?q=context_lost_integra&sq=package:chromium&g=0&l=297

from this failure:
https://chromium-swarm.appspot.com/task?id=418f6038658c0d10&refresh=10&show_raw=1

it looks like the page is never receiving the context loss:
Traceback (most recent call last):
  _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:155
    self.RunActualGpuTest(url, *args)
  RunActualGpuTest at content/test/gpu/gpu_tests/context_lost_integration_test.py:107
    getattr(self, test_name)(test_path)
  _ContextLost_WebGLBlockedAfterJSNavigation at content/test/gpu/gpu_tests/context_lost_integration_test.py:298
    timeout=wait_timeout)
  traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:245
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
...
  _Receive at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:181
    raise WebSocketException(err)
TimeoutException: Timeout after 60s while waiting for JavaScript:window.initFinished


ContextLost_WebGLUnblockedAfterUserInitiatedReload is failing multiple times and I can't see why from the logs.

Looking at the blamelist:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20FYI%20Release%20%28Nexus%205%29/12165

suspect CLs would be:
Reland "Enable Crashpad for Android"
b8d9022384f5cb730281492d6279ed40f88263ae

Actually, I think that's the most likely one.

I don't want to revert the above CL - moving to Crashpad for Android is urgent - will suppress these failures. jperaza@, others, do you have a Nexus 5 or Nexus 6 device on which you could try to reproduce these failures? Maybe something is wrong with the 32-bit Crashpad client, or something is wrong on these older devices or OS versions.

I don't have a Nexus 5 or Nexus 6, but can try reproducing on emulators.

I thought it might have to do with the raciness comment here:
https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/context_lost_integration_test.py?rcl=0bfabec2ac11decb989a7a03ea5980eaf56893e6&l=29

Crashpad almost certainly takes longer to finish crash dumping than Breakpad did, if Breakpad was enabled in this context before. I tried a couple sleeps in the tests, but it didn't resolve the problem:
https://chromium-review.googlesource.com/c/chromium/src/+/1361644
It's certainly possible there are race conditions in these tests. Still, the reliable failures on these devices are something we can investigate.

Owner: jperaza@google.com
Labels: -Type-Bug Type-Bug-Regression
Project Member

Comment 7 by bugdroid1@chromium.org, Dec 4

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8de7f378da3c9a8e224ea9fa48d4906196e0003b

commit 8de7f378da3c9a8e224ea9fa48d4906196e0003b
Author: Kenneth Russell <kbr@chromium.org>
Date: Tue Dec 04 20:03:47 2018

Skip two context_lost tests on Nexus 5 and 6.

  ContextLost_WebGLBlockedAfterJSNavigation
  ContextLost_WebGLUnblockedAfterUserInitiatedReload

These are reliably failing and/or timing out due to recent browser
changes. Skip them on these two bots.

Tbr: ynovikov@chromium.org
Bug: 911678
Change-Id: I9a8cd3f85a0578070bb57bb3896e83ff437024c7
Reviewed-on: https://chromium-review.googlesource.com/c/1361599
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#613664}
[modify] https://crrev.com/8de7f378da3c9a8e224ea9fa48d4906196e0003b/content/test/gpu/gpu_tests/context_lost_expectations.py

Status: Assigned (was: Untriaged)
I thought this was fixed?

On Fri., Dec. 7, 2018, 14:35 ccameron via monorail <
monorail+v2.2903610194@chromium.org wrote:
Sorry! I got a bit side-tracked and hadn't submit the fix yet. Rebasing/CQing now...
Project Member

Comment 11 by bugdroid1@chromium.org, Dec 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d587561ae75deb5e1f310cca81e59ab5970e7e3f

commit d587561ae75deb5e1f310cca81e59ab5970e7e3f
Author: Joshua Peraza <jperaza@chromium.org>
Date: Mon Dec 10 20:29:20 2018

android: Restore crash signal handlers for child processes

The original tombstone producing signal handlers are still used by
tests.

This can be revisited later:
https://bugs.chromium.org/p/chromium/issues/detail?id=912306

Bug: 911678, 912166, 912161
Change-Id: I55c0627bc431c6d61872a60d47423fd99567028f
Reviewed-on: https://chromium-review.googlesource.com/c/1361597
Commit-Queue: Joshua Peraza <jperaza@chromium.org>
Reviewed-by: Mark Mentovai <mark@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#615230}
[modify] https://crrev.com/d587561ae75deb5e1f310cca81e59ab5970e7e3f/components/crash/content/app/crashpad_linux.cc

Project Member

Comment 12 by bugdroid1@chromium.org, Jan 16 (6 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9e8a68b1c1f9f3ebef90b7547580cbfa83f61a39

commit 9e8a68b1c1f9f3ebef90b7547580cbfa83f61a39
Author: Joshua Peraza <jperaza@chromium.org>
Date: Wed Jan 16 16:31:40 2019

android: re-enable context lost tests

These should be running correctly after recent fixes to Crashpad.

Bug: 911678
Change-Id: Ib92b29c71a4e1195cf2031f7c9927f8f3edd5a7b
Reviewed-on: https://chromium-review.googlesource.com/c/1413332
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Joshua Peraza <jperaza@chromium.org>
Cr-Commit-Position: refs/heads/master@{#623254}
[modify] https://crrev.com/9e8a68b1c1f9f3ebef90b7547580cbfa83f61a39/content/test/gpu/gpu_tests/context_lost_expectations.py

Sign in to add a comment