Issue metadata
Sign in to add a comment
|
DevtoolsTargetCrashException: Devtools target crashed |
||||||||||||||||||||||
Issue descriptionConfig details can be found in Run #348 - https://ct.skia.org/chromium_analysis_runs/ Swarming log: https://chrome-swarming.appspot.com/task?id=39874e3fa902b310&refresh=10&show_raw=1 Chromium Patch: PatchSet 8 of https://chromium-review.googlesource.com/c/chromium/src/+/727365 Catapult Patch: PatchSet 12 of https://chromium-review.googlesource.com/c/catapult/+/727363 ===================================================================== Traceback (most recent call last): File "/b/storage/chromium/src/third_party/android_platform/development/scripts/stack", line 240, in <module> sys.exit(main(sys.argv[1:])) File "/b/storage/chromium/src/third_party/android_platform/development/scripts/stack", line 196, in main constants.CheckOutputDirectory() File "/b/storage/chromium/src/third_party/android_platform/development/scripts/../../../../build/android/pylib/constants/__init__.py", line 208, in CheckOutputDirectory raise Exception('Neither CHROMIUM_OUTPUT_DIR nor CHROMIUM_OUT_DIR ' Exception: Neither CHROMIUM_OUTPUT_DIR nor CHROMIUM_OUT_DIR has been set Traceback (most recent call last): File "/b/storage/chromium/src/build/android/tombstones.py", line 281, in <module> sys.exit(main()) File "/b/storage/chromium/src/build/android/tombstones.py", line 262, in main constants.CheckOutputDirectory() File "/b/storage/chromium/src/build/android/pylib/constants/__init__.py", line 208, in CheckOutputDirectory raise Exception('Neither CHROMIUM_OUTPUT_DIR nor CHROMIUM_OUT_DIR ' Exception: Neither CHROMIUM_OUTPUT_DIR nor CHROMIUM_OUT_DIR has been set Traceback (most recent call last): File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 104, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 326, in RunStory self._current_page.Run(self) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 114, in Run shared_state.page_test.RunNavigateSteps(self, current_tab) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 152, in RunNavigateSteps page.RunNavigateSteps(action_runner) File "/b/storage/chromium/src/tools/perf/contrib/cluster_telemetry/page_set.py", line 29, in RunNavigateSteps action_runner.Navigate(self.url) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 187, in Navigate timeout_in_seconds=timeout_in_seconds)) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 63, in _RunAction action.RunAction(self._tab) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/navigate.py", line 23, in RunAction self._timeout_in_seconds) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 291, in Navigate self._inspector_backend.Navigate(url, script_to_evaluate_on_commit, timeout) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 38, in Inner return func(inspector_backend, *args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 185, in Navigate self._page.Navigate(url, script_to_evaluate_on_commit, timeout) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 112, in Navigate res = self._inspector_websocket.SyncRequest(request, timeout) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 116, in SyncRequest res = self._Receive(timeout) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 172, in _Receive self._HandleNotification(result) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 185, in _HandleNotification self._domain_handlers[domain_name](result) File "/b/storage/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/storage/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 460, in _HandleInspectorDomainNotification raise exception DevtoolsTargetCrashException: Devtools target crashed
,
Oct 30 2017
This is a crash that we then fail to symbolize when calling tombstones here: https://codesearch.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py?rcl=1a5cf00f76340aa799bf292e7b8c7a4e893b12c6&l=738 Hard to say for certain w/o knowing what's in run_chromium_analysis, but I'd guess that the difference is between that and https://codesearch.chromium.org/chromium/src/tools/perf/run_benchmark?rcl=5e71a6de735d268f56f221b8cbc749b56c6b4dc5&l=18. In any event, we should likely be passing the directory explicitly in android_platform_backend, the way we do for logcat.
,
Oct 31 2017
Is there any way that we can get around this issue before solving it?
,
Oct 31 2017
That second suggestion in #2 -- passing the directory in android_platform_backend.py -- should be pretty quick to implement.
,
Oct 31 2017
I've patched a fix with this task, and it's fine for first several tasks for the first 36 pageSet in the first bot, but start to appear again in the following page sets and other bots. Swarming: https://chrome-swarming.appspot.com/tasklist?c=name&c=created_ts&c=bot&c=duration&c=state&et=1509461940000&f=runid%3Amaxlg-20171031040658&l=500&n=true&s=created_ts%3Adesc&st=1262304000000 #350 in https://ct.skia.org/chromium_analysis_runs/
,
Oct 31 2017
,
Oct 31 2017
Where does that patch appear in the isolate? Does it get packaged into run_chromium_analysis? The patch that appears there for #350 is wrong, but it should have at least cleared the CHROMIUM_OUTPUT_DIR error.
,
Oct 31 2017
wait, sorry, I misread a few things. I need to correct some of what I said in #2 and its consequences for #5 and #8: - this is a browser crash - we try, and fail, to symbolize the logcat w/ a direct call to .../development/scripts/stack, because we don't pass it an output directory (though telemetry tries to do so if it can find one). - we try, and fail, to obtain and symbolize the tombstones, again because we don't pass it an output directory. Your CL in #5 cleared the error in the call to tombstones, but not the error in the call to stack. (The browser crash is obviously still present.) On further consideration, this means the fix is different than I expected, as it means that telemetry can't find an output directory at all. Depending on what run_chromium_analysis is doing, symbolization w/ it may not currently be feasible... Where can we find more info about run_chromium_analysis?
,
Oct 31 2017
+Ravi who set up Cluster Telemetry infra
,
Nov 1 2017
This is run_chromium_analysis: https://skia.googlesource.com/buildbot/+/master/ct/go/worker_scripts/run_chromium_analysis/main.go#227 It calls RunBenchmark here: https://skia.googlesource.com/buildbot/+/master/ct/go/util/util.go#641 Is it looking for Chromium out/Debug or out/Release directories? The worker bots will not have them, the android APK is copied over to the workers after being built on builders.
,
Nov 1 2017
It's weird that within a bot, sometimes the directory can be found and sometimes not. If you look at https://chrome-swarming.appspot.com/task?id=39898f700ee3f710&refresh=10&show_raw=1 you will find there is some normal cases (although with other running exception) in the middle of the log.
,
Nov 1 2017
#11: Yeah. It's looking for files with which it can symbolize crashes. If those aren't present on the skia bots, then perhaps the right fix is just to have telemetry not attempt to symbolize at all if it can't find an extant output directory. You'd still have the browser crashing, but you wouldn't have this error about the output directory. #12: I see the error every time there's a DevToolsTargetCrashedException?
,
Nov 1 2017
#13 Not always. If you search "ValueError: Slices must be added in increasing timestamp order". You will find there are some normal cases that doesn't hit these exceptions.
,
Nov 1 2017
#14: Not sure I follow your point -- that's with a different exception type?
,
Nov 1 2017
#15 Yep, that's another exception that I know of, but probably it's just thrown before the DevToolsTargetCrashedException/CHROMIUM_OUTPUT_DIR&CHROMIUM_OUT_DIR Exception. I acknowledge that it's confusing due to the two different exceptions. I am scheduling other runs to get rid of "ValueError: Slices must be added in increasing timestamp order".
,
Nov 1 2017
Interestingly, small tasks usually don't have the CHROMIUM_OUT_DIR exception. Chrome Analysis Runs:#351 (4 pages 2 bots): https://chrome-swarming.appspot.com/tasklist?c=name&c=created_ts&c=bot&c=duration&c=state&et=1509555480000&f=runid%3Amaxlg-20171101155704&l=500&n=true&s=created_ts%3Adesc&st=1262304000000 Chrome Analysis Runs: #345 (1 page 1 bot): https://chrome-swarming.appspot.com/tasklist?c=name&c=created_ts&c=bot&c=duration&c=state&et=1509555780000&f=runid%3Amaxlg-20171030155328&l=500&n=true&s=created_ts%3Adesc&st=1262304000000
,
Nov 1 2017
Sorry, I have to correct myself in #17: Run #351 is successful for bot-39, but failed for CHROMIUM_OUT_DIR at the second page in bot-78
,
Nov 1 2017
#17/18: CHROMIUM_OUT_DIR doesn't show up because the browser doesn't crash. (i.e., there's no DevToolsTargetCrashedException) The output directory issue isn't causing the browser to crash; it arises when telemetry tries to handle a browser crash.
,
Nov 2 2017
#19 Got it. Thanks for your information. I've just changed the title of this bug. There are a bunch of "DevtoolsTargetCrashException: Devtools target crashed" bugs recently. Perhaps they are related? crbug/780598 Nov 2 crbug/779910 Oct 31 crbug/763727 Sep 10 crbug/763480 Sep 8 etc.
,
Nov 2 2017
Now we know there is a browser crash, but we haven't identified what causes the crash as the crash haven't been symbolized. The full log of the crash is within: https://chrome-swarming.appspot.com/task?id=39874e3fa902b310&refresh=10&show_raw=1 which contains logcat dumps and more raw details. Is there any way that we can try to identify the real cause? Can we symbolize these information locally? How can we fix the symbolizing issue?
,
Nov 2 2017
Fixing crash symbolization to be reliably is a generally hard problem, you can see Emily's survey doc in https://docs.google.com/document/d/1CAHzQlCueCZUNNDiCK9HwjNWBm6wjut5012K3PTVoAQ/edit
,
Nov 2 2017
Thanks Ned:) I've just found recording "blink" in tracing causes new tab to crash. That's probably the reason.
,
Nov 8 2017
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by nedngu...@google.com
, Oct 30 2017