New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 657674 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug-Regression



Sign in to add a comment

page_cycler_v2.intl_ar_fa_he failing on 3 builders

Project Member Reported by martiniss@chromium.org, Oct 20 2016

Issue description

Builders failed on: 
- Android Nexus5 Perf (3): 
  https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%283%29
- Android Nexus6 Perf (3): 
  https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%283%29
- Android Nexus9 Perf (3): 
  https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus9%20Perf%20%283%29

Seeing a lot of this failure in the logs: 
Traceback (most recent call last):
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 88, in _RunStoryAndProcessErrorIfNeeded
    test.Measure(state.platform, results)
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 287, in Measure
    trace_result = platform.tracing_controller.StopTracing()
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 47, in StopTracing
    return self._tracing_controller_backend.StopTracing()
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 108, in StopTracing
    self._IssueClockSyncMarker()
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 203, in _IssueClockSyncMarker
    self._RecordIssuerClockSyncMarker)
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 193, in RecordClockSyncMarker
    sync_id, record_controller_clock_sync_marker_callback)
  File "/b/rr/tmpiZyANI/w/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 170, in _RecordClockSyncMarkerAsyncEvent
    'Failed to issue clock sync to devtools client')
ChromeClockSyncError: Failed to issue clock sync to devtools client

Been failing pretty consistently for a while.

cc-ing test owners.
 
Owner: caseq@chromium.org
This is strange, the error happens when Telemetry try to do backcompat clock sync, but in this case it should be using the devtool based clock sync. The log shows that happens because Chrome revision returned by devtool is None:

INFO:root:Chrome version: None  # <--- ???
INFO:root:*************** BROWSER STANDARD OUTPUT ***************
INFO:root:Cannot get standard output on Android
INFO:root:*********** END OF BROWSER STANDARD OUTPUT ************
INFO:root:********************* BROWSER LOG *********************
INFO:root:No log file

I think this is probably a bug in devtool. The page is still running fine when this fails:

https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_13-2016-10-19_16-38-3362242.png

(https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf__3_%2F3502%2F%2B%2Frecipes%2Fsteps%2Fpage_cycler_v2.intl_ar_fa_he%2F0%2Fstdout)

Comment 3 by caseq@chromium.org, Oct 20 2016

Cc: -nednguyen@chromium.org caseq@chromium.org
Owner: nednguyen@chromium.org
Ned, I'm not sure how I can help here. Looking at the telemetry clock sync code, I think this could only happen when devtools_clients is empty here:

https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py?rcl=0&l=138

So I wonder how a bug in DevTools, whatever it might be, could affect this?
Oh, good catch. I forget that part. It means that chrome_tracing_devtools_manager.GetActiveDevToolsClients(self._platform_backend)) is empty & devtool client has crashed before that.

I am not sure why the log is failed to display though.
Cc: zh...@chromium.org
Here is what get tricky to get the log:
The page navigate 

The page crashes but Telemetry doesn't know about the crash, because Telemetry doesn't interactive with it.

We start tracing. Because we bend the tracing API to make it a platform level thing, we don't know about the crash either, all we know is there is no devtool clients available to request tracing.
Cc: -zh...@chromium.org nednguyen@chromium.org
Owner: zh...@chromium.org
Zhen: I am really swamped at this point. Can you find a way to throw ChromeCrash Exception in ChromeTracingAgent in appropriate way to show the chrome crash stack?

Comment 7 by zh...@chromium.org, Oct 21 2016

Current chrome crash stack is handled by inspector_backend.py. Why do we want to do it again in chrome tracing agent?
Zhen: inspector_backend only throw chrome crash stack if you invoke method calls on it. In this situation, Chrome crashes somewhere between the last call to inspector_backend  & platform.tracing_controller.Start(), hence we completely have no information about what went wrong.

Comment 9 by zh...@chromium.org, Oct 21 2016

That seems not Chrome tracing agent's responsibility. Chrome tracing agent has no knowledge about whether Chrome is running or not before starting tracing. If not, it will just do startup tracing. After tracing is started, Telemetry should ensure browser is running, or start the browser.

So I think shared_page_state.WillRunStory should make sure browser is up and running?
Cc: zh...@chromium.org
Owner: ksakamoto@chromium.org
Hmhh, then it seems like a fix here could be we should add s.t in page_cycler_v2 page to ensure the browser is mostly idle after load. If the browser crashed, the logic of checking browser's idle state will report the crash.

Reassign to ksakamoto@ to add such thing to to page_cycler_v2.intl_ar_fa_he. Maybe we can use https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py?rcl=0&l=138
Project Member

Comment 11 by bugdroid1@chromium.org, Oct 24 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/59ccf0918519486b7555d670af981eb9579532e1

commit 59ccf0918519486b7555d670af981eb9579532e1
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Mon Oct 24 19:41:59 2016

Roll src/third_party/catapult/ 4d43e1baa..dd582ea21 (3 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/4d43e1baaf06..dd582ea2140c

$ git log 4d43e1baa..dd582ea21 --date=short --no-merges --format='%ad %ae %s'
2016-10-24 jbudorick [dependency manager] Add an update script.
2016-10-24 bpastene devil: Change in type of temp dumped in device monitor to float.
2016-10-24 nednguyen Update ChromeTracingAgent exception handling

BUG=649481, 657674 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2444033002
Cr-Commit-Position: refs/heads/master@{#427120}

[modify] https://crrev.com/59ccf0918519486b7555d670af981eb9579532e1/DEPS

Comment 12 by zh...@chromium.org, Nov 15 2016

Status: Fixed (was: Available)

Sign in to add a comment