New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: 2018-12-11
OS: ----
Pri: 2
Type: ----



Sign in to add a comment
link

Issue 911214: system_health.common_desktop/multitab:misc:typical24 and 1 other(s) in performance_test_suite failing on chromium.perf/win-10-perf

Reported by sheriff-...@appspot.gserviceaccount.com, Dec 3 Project Member

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of oysteine@google.com

system_health.common_desktop/multitab:misc:typical24 and 1 other(s) in performance_test_suite failing on chromium.perf/win-10-perf

Builders failed on: 
- win-10-perf: 
  https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf
 

Comment 1 by oysteine@chromium.org, Dec 3

Cc: perezju@chromium.org
INFO) 2018-12-03 07:17:30,061 trace_data.Serialize:192  Trace sizes in bytes: {'cpuSnapshots': 5044961L, 'traceEvents': 1010115312L, 'telemetry': 739778L}

(INFO) 2018-12-03 07:17:56,525 trace_data.Serialize:201  trace2html finished in 26.46 seconds.

(INFO) 2018-12-03 07:17:56,526 timeline_based_measurement._ComputeTimelineBasedMetrics:317  Starting to compute metrics on trace

(INFO) 2018-12-03 07:38:11,378 timeline_based_measurement._ComputeTimelineBasedMetrics:323  Processing resulting traces took 1214.853 seconds

(ERROR) 2018-12-03 07:38:11,378 page_test_results.Fail:587  Failure recorded: RangeError: Maximum call stack size exceeded

    at Function.findToplevelSchedulerTasks (/tracing/extras/chrome/event_finder_utils.html:221:13)

    at getTasks (/tracing/metrics/system_health/expected_queueing_time_metric.html:121:38)

    at addExpectedQueueingTimeMetric_ (/tracing/metrics/system_health/expected_queueing_time_metric.html:144:21)

    at new expectedQueueingTimeMetric (/tracing/metrics/system_health/expected_queueing_time_metric.html:86:5)

    at runMetrics (c:\b\s\w\ir\third_party\catapult\tracing\tracing\metrics\metric_map_function.html:61:16)

    at metricMapFunction (c:\b\s\w\ir\third_party\catapult\tracing\tracing\metrics\metric_map_function.html:190:24)

    at Object.mapSingleTrace (/tracing/mre/map_single_trace.html:39:7)

    at eval (c:\b\s\w\ir\third_party\catapult\tracing\tracing\mre\map_single_trace_cmdline.html:61:18)

    at Object.runAndConvertErrorsToFailures (/tracing/mre/map_single_trace.html:24:10)

    at mapSingleTraceWithResult (c:\b\s\w\ir\third_party\catapult\tracing\tracing\mre\map_single_trace_cmdline.html:52:12)



perezju: the traces here at 1GB, that seems a little high? :)

Comment 2 by perezju@google.com, Dec 4

Cc: cbruni@chromium.org vovoy@chromium.org
Components: Speed>Benchmarks>Waterfall
multitab:misc:typical24 has a lot of tabs, so yeah, no wonder. :)

+vovoy and +cbruni who created the original story and did the recent :2108 refresh of it.

What are the alternatives here? Reduce the number of tabs? Can the size of the trace still be reduced?

Comment 3 by vovoy@chromium.org, Dec 4

Cc: cywang@chromium.org
Owner: vovoy@chromium.org
I will check if the trace size can be reduced.

Comment 4 by cbruni@google.com, Dec 4

We could reduce the number of tabs, but that might be a very good stress test for the chrome itself.

Comment 5 by oysteine@chromium.org, Dec 6

Note that just processing these traces on the Telemetry side takes 20 minutes. If we can constrain these tests to collecting some smaller subset of trace data that would definitely help a lot.

Comment 6 by oysteine@chromium.org, Dec 6

An alternative could be to move it out of common_desktop and make a separate multitab_desktop test?

Comment 7 by vovoy@chromium.org, Dec 6

I think making a separate multitab_desktop test is a sane solution. multitab:misc:typical24 should require only the tracingMetric trace data.

Comment 8 by vovoy@chromium.org, Dec 6

Owner: ----
As the solution may require changing system_health tests. I might not be the right owner. (You could assign it to me if you think I am the right owner.)

Comment 9 by perezju@chromium.org, Dec 6

Cc: nedngu...@google.com tdres...@chromium.org
Components: Speed>Benchmarks
+Speed>Benchmarks and +nednguyen/+tdresser explicitly for queries on #6/#7 re adding a new benchmark.

Comment 10 by nedngu...@google.com, Dec 6

Cc: crouleau@chromium.org
I would rather reduce the number of tabs than moving it to a separate benchmark. IMO, multi-tab test cases is very important of Chrome browser & thus should be in system health benchmark

Comment 11 by crouleau@chromium.org, Dec 6

Ned, is there any way to have some stories collect different trace data than others? I guess probably there is not. In that case, there is going to be tension between configurability and http://bit.ly/why-merge-benchmarks.

I'm wondering why this just now started failing. It used to work, right?

Comment 12 by bugdroid1@chromium.org, Dec 7

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5a14fdf87276e7460ccd4d18a5eba301514bd3ad

commit 5a14fdf87276e7460ccd4d18a5eba301514bd3ad
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Fri Dec 07 01:32:40 2018

Disable system_health.common_desktop/multitab on Win 10

NOTRY=true

Bug: 911214
Change-Id: Ia853c50260f4a46835818ef4b364b4ee97091244
Reviewed-on: https://chromium-review.googlesource.com/c/1367109
Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614566}
[modify] https://crrev.com/5a14fdf87276e7460ccd4d18a5eba301514bd3ad/tools/perf/expectations.config

Comment 13 by perezju@google.com, Dec 7

> I'm wondering why this just now started failing. It used to work, right?

I guess it's related to the switch to perfetto for tracing backend, which has caused an increase in trace size (and it particularly magnifies in this story). I believe oysteine has been working on addressing the trace size issue in general.

Comment 14 by crouleau@chromium.org, Dec 7

Could we switch back off of perfetto until it is ready to support this use case? I don't feel good about breaking peoples' tests. Perhaps someone can link to the design/launch/implementation bugs for perfetto?

Comment 15 by oysteine@chromium.org, Dec 7

Perfetto increased trace sizes by about 30% temporarily for traces without memory-infra dumps but is now back to normal as of  https://chromium-review.googlesource.com/c/chromium/src/+/1359304

Perfetto has been enabled for well over a month though, so it wouldn't be the cause of recent failures. If something else caused the trace sizes here to increase recently though, the CL above may have taken it below the failure threshold again so could be worth re-enabling the test to see?

Comment 16 by perezju@chromium.org, Dec 10

Let's give that a try!

Comment 17 by bugdroid1@chromium.org, Dec 10

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec24b70caddcb99a897ab89a5bbf60b3ca621349

commit ec24b70caddcb99a897ab89a5bbf60b3ca621349
Author: Juan Antonio Navarro Pérez <perezju@chromium.org>
Date: Mon Dec 10 11:51:20 2018

Revert "Disable system_health.common_desktop/multitab on Win 10"

This reverts commit 5a14fdf87276e7460ccd4d18a5eba301514bd3ad.

Reason for revert: Root cause may have been fixed by:
https://chromium-review.googlesource.com/c/chromium/src/+/1359304

Original change's description:
> Disable system_health.common_desktop/multitab on Win 10
> 
> NOTRY=true
> 
> Bug: 911214
> Change-Id: Ia853c50260f4a46835818ef4b364b4ee97091244
> Reviewed-on: https://chromium-review.googlesource.com/c/1367109
> Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#614566}

TBR=sullivan@chromium.org,tdresser@chromium.org,charliea@chromium.org,crouleau@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

TBR=nednguyen@google.com
NOTRY=true
Bug: 911214

Change-Id: I0835b698da2cb3f2aec50d12305ef23ed9842295
Reviewed-on: https://chromium-review.googlesource.com/c/1369793
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#615092}
[modify] https://crrev.com/ec24b70caddcb99a897ab89a5bbf60b3ca621349/tools/perf/expectations.config

Comment 18 by perezju@chromium.org, Dec 10

NextAction: 2018-12-11

Comment 19 by monor...@bugs.chromium.org, Dec 11

The NextAction date has arrived: 2018-12-11

Comment 20 by oysteine@chromium.org, Dec 11

Still failing:

(INFO) 2018-12-11 06:02:22,746 trace_data.Serialize:192  Trace sizes in bytes: {'cpuSnapshots': 6822051L, 'traceEvents': 754940138L, 'telemetry': 827447L}

(INFO) 2018-12-11 06:02:45,730 trace_data.Serialize:201  trace2html finished in 22.98 seconds.

(INFO) 2018-12-11 06:02:45,730 timeline_based_measurement._ComputeTimelineBasedMetrics:317  Starting to compute metrics on trace

(INFO) 2018-12-11 06:11:59,328 timeline_based_measurement._ComputeTimelineBasedMetrics:323  Processing resulting traces took 553.598 seconds

(ERROR) 2018-12-11 06:11:59,328 page_test_results.Fail:587  Failure recorded: RangeError: Maximum call stack size exceeded

    at Function.findToplevelSchedulerTasks (/tracing/extras/chrome/event_finder_utils.html:221:13)

    at getTasks (/tracing/metrics/system_health/expected_queueing_time_metric.html:121:38)

    at addExpectedQueueingTimeMetric_ (/tracing/metrics/system_health/expected_queueing_time_metric.html:144:21)

    at new expectedQueueingTimeMetric (/tracing/metrics/system_health/expected_queueing_time_metric.html:86:5)

    at runMetrics (c:\b\s\w\ir\third_party\catapult\tracing\tracing\metrics\metric_map_function.html:61:16)

    at metricMapFunction (c:\b\s\w\ir\third_party\catapult\tracing\tracing\metrics\metric_map_function.html:190:24)

    at Object.mapSingleTrace (/tracing/mre/map_single_trace.html:39:7)

    at eval (c:\b\s\w\ir\third_party\catapult\tracing\tracing\mre\map_single_trace_cmdline.html:61:18)

    at Object.runAndConvertErrorsToFailures (/tracing/mre/map_single_trace.html:24:10)

    at mapSingleTraceWithResult (c:\b\s\w\ir\third_party\catapult\tracing\tracing\mre\map_single_trace_cmdline.html:52:12)

(INFO) 2018-12-11 06:11:59,447 browser.Close:209  Closing browser (pid=2300) ...

(INFO) 2018-12-11 06:12:01,237 desktop_browser_backend._TryCooperativeShutdown:561  Successfully shut down browser cooperatively

(INFO) 2018-12-11 06:12:01,237 browser.Close:223  Browser is closed.

[  FAILED  ] system_health.common_desktop/multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'} (798870 ms)

Comment 21 by perezju@chromium.org, Dec 13

Too bad :( Let's disable again then.

Comment 22 by bugdroid1@chromium.org, Dec 13

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d136542c0334c29b3531c6d9c400a394e8ec8e85

commit d136542c0334c29b3531c6d9c400a394e8ec8e85
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Thu Dec 13 11:19:42 2018

Reland "Disable system_health.common_desktop/multitab on Win 10"

This is a reland of 5a14fdf87276e7460ccd4d18a5eba301514bd3ad

Original change's description:
> Disable system_health.common_desktop/multitab on Win 10
>
> NOTRY=true
>
> Bug: 911214
> Change-Id: Ia853c50260f4a46835818ef4b364b4ee97091244
> Reviewed-on: https://chromium-review.googlesource.com/c/1367109
> Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#614566}

TBR=nednguyen@google.com
NOTRY=true

Bug: 911214
Change-Id: I67dc1d3f3b1c797e23a580264726a1b9ffd1884a
Reviewed-on: https://chromium-review.googlesource.com/c/1375710
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616268}
[modify] https://crrev.com/d136542c0334c29b3531c6d9c400a394e8ec8e85/tools/perf/expectations.config

Comment 23 by bugdroid, Feb 21 (2 days ago)

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/17c7599b8f3176c176bf01ef89cfc982ef81aea1

commit 17c7599b8f3176c176bf01ef89cfc982ef81aea1
Author: Annie Sullivan <sullivan@chromium.org>
Date: Thu Feb 21 15:08:14 2019

Disable multitab:misc:typical24 on Windows

Bug: 911214
Change-Id: I68beca365e54029ee6bbb44dfbf69470de468ac3
TBR: crouleau@chromium.org
NOTRY: true
Reviewed-on: https://chromium-review.googlesource.com/c/1481057
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Annie Sullivan <sullivan@chromium.org>
Auto-Submit: Annie Sullivan <sullivan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#634191}
[modify] https://crrev.com/17c7599b8f3176c176bf01ef89cfc982ef81aea1/tools/perf/expectations.config

Comment 24 by nedngu...@google.com, Feb 21 (2 days ago)

Cc: -nedngu...@google.com

Sign in to add a comment