New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 709247 link

Starred by 3 users

Issue metadata

Status: Archived
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 722367



Sign in to add a comment

performance_browser_tests failing on Windows 7 64 bit tester bots on chromium.perf

Project Member Reported by martiniss@chromium.org, Apr 6 2017

Issue description

performance_browser_tests has been failing on most windows 7 builders for a while now. It's flakily failing; happens most builds, but not every one. 

Affected builders:
https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20x64%20Perf/
https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20ATI%20GPU%20Perf/
https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20Intel%20GPU%20Perf/
https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20Nvidia%20GPU%20Perf/

The log doesn't look super readable (rather spammy with errors that could be unrelated), so I'm not sure what's causing the failure. Some sample logs:

https://chromium-swarm.appspot.com/task?id=355cc91703b37f10&refresh=10
https://chromium-swarm.appspot.com/task?id=355ba6fa1f602910&refresh=10&show_raw=1


I do see the following line in the log:
[4424:4464:0406/055100.069:INFO:CONSOLE(43)] "Uncaught TypeError: capture_stream.stop is not a function", source: chrome-extension://ddchlicdkolnonkihahngkmmmjnjlkkf/performance.js (43)

The file this is probably coming from hasn't been changed recently at all. (https://chromium.googlesource.com/chromium/src/+log/be9dfc362a8b2b90e054b5142c55c28334a6bded/chrome/test/data/extensions/api_test/tab_capture/performance.js)
 
Cc: -nednguyen@chromium.org nedngu...@google.com

Comment 2 by m...@chromium.org, Apr 10 2017

Cc: -justinlin@chromium.org
Owner: m...@chromium.org
Status: Assigned (was: Available)
FWIW, I've been trying to diagnose this, but my Win desktop is busted and it's taking a long time to get it reimaged. Please stand by...  :)

Comment 3 by m...@chromium.org, Apr 13 2017

Status: Started (was: Assigned)
Finally was able to run the tests locally. Only one test failed, but a very obvious error showed an EXPECT failed on line 626 of cast_streaming/performance_test.cc:

[ RUN      ] CastV2PerformanceTest.Performance/7
[4636:17544:0412/182132.292:ERROR:process_win.cc(140)] Unable to terminate process: Access is denied. (0x5)
[4636:16276:0412/182132.529:WARNING:cast_transport_impl.cc(159)] Unsafe to send stream with encryption DISABLED.
[4636:16276:0412/182132.531:WARNING:cast_transport_impl.cc(159)] Unsafe to send stream with encryption DISABLED.
[4636:12332:0412/182132.534:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.543:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.553:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.563:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.572:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.584:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:4360:0412/182132.633:WARNING:video_capture_oracle.cc(271)] Out of order frame delivery detected (have #0, last was #2).  Dropping frame.
[4636:12332:0412/182132.670:INFO:performance_test.cc(273)] Failed to decode barcode!
Still waiting for the following processes to finish:
        "D:\src\chromium\src\out\Default\performance_browser_tests.exe" --enable-gpu --gtest_also_run_disabled_tests --gtest_filter=CastV2PerformanceTest.Performance/7 --single_process --user-data-dir="D:\src\temp\scoped_dir32056_25124\d32056_10976"
[4636:4360:0412/182152.537:INFO:CONSOLE(0)] "[SUCCESS] sendTestPatterns", source: chrome-extension://ddchlicdkolnonkihahngkmmmjnjlkkf/performance30.html?port=56511 (0)

d:\src\chromium\src\chrome\browser\extensions\api\cast_streaming\performance_test.cc(626): error: Expected: (frame_data.num_values) > (0UL), actual: 0 vs 0
[4636:4360:0412/182152.792:ERROR:performance_test.cc(145)] Not enough events for CastV2Performance_gpu_30fps_fast time_between_frames

*RESULT CastV2Performance_gpu_30fps_fast: time_between_captures= {33.414713,1.894323} ms
*RESULT CastV2Performance_gpu_30fps_fast: av_sync= {8.481681,7.039743} ms
[4636:4360:0412/182152.797:INFO:performance_test.cc(161)] Jitter analyzis on 1995 values.
*RESULT CastV2Performance_gpu_30fps_fast: audio_jitter= {0.000433,0.000215} ms
[4636:4360:0412/182152.797:INFO:performance_test.cc(161)] Jitter analyzis on 595 values.
*RESULT CastV2Performance_gpu_30fps_fast: video_jitter= {0.119331,0.679868} ms
*RESULT CastV2Performance_gpu_30fps_fast: total_latency= {33.992100,3.843812} ms
*RESULT CastV2Performance_gpu_30fps_fast: capture_duration= {10.429500,1.112197} ms
*RESULT CastV2Performance_gpu_30fps_fast: send_to_renderer= {0.278500,0.031834} ms
*RESULT CastV2Performance_gpu_30fps_fast: encode= {7.498267,1.553666} ms
*RESULT CastV2Performance_gpu_30fps_fast: transmit= {0.825633,1.687701} ms
*RESULT CastV2Performance_gpu_30fps_fast: decode= {14.952600,0.909771} ms
*RESULT CastV2Performance_gpu_30fps_fast: cast_latency= {23.284100,3.262347} ms
[4636:4360:0412/182152.868:INFO:user_input_monitor_win.cc(157)] RegisterRawInputDevices() failed for RIDEV_REMOVE: The parameter is incorrect. (0x57)
[  FAILED  ] CastV2PerformanceTest.Performance/7, where GetParam() = 529 (21321 ms)
[----------] 1 test from CastV2PerformanceTest (21321 ms total)


Comment 4 by m...@chromium.org, Apr 13 2017

Labels: M-59
Correction: All tests failed with the same error message.

Comment 5 by m...@chromium.org, Apr 16 2017

Cc: picksi@chromium.org eyaich@chromium.org
 Issue 673356  has been merged into this issue.
Any update on this? Should we disable these tests while you debug?

Comment 7 by m...@chromium.org, Apr 26 2017

No, we should keep them running. The failure is only caused because one (of several) perf results is not available. Meaning, the rest are working just fine, so we want to continue monitoring them.

I have a WIP change on my Win desktop that fixes the issue. Trying to wrap it up while simultaneously launching a new product feature, hosting visitors, day-long design discussions, etc... Probably in the next week or two.

Comment 8 by m...@chromium.org, May 4 2017

Cc: justinlin@chromium.org
 Issue 718289  has been merged into this issue.
"Only one test failed, but a very obvious error showed an EXPECT failed on line 626 of cast_streaming/performance_test.cc:"

Can you disable that single failed tests for now. This is keep failing on many builders & makes it hard for perf sherrifs to green up the tree.

Comment 10 by m...@chromium.org, May 6 2017

No, I cannot disable the tests. They are generating good results. :) The failure is basically one (out of 8) measurements that recently broke. I don't want to halt all perf testing for just one measurement, and we also have active alerting on the other measurements.

Short-term: I'll put in a hack on Monday to make the bots go green again.

Less short-term: I've made good progress today on resolving the root cause of the failure, and on further stabilizing measurements, and otherwise doing a lot of housekeeping on these perf tests. I'm about to get the work code reviewed, so this should be resolved by end of next week.
Project Member

Comment 11 by bugdroid1@chromium.org, May 6 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/06c04ee30859f61818280f9c5254d6878452039e

commit 06c04ee30859f61818280f9c5254d6878452039e
Author: miu <miu@chromium.org>
Date: Sat May 06 22:26:32 2017

Temporary fix for failing performance_browser_tests runs.

A longer-term fix for the issue is being worked on and will be committed
in approx. one week. For now, the purpose of this change is to get the
perf bots green again.

BUG= 709247 
TBR=xjz@chromium.org

Review-Url: https://codereview.chromium.org/2868673002
Cr-Commit-Position: refs/heads/master@{#469892}

[modify] https://crrev.com/06c04ee30859f61818280f9c5254d6878452039e/chrome/browser/extensions/api/cast_streaming/performance_test.cc

Comment 12 by m...@chromium.org, May 23 2017

Blocking: 722367

Comment 13 by m...@chromium.org, May 24 2017

Fixed by change currently almost through the CQ: https://codereview.chromium.org/2866943002/

Project Member

Comment 14 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91

commit 50ca2ed80efff90dcc3192c1e3e1c2d782d14a91
Author: miu <miu@chromium.org>
Date: Wed May 24 06:55:53 2017

Give performance_browser_tests lots of love and attention.

This change started by fixing a bug where a trace event was missing,
causing failures on the Win7 bots. It quickly evolved into a number of
improvements, focusing on measurement stability, and adding a few
additional useful analyses. More-specifically:

1. The logic that tracks a frame in the CastV2PerformanceTest was
simplified since exact timestamps are now available in the tracing data
(i.e., no need for fabs(TS_a - TS_b) < 1 millisecond" searches for prior
events).

2. Trimmed events at both the start and end of the tests, and updated
the number of trimmed events, to promote more stable measurements in the
perf waterfall. Both CastV2PerformanceTests and
TabCapturePerformanceTests now run for 15 seconds (before, this was 20
and 10, respectively).

3. Added new "frame_drop_rate" measurement to CastV2PerformanceTest.
Similarly, added a CaptureFailRate measurement to
TabCapturePerformanceTest.

4. Added performance testing with auto throttling (aka "zero config")
enabled to CastV2PerformanceTest, through simulated "slow, but reliable
Wifi." If measurements are stable, we can add monitoring for the average
capture resolution (larger is better) and number of resolution changes
(lower is better) as way to detect perf regressions that will have a
direct impact on the user experience.

5. Enabled encryption on the Cast Streaming transport, to ensure crypto
is also being considered when measuring performance and alerting on
regressions.

6. Minor code clean-up: style stuff, de-duping utility functions, fixing
use of the deprecated MediaStream.stop() call, adding/clarifying
comments to account for changes over the past couple of years.

Added results:
  CastV2Performance*: playout_resolution
  CastV2Performance*: resolution_changes
  CastV2Performance*: frame_drop_rate
  TabCapturePerformance*: CaptureLatency
  TabCapturePerformance*: CaptureFailRate

Removed results:
  TabCapturePerformance*: CaptureSucceeded

BUG=567848, 709247 

Review-Url: https://codereview.chromium.org/2866943002
Cr-Commit-Position: refs/heads/master@{#474189}

[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/browser/extensions/api/cast_streaming/performance_test.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/renderer/media/cast_rtp_stream.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/test/data/extensions/api_test/cast_streaming/performance.js
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/test/data/extensions/api_test/tab_capture/performance.js
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/capture/content/thread_safe_capture_oracle.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/receiver/cast_receiver_impl.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/sender/video_sender.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/test/utility/udp_proxy.cc
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/test/utility/udp_proxy.h
[modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/test/utility/udp_proxy_main.cc

Comment 15 by m...@chromium.org, May 30 2017

Status: Fixed (was: Started)

Comment 16 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment