performance_browser_tests failing on Windows 7 64 bit tester bots on chromium.perf |
|||||||
Issue descriptionperformance_browser_tests has been failing on most windows 7 builders for a while now. It's flakily failing; happens most builds, but not every one. Affected builders: https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20x64%20Perf/ https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20ATI%20GPU%20Perf/ https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20Intel%20GPU%20Perf/ https://luci-milo.appspot.com/buildbot/chromium.perf/Win%207%20Nvidia%20GPU%20Perf/ The log doesn't look super readable (rather spammy with errors that could be unrelated), so I'm not sure what's causing the failure. Some sample logs: https://chromium-swarm.appspot.com/task?id=355cc91703b37f10&refresh=10 https://chromium-swarm.appspot.com/task?id=355ba6fa1f602910&refresh=10&show_raw=1 I do see the following line in the log: [4424:4464:0406/055100.069:INFO:CONSOLE(43)] "Uncaught TypeError: capture_stream.stop is not a function", source: chrome-extension://ddchlicdkolnonkihahngkmmmjnjlkkf/performance.js (43) The file this is probably coming from hasn't been changed recently at all. (https://chromium.googlesource.com/chromium/src/+log/be9dfc362a8b2b90e054b5142c55c28334a6bded/chrome/test/data/extensions/api_test/tab_capture/performance.js)
,
Apr 10 2017
FWIW, I've been trying to diagnose this, but my Win desktop is busted and it's taking a long time to get it reimaged. Please stand by... :)
,
Apr 13 2017
Finally was able to run the tests locally. Only one test failed, but a very obvious error showed an EXPECT failed on line 626 of cast_streaming/performance_test.cc:
[ RUN ] CastV2PerformanceTest.Performance/7
[4636:17544:0412/182132.292:ERROR:process_win.cc(140)] Unable to terminate process: Access is denied. (0x5)
[4636:16276:0412/182132.529:WARNING:cast_transport_impl.cc(159)] Unsafe to send stream with encryption DISABLED.
[4636:16276:0412/182132.531:WARNING:cast_transport_impl.cc(159)] Unsafe to send stream with encryption DISABLED.
[4636:12332:0412/182132.534:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.543:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.553:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.563:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.572:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:12332:0412/182132.584:INFO:performance_test.cc(255)] Failed to decode audio timestamp!
[4636:4360:0412/182132.633:WARNING:video_capture_oracle.cc(271)] Out of order frame delivery detected (have #0, last was #2). Dropping frame.
[4636:12332:0412/182132.670:INFO:performance_test.cc(273)] Failed to decode barcode!
Still waiting for the following processes to finish:
"D:\src\chromium\src\out\Default\performance_browser_tests.exe" --enable-gpu --gtest_also_run_disabled_tests --gtest_filter=CastV2PerformanceTest.Performance/7 --single_process --user-data-dir="D:\src\temp\scoped_dir32056_25124\d32056_10976"
[4636:4360:0412/182152.537:INFO:CONSOLE(0)] "[SUCCESS] sendTestPatterns", source: chrome-extension://ddchlicdkolnonkihahngkmmmjnjlkkf/performance30.html?port=56511 (0)
d:\src\chromium\src\chrome\browser\extensions\api\cast_streaming\performance_test.cc(626): error: Expected: (frame_data.num_values) > (0UL), actual: 0 vs 0
[4636:4360:0412/182152.792:ERROR:performance_test.cc(145)] Not enough events for CastV2Performance_gpu_30fps_fast time_between_frames
*RESULT CastV2Performance_gpu_30fps_fast: time_between_captures= {33.414713,1.894323} ms
*RESULT CastV2Performance_gpu_30fps_fast: av_sync= {8.481681,7.039743} ms
[4636:4360:0412/182152.797:INFO:performance_test.cc(161)] Jitter analyzis on 1995 values.
*RESULT CastV2Performance_gpu_30fps_fast: audio_jitter= {0.000433,0.000215} ms
[4636:4360:0412/182152.797:INFO:performance_test.cc(161)] Jitter analyzis on 595 values.
*RESULT CastV2Performance_gpu_30fps_fast: video_jitter= {0.119331,0.679868} ms
*RESULT CastV2Performance_gpu_30fps_fast: total_latency= {33.992100,3.843812} ms
*RESULT CastV2Performance_gpu_30fps_fast: capture_duration= {10.429500,1.112197} ms
*RESULT CastV2Performance_gpu_30fps_fast: send_to_renderer= {0.278500,0.031834} ms
*RESULT CastV2Performance_gpu_30fps_fast: encode= {7.498267,1.553666} ms
*RESULT CastV2Performance_gpu_30fps_fast: transmit= {0.825633,1.687701} ms
*RESULT CastV2Performance_gpu_30fps_fast: decode= {14.952600,0.909771} ms
*RESULT CastV2Performance_gpu_30fps_fast: cast_latency= {23.284100,3.262347} ms
[4636:4360:0412/182152.868:INFO:user_input_monitor_win.cc(157)] RegisterRawInputDevices() failed for RIDEV_REMOVE: The parameter is incorrect. (0x57)
[ FAILED ] CastV2PerformanceTest.Performance/7, where GetParam() = 529 (21321 ms)
[----------] 1 test from CastV2PerformanceTest (21321 ms total)
,
Apr 13 2017
Correction: All tests failed with the same error message.
,
Apr 16 2017
,
Apr 26 2017
Any update on this? Should we disable these tests while you debug?
,
Apr 26 2017
No, we should keep them running. The failure is only caused because one (of several) perf results is not available. Meaning, the rest are working just fine, so we want to continue monitoring them. I have a WIP change on my Win desktop that fixes the issue. Trying to wrap it up while simultaneously launching a new product feature, hosting visitors, day-long design discussions, etc... Probably in the next week or two.
,
May 4 2017
,
May 6 2017
"Only one test failed, but a very obvious error showed an EXPECT failed on line 626 of cast_streaming/performance_test.cc:" Can you disable that single failed tests for now. This is keep failing on many builders & makes it hard for perf sherrifs to green up the tree.
,
May 6 2017
No, I cannot disable the tests. They are generating good results. :) The failure is basically one (out of 8) measurements that recently broke. I don't want to halt all perf testing for just one measurement, and we also have active alerting on the other measurements. Short-term: I'll put in a hack on Monday to make the bots go green again. Less short-term: I've made good progress today on resolving the root cause of the failure, and on further stabilizing measurements, and otherwise doing a lot of housekeeping on these perf tests. I'm about to get the work code reviewed, so this should be resolved by end of next week.
,
May 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/06c04ee30859f61818280f9c5254d6878452039e commit 06c04ee30859f61818280f9c5254d6878452039e Author: miu <miu@chromium.org> Date: Sat May 06 22:26:32 2017 Temporary fix for failing performance_browser_tests runs. A longer-term fix for the issue is being worked on and will be committed in approx. one week. For now, the purpose of this change is to get the perf bots green again. BUG= 709247 TBR=xjz@chromium.org Review-Url: https://codereview.chromium.org/2868673002 Cr-Commit-Position: refs/heads/master@{#469892} [modify] https://crrev.com/06c04ee30859f61818280f9c5254d6878452039e/chrome/browser/extensions/api/cast_streaming/performance_test.cc
,
May 23 2017
,
May 24 2017
Fixed by change currently almost through the CQ: https://codereview.chromium.org/2866943002/
,
May 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91 commit 50ca2ed80efff90dcc3192c1e3e1c2d782d14a91 Author: miu <miu@chromium.org> Date: Wed May 24 06:55:53 2017 Give performance_browser_tests lots of love and attention. This change started by fixing a bug where a trace event was missing, causing failures on the Win7 bots. It quickly evolved into a number of improvements, focusing on measurement stability, and adding a few additional useful analyses. More-specifically: 1. The logic that tracks a frame in the CastV2PerformanceTest was simplified since exact timestamps are now available in the tracing data (i.e., no need for fabs(TS_a - TS_b) < 1 millisecond" searches for prior events). 2. Trimmed events at both the start and end of the tests, and updated the number of trimmed events, to promote more stable measurements in the perf waterfall. Both CastV2PerformanceTests and TabCapturePerformanceTests now run for 15 seconds (before, this was 20 and 10, respectively). 3. Added new "frame_drop_rate" measurement to CastV2PerformanceTest. Similarly, added a CaptureFailRate measurement to TabCapturePerformanceTest. 4. Added performance testing with auto throttling (aka "zero config") enabled to CastV2PerformanceTest, through simulated "slow, but reliable Wifi." If measurements are stable, we can add monitoring for the average capture resolution (larger is better) and number of resolution changes (lower is better) as way to detect perf regressions that will have a direct impact on the user experience. 5. Enabled encryption on the Cast Streaming transport, to ensure crypto is also being considered when measuring performance and alerting on regressions. 6. Minor code clean-up: style stuff, de-duping utility functions, fixing use of the deprecated MediaStream.stop() call, adding/clarifying comments to account for changes over the past couple of years. Added results: CastV2Performance*: playout_resolution CastV2Performance*: resolution_changes CastV2Performance*: frame_drop_rate TabCapturePerformance*: CaptureLatency TabCapturePerformance*: CaptureFailRate Removed results: TabCapturePerformance*: CaptureSucceeded BUG=567848, 709247 Review-Url: https://codereview.chromium.org/2866943002 Cr-Commit-Position: refs/heads/master@{#474189} [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/browser/extensions/api/cast_streaming/performance_test.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/renderer/media/cast_rtp_stream.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/test/data/extensions/api_test/cast_streaming/performance.js [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/chrome/test/data/extensions/api_test/tab_capture/performance.js [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/capture/content/thread_safe_capture_oracle.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/receiver/cast_receiver_impl.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/sender/video_sender.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/test/utility/udp_proxy.cc [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/test/utility/udp_proxy.h [modify] https://crrev.com/50ca2ed80efff90dcc3192c1e3e1c2d782d14a91/media/cast/test/utility/udp_proxy_main.cc
,
May 30 2017
,
Jan 22 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by martiniss@chromium.org
, Apr 10 2017