video_PlaybackPerf [1] is a test that play a video with Chrome and reports performance with/without HW acceleration. We migrate this to a new test framework, Tast [2]. [1] https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/client/site_tests/video_PlaybackPerf [2] https://chromium.googlesource.com/chromiumos/platform/tast/
I upload WIP CL, https://chromium-review.googlesource.com/c/chromiumos/platform/tast-tests/+/1253581
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/8c909c2c60d5e446b52438df0f3e79b2d5c97b55 commit 8c909c2c60d5e446b52438df0f3e79b2d5c97b55 Author: Hirokazu Honda <hiroh@chromium.org> Date: Thu Oct 04 00:38:03 2018 Add video data for video.PlaybackPerfH264 BUG= chromium:890733 TEST=video.PlaybackPerfH264 Change-Id: Ie1e702a0d3846b46d7c178bc0eb368f0cd89f767 Reviewed-on: https://chromium-review.googlesource.com/1256643 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/8c909c2c60d5e446b52438df0f3e79b2d5c97b55/chromeos-base/tast-local-tests-cros/tast-local-tests-cros-9999.ebuild [modify] https://crrev.com/8c909c2c60d5e446b52438df0f3e79b2d5c97b55/chromeos-base/tast-local-tests-cros/Manifest [modify] https://crrev.com/8c909c2c60d5e446b52438df0f3e79b2d5c97b55/chromeos-base/tast-local-tests-cros/files/external_data.conf
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/1b5e90295234e09831409fc8972325def715ea99 commit 1b5e90295234e09831409fc8972325def715ea99 Author: Keiichi Watanabe <keiichiw@chromium.org> Date: Thu Oct 18 05:43:15 2018 Add Tast informational tests to performance dashboard BUG= chromium:882433 , chromium:890733 TEST=None Change-Id: Ie9fdcc020072625f70408d5b52e22f4105976414 Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1282144 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/1b5e90295234e09831409fc8972325def715ea99/tko/perf_upload/perf_dashboard_config.json
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/c8edaafd0a2e32ba252631f978c3a2910cc9bae8 commit c8edaafd0a2e32ba252631f978c3a2910cc9bae8 Author: Hirokazu Honda <hiroh@chromium.org> Date: Fri Oct 19 05:37:59 2018 video: Add PlaybackPerfH264 to Tast video_PlaybackPerfH264 is video_PlaybackPerf.h264 in autotest. It measures SW/HW playback performance and reports the performance values on dashboard. This is the first step to migrate video_PlaybackPerf. It measures the number of dropped frames and the rate of dropped frames. BUG= chromium:890733 TEST=video.PlaybackPerfH264 on veyron_minnie Change-Id: I331442d72d5bfe0a589d4e61a0f2fe5d4466c651 Reviewed-on: https://chromium-review.googlesource.com/1249401 Commit-Ready: Hirokazu Honda <hiroh@chromium.org> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/c8edaafd0a2e32ba252631f978c3a2910cc9bae8/src/chromiumos/tast/local/bundles/cros/video/lib/logging/log_helper.go [add] https://crrev.com/c8edaafd0a2e32ba252631f978c3a2910cc9bae8/src/chromiumos/tast/local/bundles/cros/video/playback_perf_h264.go [add] https://crrev.com/c8edaafd0a2e32ba252631f978c3a2910cc9bae8/src/chromiumos/tast/local/bundles/cros/video/lib/constants/constants.go [add] https://crrev.com/c8edaafd0a2e32ba252631f978c3a2910cc9bae8/src/chromiumos/tast/local/bundles/cros/video/playback/playback.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f541f7b3facf06975bfc18264e4fa4b85abc642f commit f541f7b3facf06975bfc18264e4fa4b85abc642f Author: Hirokazu Honda <hiroh@chromium.org> Date: Thu Oct 25 11:26:38 2018 video.PlaybackPerf: Wait until video element is loaded in page Formerly, video.PlaybackPerf can have tried to access a video element before it is loaded. It fails the test due to the failure on accessing the video element. This change fixes the issue as the test waits for the video element loading. BUG= chromium:897588 , chromium:890733 TEST=video.PlaybackPerfH264 Change-Id: I285c62e0819f284f73f977fa3f29ac9aca9db3c8 Reviewed-on: https://chromium-review.googlesource.com/1297795 Commit-Ready: Hirokazu Honda <hiroh@chromium.org> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/f541f7b3facf06975bfc18264e4fa4b85abc642f/src/chromiumos/tast/local/bundles/cros/video/playback/playback.go
video.PlaybackPerfH264 seems flaky. I have to analyze the reasons. https://stainless.corp.google.com/search?view=list&first_date=2018-10-27&last_date=2018-10-29&test=tast.video.Playback&status=WARN&status=FAIL&status=ERROR&status=ABORT&status=ALERT&exclude_cts=false&exclude_not_run=false&exclude_non_release=false&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/8ac0621bb7738fcc6180d7af0e40e4d4a82409fe commit 8ac0621bb7738fcc6180d7af0e40e4d4a82409fe Author: Hirokazu Honda <hiroh@chromium.org> Date: Wed Oct 31 15:17:33 2018 video.PlaybackPerf*: Measure CPU usage Besides decoded frames and percent, video.PlaybackPerf measures cpu usage and reports the value on perf dashboard. BUG= chromium:890733 TEST=video.PlaybackPerfH264 on kevin Change-Id: Ide0a83811cab3af0468220505c828a4467bcbcb5 Reviewed-on: https://chromium-review.googlesource.com/1303776 Commit-Ready: Hirokazu Honda <hiroh@chromium.org> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Hirokazu Honda <hiroh@chromium.org> [add] https://crrev.com/8ac0621bb7738fcc6180d7af0e40e4d4a82409fe/src/chromiumos/tast/local/bundles/cros/video/lib/cpu/cpu.go [modify] https://crrev.com/8ac0621bb7738fcc6180d7af0e40e4d4a82409fe/src/chromiumos/tast/local/bundles/cros/video/playback/playback.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/fef0836f589c87b511ffc2ddd5bc136bda6fbceb commit fef0836f589c87b511ffc2ddd5bc136bda6fbceb Author: Hirokazu Honda <hiroh@chromium.org> Date: Wed Oct 31 15:17:30 2018 video.PlaybackPerfH264: Lengthen timeout to 3 minutes Tast default timeout is two minutes. In video.PlaybackPerfH264, video playes for 30 seconds two times. Therefore, 1 minutes are remained for other tasks, e.g., loging chrome and getting histogram values. It turns out from log results that 1 minutes is not enough for the tasks on low-end devices. This change lengthen test timeout to 3 minutes. BUG= chromium:890733 TEST=video.PlaybackPerfH264 on daisy Change-Id: Iabb82c67c47f0ab28373a596a1227511215b39a7 Reviewed-on: https://chromium-review.googlesource.com/1307175 Commit-Ready: Hirokazu Honda <hiroh@chromium.org> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/fef0836f589c87b511ffc2ddd5bc136bda6fbceb/src/chromiumos/tast/local/bundles/cros/video/playback_perf_h264.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/6a169c20485294fdeb32472b9b103d9cbbe0d5c1 commit 6a169c20485294fdeb32472b9b103d9cbbe0d5c1 Author: Hirokazu Honda <hiroh@chromium.org> Date: Sat Dec 29 00:14:04 2018 video: Add functions to Mute/Unmute not to make a sound during testing Videos in some tests has an audio. It doesn't matter whether DUT sounds or not. It is convenient for developers that test automatically makes DUT mute during the test. Note that it matters that chrome processes for sound, because not only we want to measure metrics in chrome standard video playback but also chrome video playback behavior is changed by the existence of sound in the video. BUG= chromium:890733 , chromium:911095 TEST=tast run video.Play* Change-Id: Ibe4895c1cfc6991beaf5672724bb8c9bf15f194c Reviewed-on: https://chromium-review.googlesource.com/1390677 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/6a169c20485294fdeb32472b9b103d9cbbe0d5c1/src/chromiumos/tast/local/bundles/cros/video/lib/audio/audio.go [modify] https://crrev.com/6a169c20485294fdeb32472b9b103d9cbbe0d5c1/src/chromiumos/tast/local/bundles/cros/video/playback/playback.go [modify] https://crrev.com/6a169c20485294fdeb32472b9b103d9cbbe0d5c1/src/chromiumos/tast/local/bundles/cros/video/play/play.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/a8f02b2a2ff6c9111045402af29cf7f75db45915 commit a8f02b2a2ff6c9111045402af29cf7f75db45915 Author: Hirokazu Honda <hiroh@chromium.org> Date: Sat Dec 29 00:14:07 2018 video: Add more tests and videos for tast.video.PlaybackPerf This CL adds the following videos for video playback performance test; * H264 1080P@30FPS * H264 1080P@60FPS * H264 2160P@30FPS * H264 2160P@60FPS * VP8 1080P@30FPS * VP8 1080P@60FPS * VP8 2160P@30FPS * VP8 2160P@60FPS * VP9 1080P@30FPS * VP9 1080P@60FPS * VP9 2160P@30FPS * VP9 2160P@60FPS All of them are 10 seconds video and have sound. BUG= chromium:890733 TEST=tast run video.PlaybackPerf* Change-Id: I0d2db1050922c6bab569c83a9f2f8afba91aa6fc Reviewed-on: https://chromium-review.googlesource.com/1390678 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Hirokazu Honda <hiroh@chromium.org> Reviewed-by: Hirokazu Honda <hiroh@chromium.org> [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp8_2160p_30fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp9_1080p_30fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp9_2160p_60fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp8_2160p_60fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/2160p_30fps_300frames.vp8.webm.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/1080p_30fps_300frames.vp9.webm.external [rename] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_h264_2160p_30fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/1080p_30fps_300frames.h264.mp4.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp8_1080p_60fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp9_1080p_60fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/1080p_60fps_600frames.vp8.webm.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/2160p_30fps_300frames.vp9.webm.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/1080p_60fps_600frames.vp9.webm.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/2160p_60fps_600frames.vp8.webm.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp9_2160p_30fps.go [delete] https://crrev.com/6a169c20485294fdeb32472b9b103d9cbbe0d5c1/src/chromiumos/tast/local/bundles/cros/video/data/traffic-1920x1080-8005020218f6b86bfa978e550d04956e.mp4.external [copy] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_h264_1080p_30fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/1080p_60fps_600frames.h264.mp4.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/2160p_30fps_300frames.h264.mp4.external [copy] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_h264_2160p_60fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/2160p_60fps_600frames.vp9.webm.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/2160p_60fps_600frames.h264.mp4.external [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_vp8_1080p_30fps.go [copy] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/playback_perf_h264_1080p_60fps.go [add] https://crrev.com/a8f02b2a2ff6c9111045402af29cf7f75db45915/src/chromiumos/tast/local/bundles/cros/video/data/1080p_30fps_300frames.vp8.webm.external
Looks good. Close. https://stainless.corp.google.com/search?view=matrix&row=board&col=build&first_date=2018-12-25&last_date=2018-12-31&test=tast.video.PlaybackPerf&exclude_cts=false&exclude_not_run=false&exclude_non_release=false&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false
Comment 1 by hiroh@chromium.org
, Oct 1