New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 772694 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Oct 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Task



Sign in to add a comment

Move video_VDAPerf and video_PlaybackPerf test runs from nightly to per build

Project Member Reported by hiroh@chromium.org, Oct 8 2017

Issue description

VDAPerf 's suite setting was changed few years ago ( crbug.com/537282 ), because they are not on dashboard.
video_PlaybackPerf's suite setting is nightly too.
They are on dashboard now and the metrics measured by the tests are important, which these regression are not allowed on the present ChromeOS.
Let us change the suite settings from crosbolt_perf_nightly to crosbolt_perf_perbuild.
It is necessary to make sure that the change doesn't affect suite scheduler too much.
 

Comment 1 by hiroh@chromium.org, Oct 17 2017

Cc: bccheng@chromium.org wuchengli@chromium.org posciak@chromium.org
Status: Started (was: Untriaged)
The number of nightly tests are 5 tests for video_PlaybackPerf and 6 tests for video_VDAPerf.
Each test of video_PlaybackPerf takes about 6 minutes on most boards.
(I found it takes about 10 minutes on link, but I guess it won't be the case since two tests run on link.)
Each test of video_VDAPef takes about 2 minutes.
If these tests are moved to perbuild, the perbuild run time will increase by 45 minutes at most.
As far as I checked, the run time of executing routing jobs are short enough that adding them doesn't matter.
That is definitely my expectation though.
It seems to be hard to make 100% sure that this moving doesn't cause sticking many tests in the job queue, because the run time depends on much things (user manual job, available DUTs on lab, a test timeout).
Therefore, let us move them experimentally, and move back again only if the problem happens.
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 17 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/81fe1afb25dedf39ce53759b0a49deae538ad5d2

commit 81fe1afb25dedf39ce53759b0a49deae538ad5d2
Author: Hirokazu Honda <hiroh@chromium.org>
Date: Tue Oct 17 13:54:23 2017

autotest: Move video_VDAPerf and video_PlaybackPerf test runs from nightly to per build

video_VDAPerf and video_PlaybackPerf are on dashboard.
The metrics measured by the tests are important, which these regressions
are not allowed on the present ChromeOS.
Let us change their suite settings from crosbolt_perf_nightly to
crosbolt_perf_perbuild to increase run frequency to improve the test coverage.

BUG= chromium:772694 
TEST=None

Change-Id: I3281dd21714f986c809f9d9642513642c985cb0e
Reviewed-on: https://chromium-review.googlesource.com/721223
Commit-Ready: Hirokazu Honda <hiroh@chromium.org>
Tested-by: Hirokazu Honda <hiroh@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Owen Lin <owenlin@chromium.org>
Reviewed-by: Ben Cheng <bccheng@chromium.org>

[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_PlaybackPerf/control.h264.4k
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_PlaybackPerf/control.vp9.hfr
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_PlaybackPerf/control.vp9.4k.60
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_PlaybackPerf/control.vp9.4k
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_PlaybackPerf/control.vp8.4k
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_VDAPerf/control.vp8
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_VDAPerf/control.h264
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_VDAPerf/control.vp9.4k
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_VDAPerf/control.h264.4k
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_VDAPerf/control.vp8.4k
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_VDAPerf/control.vp9
[modify] https://crrev.com/81fe1afb25dedf39ce53759b0a49deae538ad5d2/client/site_tests/video_PlaybackPerf/control.vp8

Comment 3 by hiroh@chromium.org, Oct 18 2017

The above CL was landed to ChromeOS 10043.0.0.

Comment 4 by hiroh@chromium.org, Oct 19 2017

Quick update; One day have passed since these tests are moved to perbuild.
Looks any timeout doesn't happen because of this moving.
The number of PlaybackPerf and VDAPerf executed tests are stabilized on all the boards.
I continue monitoring carefully in one week though.

Comment 5 by hiroh@chromium.org, Oct 26 2017

Status: Verified (was: Started)
I has been looking for 1 week.
Looks like any timeout hasn't happened by this moving.

Sign in to add a comment