New issue
Advanced search Search tips

Issue 902466 link

Starred by 1 user
Status: Fixed
Owner:
zhizhouy@chromium.org
Closed: Dec 6
Cc:
llozano@chromium.org
cmt...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug
Build-Toolchain



Sign in to add a comment

Crosperf should handle benchmarks with different test_args 'story-filter'/'story-tag-filter'

Project Member Reported by zhizhouy@chromium.org, Nov 6 
Crosperf will report an error if two benchmarks have same name in experiment file.

But by introducing test_args: 'story-filter'/'story-tag-filter', we want to treat benchmarks with different tag as different benchmarks and run.

Here is an example:
 
benchmark: rendering.desktop {
    iterations: 1
    test_args: --story-tag-filter=top_real_world_desktop
    suite: telemetry_Crosperf
}

benchmark: rendering.desktop {
    iterations: 1
    test_args: --story-tag-filter=tough_canvas
    suite: telemetry_Crosperf
}

We want to run rendering.desktop benchmark twice with different --story-tag-filter. But now crosperf will raise an error since the name of the two benchmarks are the same.
Project Member

Comment 1 by bugdroid1@chromium.org, Nov 9 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/toolchain-utils/+/aa8d023f42fa991587e291e067820b2bbd6031a3

commit aa8d023f42fa991587e291e067820b2bbd6031a3
Author: Zhizhou Yang <zhizhouy@google.com>
Date: Fri Nov 09 19:05:09 2018

Crosperf: Treat benchmarks with story- test_args as different benchmarks

The bug refers to  https://crbug.com/902466 .

This patch modified the behavior of crosperf on determining duplicate
benchmarks. When there are two blocks with same benchmark name in
experiment file, we will append them to benchmark list first. After
that, we will rename the benchmark if it has test_args: 'story-filter'
or 'story-tag-filter'. The new benchmark name will be passed to
experiment so that we will treat them as two different runs and get
separate tables on the report.

BUG= chromium:902466 
TEST=Passed all up-to-date unittests, tested and verified with one
experiment file containing following benchmark settings:
    benchmark: webrtc {
        iterations: 3
        test_args: --story-filter=datachannel
        suite: telemetry_Crosperf
    }
    benchmark: webrtc {
        iterations: 3
        test_args: --story-tag-filter=smoothness
        suite: telemetry_Crosperf
    }

Change-Id: Id733273a5f9f43d149407055c9c0da3b761ddeef
Reviewed-on: https://chromium-review.googlesource.com/1321415
Commit-Ready: Zhizhou Yang <zhizhouy@google.com>
Tested-by: Zhizhou Yang <zhizhouy@google.com>
Reviewed-by: Caroline Tice <cmtice@chromium.org>

[modify] https://crrev.com/aa8d023f42fa991587e291e067820b2bbd6031a3/crosperf/experiment_file.py
[modify] https://crrev.com/aa8d023f42fa991587e291e067820b2bbd6031a3/crosperf/experiment_factory_unittest.py
[modify] https://crrev.com/aa8d023f42fa991587e291e067820b2bbd6031a3/crosperf/experiment_factory.py
[modify] https://crrev.com/aa8d023f42fa991587e291e067820b2bbd6031a3/crosperf/experiment_file_unittest.py
Project Member

Comment 2 by bugdroid1@chromium.org, Nov 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/toolchain-utils/+/69c558b42977054d6168ad13829ebdd80ad77a7c

commit 69c558b42977054d6168ad13829ebdd80ad77a7c
Author: Zhizhou Yang <zhizhouy@google.com>
Date: Thu Nov 15 10:16:59 2018

crosperf: do not allow more than one story tags in test_args

This patch added a check at stage of parsing benchmark, to ensure that
story tag only appear once in test_args. This is because more than one
story tags will cause an implicit failure when running benchmark on DUT
and crosperf cannot catch it.

BUG= chromium:902466 
TEST=Passed unit tests and a simple test

Change-Id: Ic8e4aa19928911fb4ed03529ab89471856cec05c
Reviewed-on: https://chromium-review.googlesource.com/1330010
Commit-Ready: Zhizhou Yang <zhizhouy@google.com>
Tested-by: Zhizhou Yang <zhizhouy@google.com>
Reviewed-by: Caroline Tice <cmtice@chromium.org>

[modify] https://crrev.com/69c558b42977054d6168ad13829ebdd80ad77a7c/crosperf/experiment_factory_unittest.py
[modify] https://crrev.com/69c558b42977054d6168ad13829ebdd80ad77a7c/crosperf/experiment_factory.py

Comment 3 by zhizhouy@chromium.org, Dec 6

Status: Fixed (was: Started)
►

Sign in to add a comment