New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 839491 link

Starred by 3 users
Status: Assigned
Owner:
█ stephanstross@google.com
Last visit > 30 days ago
Cc:
sergeyu@chromium.org
w...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 2
Type: Task



Sign in to add a comment

Comparative Benchmarking tool for Fuchsia

Project Member Reported by stephanstross@google.com, May 3 2018 
Software is needed to automate (or at least remove the most tedious aspects of) the comparative benchmarking of Chromium in Fuchsia vs other operating systems.

It needs to be able to:

1) Produce performance numbers for Chromium test suites
2) Compare performance of test suites, parameterized by OS versions
3) Visualize that information for easy overview by developers.
$) Detect performance regressions
Project Member

Comment 1 by bugdroid1@chromium.org, May 14 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/10782a74a1902d5e7dfec29ca0573319cc726e22

commit 10782a74a1902d5e7dfec29ca0573319cc726e22
Author: Stephan Stross <stephanstross@google.com>
Date: Mon May 14 17:30:58 2018

Added script to run tests and compare their outputs on linux and fuchsia NUCs.

comparative_tester.py contains the initial comparative testing script, designed
to accept a list of test targets, compile and build the targets(and generate
them with default args using gn if necessary), build them using ninja and goma,
and then execute them, sending targets one at a time to both linux and fuchsia
NUCs to be executed. Once a test is completed, its output is filtered for
informational lines, they are collated, and written to csv.

Bug: 839491

Change-Id: I287470ae9f50f169fbd1d9296c69ee9560318837
Reviewed-on: https://chromium-review.googlesource.com/1031294
Commit-Queue: Stephan Stross <stephanstross@google.com>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558362}
[add] https://crrev.com/10782a74a1902d5e7dfec29ca0573319cc726e22/tools/fuchsia/comparative_tester/comparative_tester.py
[add] https://crrev.com/10782a74a1902d5e7dfec29ca0573319cc726e22/tools/fuchsia/comparative_tester/target_spec.py
Project Member

Comment 2 by bugdroid1@chromium.org, May 25 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a974627227f417325c87948243897f4a76c1c29b

commit a974627227f417325c87948243897f4a76c1c29b
Author: Stephan Stross <stephanstross@google.com>
Date: Fri May 25 23:35:07 2018

[Fuchsia] Update comparative_tester.py to use filter test files.

This change alters the way filters are ingested and used by the
comparative_tester.py script, to be more flexible, and more in-line
with how trybot filter files are specified.

Bug: 839491
Change-Id: Ib8bc55f6b5d6697581ad9659d824abdc31b60be6
Reviewed-on: https://chromium-review.googlesource.com/1067685
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Commit-Queue: Stephan Stross <stephanstross@google.com>
Cr-Commit-Position: refs/heads/master@{#562062}
[modify] https://crrev.com/a974627227f417325c87948243897f4a76c1c29b/tools/fuchsia/comparative_tester/comparative_tester.py
[modify] https://crrev.com/a974627227f417325c87948243897f4a76c1c29b/tools/fuchsia/comparative_tester/target_spec.py
Project Member

Comment 3 by bugdroid1@chromium.org, Jun 12 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/00e41b5c88db19d6f5b0ed8c083d672992d1186a

commit 00e41b5c88db19d6f5b0ed8c083d672992d1186a
Author: Stephan Stross <stephanstross@google.com>
Date: Tue Jun 12 01:36:47 2018

[Fuchsia] Switch stat-generation over to JSON output

This CL alters the python scripts in tools/fuchsia/comparative_tester
to interpret the stdout of the GTest suites for relevant statistical
output, and then parsing it into a JSON file for further stats-work

Bug: 839491
Change-Id: I9836cce70005f49172487b50d425265857113c72
Reviewed-on: https://chromium-review.googlesource.com/1072465
Commit-Queue: Stephan Stross <stephanstross@google.com>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566227}
[add] https://crrev.com/00e41b5c88db19d6f5b0ed8c083d672992d1186a/testing/buildbot/filters/fuchsia.base_perftests.filter
[add] https://crrev.com/00e41b5c88db19d6f5b0ed8c083d672992d1186a/testing/buildbot/filters/fuchsia.net_perftests.filter
[modify] https://crrev.com/00e41b5c88db19d6f5b0ed8c083d672992d1186a/tools/fuchsia/comparative_tester/comparative_tester.py
[modify] https://crrev.com/00e41b5c88db19d6f5b0ed8c083d672992d1186a/tools/fuchsia/comparative_tester/target_spec.py
[add] https://crrev.com/00e41b5c88db19d6f5b0ed8c083d672992d1186a/tools/fuchsia/comparative_tester/test_results.py

Comment 5 by benhenry@chromium.org, Aug 3

Status: Assigned (was: Untriaged)
This bug has an owner, thus, it's been triaged. Changing status to "assigned".
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ecd7d2a2b9c1323dd406d7989e182b5b79c876fc

commit ecd7d2a2b9c1323dd406d7989e182b5b79c876fc
Author: Stephan Stross <stephanstross@google.com>
Date: Wed Aug 15 21:46:20 2018

Uploaded display.html and additional json helper functions.

display.html prompts a user for a stat-file produced by
generate_perf_report.py and displays it in a table.

Change-Id: Ia82ab50eaedaef35e8c38ca49d8e602f2b6d22b6

Bug: 839491
Change-Id: Ia82ab50eaedaef35e8c38ca49d8e602f2b6d22b6
Reviewed-on: https://chromium-review.googlesource.com/1108965
Commit-Queue: Stephan Stross <stephanstross@google.com>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583408}
[add] https://crrev.com/ecd7d2a2b9c1323dd406d7989e182b5b79c876fc/tools/fuchsia/comparative_tester/display_perf_results.html
[add] https://crrev.com/ecd7d2a2b9c1323dd406d7989e182b5b79c876fc/tools/fuchsia/comparative_tester/display_perf_results.js
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 17 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4d472bbde84cd0922aecebbc147a2315c7b72373

commit 4d472bbde84cd0922aecebbc147a2315c7b72373
Author: Stephan Stross <stephanstross@google.com>
Date: Fri Aug 17 16:47:37 2018

Added the --gtest_filter flag in comparative_tester.py

The ability to filter out tests is vitally important when diagnosing
regressions or otherwise attempting to root out the cause of
performance degradations. Support wasn't present for the gtest_filter
flag beforehand, so this CL adds it, as well as updating the relevant
documentation.

Bug: 839491
Change-Id: I5ebb098f247b2c26d667fcfb3c699b5c27791a09
Reviewed-on: https://chromium-review.googlesource.com/1179061
Commit-Queue: Stephan Stross <stephanstross@google.com>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584102}
[modify] https://crrev.com/4d472bbde84cd0922aecebbc147a2315c7b72373/tools/fuchsia/comparative_tester/README.md
[modify] https://crrev.com/4d472bbde84cd0922aecebbc147a2315c7b72373/tools/fuchsia/comparative_tester/comparative_tester.py
►

Sign in to add a comment