New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 827442 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: May 2018
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug
Proj-Servicification

Blocking:
issue 822484



Sign in to add a comment

Run Control and Enabled tests on the same bot and upload the difference

Project Member Reported by chongz@chromium.org, Mar 30 2018

Issue description

We want to reduce the noise by monitoring the difference between feature on/off on the same bot.

More Discussions:
https://groups.google.com/a/google.com/forum/#!topic/chrome-speed-services/MVp4dMdkBxk
 

Comment 1 by chongz@chromium.org, Mar 30 2018

Example 'results-chart.json' from https://crrev.com/c/987305/4 with the following command:
```
tools/perf/run_benchmark loading.desktop.network_service --browser=default --story-tag-filter=typical --pageset-repeat=2 --output-dir=~/123 --output-format=chartjson --upload-results
```
results-chart.json
4.5 MB View Download
Project Member

Comment 2 by bugdroid1@chromium.org, Apr 6 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/51164e7b6bc047ac6ff0c0cd21b5f544a777fac4

commit 51164e7b6bc047ac6ff0c0cd21b5f544a777fac4
Author: Chong Zhang <chongz@chromium.org>
Date: Fri Apr 06 19:06:29 2018

Run loading.desktop with Network Service on/off and record the difference

Previously we run loading.desktop on the perf fyi bot with Network
Service on, but it's hard to compare the numbers manually or setup
sheriff alerts.

This CL modified loading.desktop.network_service to run the test twice
with feature on/off on the same bot, and report the diff to the perf
dashboard (as well as the original data).

More specifically, we will load the resulting 'results-chart.json' from
two runs and combine them into one (with appropriated suffix).
e.g. We will turn 'warm@@timeToFirstContentfulPaint_avg' into:
  * 'warm@@timeToFirstContentfulPaint_avg_control'
  * 'warm@@timeToFirstContentfulPaint_avg_enabled'
  * 'warm@@timeToFirstContentfulPaint_avg_diff'

And things like 'timeToFirstContentfulPaint_avg_diff' will be the
interesting metrics.

Note: Please see bug for an example generated 'results-chart.json'.

Also Note: --output-format=chartjson is going away soon, this CL added
myself to 'tools/perf/contrib/network_service/OWNERS' for future
maintainence.

Bug:  827442 
Change-Id: I26cd027d4b59f6b927736a43f068bd0d89d279c8
Reviewed-on: https://chromium-review.googlesource.com/987305
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Simon Hatch <simonhatch@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Chong Zhang <chongz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#548879}
[modify] https://crrev.com/51164e7b6bc047ac6ff0c0cd21b5f544a777fac4/tools/perf/contrib/network_service/OWNERS
[modify] https://crrev.com/51164e7b6bc047ac6ff0c0cd21b5f544a777fac4/tools/perf/contrib/network_service/loading.py

Project Member

Comment 3 by bugdroid1@chromium.org, Apr 10 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/29ea2a37f4e2c7b2af5596a56dfee23960aaf272

commit 29ea2a37f4e2c7b2af5596a56dfee23960aaf272
Author: Chong Zhang <chongz@chromium.org>
Date: Tue Apr 10 06:17:08 2018

[Mojo FYI] Only merge common charts and points

'loading.desktop.network_service' fails randomly when uploading
json results, e.g.
https://ci.chromium.org/buildbot/chromium.perf.fyi/Mojo%20Linux%20Perf/5265

It seems that we might miss some charts in certain conditions.
e.g. We don't have 'timeToInteractive_std' when all values are None.

This CL handled the case by only doing diff on common charts / points
between control and enabled chartjson result.

Bug:  827442 
Change-Id: Ibb1b9004d48bdfc2fa017c4acf57c989ae15c17f
Reviewed-on: https://chromium-review.googlesource.com/1003520
Reviewed-by: Kouhei Ueno <kouhei@chromium.org>
Commit-Queue: Chong Zhang <chongz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#549434}
[modify] https://crrev.com/29ea2a37f4e2c7b2af5596a56dfee23960aaf272/tools/perf/contrib/network_service/loading.py

Project Member

Comment 4 by bugdroid1@chromium.org, May 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b5d85ba7122683f2522c67c77dc89c7aa1264112

commit b5d85ba7122683f2522c67c77dc89c7aa1264112
Author: Chong Zhang <chongz@chromium.org>
Date: Tue May 01 01:20:34 2018

[Mojo Perf FYI] Add none_value_reason to artificial points with None value

The Mojo Linux Perf bot frequently fails due to invalid JSON.

This patch fixes the issue by adding 'none_value_reason' to artificial
points that has None value.

Bug:  827442 
Change-Id: I508091a052184c439f4de712724f8f691bd1bbd9
Reviewed-on: https://chromium-review.googlesource.com/1036463
Reviewed-by: Jun Cai <juncai@chromium.org>
Commit-Queue: Chong Zhang <chongz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554954}
[modify] https://crrev.com/b5d85ba7122683f2522c67c77dc89c7aa1264112/tools/perf/contrib/network_service/loading.py

Status: Fixed (was: Started)
The build is consistently green since https://ci.chromium.org/buildbot/chromium.perf.fyi/Mojo%20Linux%20Perf/5917

Sign in to add a comment