New issue
Advanced search Search tips

Issue 904364 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 3
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: Bug



Sign in to add a comment

Clean up system health orderfile tests

Project Member Reported by mattcary@chromium.org, Nov 12

Issue description

This has been failing in local testing, and I just discovered on the bot that background:tools:gmail is disabled on Nexus_5 due to crbug.com/784400.

As my clients are all currently occupied in testing I'm creating this bug so that I don't drop this issue.
 
Labels: -Pri-3 Pri-1
And load:tools:gmail.

(load:tools:gmail is disabled on Android due to crbug.com/657433)
Summary: Clean up system health orderfile tests (was: disable background:tools:gmail from orderfile)
load:news:cnn:2018 seems to be running okay on the bots, but is failing for me.

The test in #1 is background:tools:gmail.
browse:social:facebook_infinite_scroll:2018

and browse:social:facebook due to crbug.com/780779.
Side note: this could be (semi) automated by parsing tools/perf/expectations.config. But I'm not sure about that. For now updating manually may be okay. Additional tests from that file:

background:news:nytimes
background:tools:gmail
browse:chrome:newtab
browse:chrome:omnibox
browse:news:cnn
browse:news:cnn:2018
browse:news:globo
browse:news:toi
browse:shopping:avito
browse:shopping:flipkart
browse:social:facebook
browse:social:facebook_infinite_scroll
browse:social:pinterest_infinite_scroll
browse:social:tumblr_infinite_scroll
browse:tech:discourse_infinite_scroll:2018
load:media:soundcloud
load:news:cnn
load:news:washingtonpost
load:tools:drive
load:tools:gmail
long_running:tools:gmail-background
long_running:tools:gmail-foreground

Possible failing tests found during testing:

[  FAILED  ]  orderfile_generation.testing/browse:shopping:lazada@{'case': 'browse', 'group': 'shopping'}
[  FAILED  ]  orderfile_generation.testing/load:games:bubbles@{'case': 'load', 'group': 'games'}

The two tests in #6 just seem to be flaky, they passed the second time around.
Project Member

Comment 8 by bugdroid1@chromium.org, Nov 27

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bcac190e337ca005d030fb315549ff85785cf4cc

commit bcac190e337ca005d030fb315549ff85785cf4cc
Author: Matthew Cary <mattcary@chromium.org>
Date: Tue Nov 27 16:59:59 2018

Orderfile: update orderfile test suite.

The orderfile generation and validation benchmark, based on
system_health, does not parse tools/perf/expectations.config and
instead manually blacklists failing tests. This CL updates that
blacklist.

As the blacklist has grown much larger, some assumptions in the tests
about how many variations could be fit have been broken, requiring the
test to be updated as well.

Bug:  904364 
Change-Id: I342ed77fb07b3f562eef9fd9e16600a7c424b2eb
Reviewed-on: https://chromium-review.googlesource.com/c/1350974
Reviewed-by: Egor Pasko <pasko@chromium.org>
Reviewed-by: Benoit L <lizeb@chromium.org>
Commit-Queue: Matthew Cary <mattcary@chromium.org>
Cr-Commit-Position: refs/heads/master@{#611149}
[modify] https://crrev.com/bcac190e337ca005d030fb315549ff85785cf4cc/tools/perf/contrib/orderfile/orderfile.py
[modify] https://crrev.com/bcac190e337ca005d030fb315549ff85785cf4cc/tools/perf/contrib/orderfile/orderfile_unittest.py

Status: Fixed (was: Started)

Sign in to add a comment