New issue
Advanced search Search tips

Issue 875527 link

Starred by 1 user
Status: Fixed
Owner:
derat@chromium.org
Closed: Sep 5
Cc:
akes...@chromium.org
jclinton@chromium.org
nya@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Task



Sign in to add a comment

Make Tast test failures matter on the Chrome OS CQ, Chrome PFQ, and Android PFQ

Project Member Reported by derat@chromium.org, Aug 18 
I think that the bvt-tast-cq, bvt-tast-chrome-pfq, and bvt-tast-android-pfq Tast test suites are stable enough that failures should really count. Here's what the release builders looked like for Aug 12-18:

http://stainless/search?view=matrix&row=model&col=build&first_date=2018-08-12&last_date=2018-08-18&test=%5Etast%5C.mustpass&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true

At this point, this should just be a matter of removing the warn_only attribute from these suites.

After this is done, we can start talking about removing slower/harder-to-debug Autotest-based tests in the places where there's overlap.
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 20 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f8268f7ccb655e300527e2374056fb3b4be1a658

commit f8268f7ccb655e300527e2374056fb3b4be1a658
Author: Daniel Erat <derat@chromium.org>
Date: Mon Aug 20 13:56:20 2018

chromeos_config: Remove warn_only from Tast suites.

Update Tast test suites (bvt-tast-cq, bvt-tast-chrome-pfq,
bvt-tast-android-pfq) to not set warn_only to true. This has
the effect of making Tast test failures "important" on the
CQ and PFQs.

BUG= chromium:875527 
TEST=looked at updated config_dump.json

Change-Id: I59955efef4a05e8aface3fba23ff6b3e68c5116c
Reviewed-on: https://chromium-review.googlesource.com/1180518
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/f8268f7ccb655e300527e2374056fb3b4be1a658/config/chromeos_config.py
[modify] https://crrev.com/f8268f7ccb655e300527e2374056fb3b4be1a658/config/config_dump.json

Comment 2 by derat@chromium.org, Aug 20

Status: Started (was: Assigned)
I'll keep this open until I'm sure that it's sticking.
Project Member

Comment 3 by bugdroid1@chromium.org, Sep 5 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/35e33633eada9ee699397d691ad4ebc8e30839b1

commit 35e33633eada9ee699397d691ad4ebc8e30839b1
Author: Daniel Erat <derat@chromium.org>
Date: Wed Sep 05 16:15:23 2018

tast-tests: Promote tests to be non-informational.

Promote various stable tests that replicate existing BVT
Autotest tests to be non-informational so they'll run on the
CQ and PFQs:

audio.DevicePlay
audio.DeviceRecord
network.DefaultProfile
platform.CheckProcesses
platform.TPMResponsive
ui.ChromeRespawn
ui.SessionManagerRespawn

BUG= chromium:875527 
TEST=checked R71 release builder test results from the
     previous day and verified that all of these tests had
     100% pass rates

Change-Id: Ia5cd7b5724df239b9f7ccfd29543a8afc28dec94
Reviewed-on: https://chromium-review.googlesource.com/1203634
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/ui/session_manager_respawn.go
[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/network/default_profile.go
[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/platform/tpm_responsive.go
[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/platform/check_processes.go
[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/audio/device_play.go
[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/audio/device_record.go
[modify] https://crrev.com/35e33633eada9ee699397d691ad4ebc8e30839b1/src/chromiumos/tast/local/bundles/cros/ui/chrome_respawn.go

Comment 4 by derat@chromium.org, Sep 5

Status: Fixed (was: Started)
It's been on now for 2.5 weeks without any issues.
►

Sign in to add a comment