New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 704843 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
User never visited
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 2
Type: Feature



Sign in to add a comment

Make iOS retried tests that passes not fail the build

Project Member Reported by kjellander@chromium.org, Mar 24 2017

Issue description

Today, when a test fails on an iOS simulator bot but passes after a retry, it's still causing the test execution to fail (even if all tests eventually passed after retries).

This should be changed to counting the run as a successful run, since there are other infrastructure pieces that can be used to address the flakiness in a more efficient manner: go/flakiness-dashboard and the Try Flakes (https://cs.corp.google.com/chromium_infra/appengine/chromium_try_flakes/) service.

Using those, it's possible to address the flakiness in a more efficient way, since they'll be tracked in bugs which can be assigned and/or used as input for a FixIt.

Spamming random developers with random flakes in e-mail is unlikely to get addressed.

After changing this, it should be ensured that the flaky tests during iOS Simulation runs are actually reported properly to the flakiness dashboard (out of scope for this bug and should probably be tracked in a separate bug).
 
Project Member

Comment 1 by bugdroid1@chromium.org, Mar 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8e05f43c211c34bc5ad3f796021953d724e9f687

commit 8e05f43c211c34bc5ad3f796021953d724e9f687
Author: smut <smut@google.com>
Date: Fri Mar 31 22:35:54 2017

Consider passing retries to be flake

Considering it flake still logs the failure, but the return code will be zero and the step will be considered a success.

BUG= 704843 

Review-Url: https://codereview.chromium.org/2780513006
Cr-Commit-Position: refs/heads/master@{#461256}

[modify] https://crrev.com/8e05f43c211c34bc5ad3f796021953d724e9f687/ios/build/bots/scripts/test_runner.py

Comment 2 by s...@google.com, Mar 31 2017

Status: Fixed (was: Assigned)

Comment 3 by s...@google.com, Jun 23 2017

Owner: smut@chromium.org

Sign in to add a comment