New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 914929 link

Starred by 1 user
Status: Fixed
Owner:
erikc...@chromium.org
Closed: Dec 19
Cc:
█ nednguyen@chromium.org
sky@chromium.org
estaab@chromium.org
jbudorick@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

gtest timeouts should not set the tag UNRELIABLE_RESULTS

Project Member Reported by erikc...@chromium.org, Dec 13 
Example build: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/cast_shell_linux/200224

Task:
https://chromium-swarm.appspot.com/task?id=41b3f93a6c058b10&refresh=10&show_raw=1

Task output:
https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=540b3d4a5c36159ee263847c5ba3b5f5d58d5843

The UNRELIABLE_RESULTS tag causes the chromium_test recipe to throw away all results from the shard, as if it were never run. But in a timeout, there's actually plenty of information in the shard.

e.g. test A was run and passed, test C was never run due to running out of time, test B timed out repeatedly.

These results are reliable, but present partial information. This allows the chromium_test recipe to retry the subset of failing/timing-out/never-run tests.

Without this, the chromium_test recipe automatically retries all failing tests in all shards, a significant waste of resources.
Project Member

Comment 1 by bugdroid1@chromium.org, Dec 17 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f72be4473accc49df61f327f7c7d0ec9a081abb4

commit f72be4473accc49df61f327f7c7d0ec9a081abb4
Author: erikchen <erikchen@chromium.org>
Date: Mon Dec 17 22:45:11 2018

Generate a placeholder iteration for TestLauncher.

The placeholder iteration has an entry for every test that will be run, with the
status: NOTRUN. This is emitted even if there's a timeout, or the process
receives a signal, or there's partial completion of tests.

This is the first step to allowing the chromium_tests recipe to parse partial
results from gtest-based test suites.

Bug:  914929 
Change-Id: Ib81d380108946f478e666cdd02e8e5afec368166
Reviewed-on: https://chromium-review.googlesource.com/c/1377137
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#617270}
[modify] https://crrev.com/f72be4473accc49df61f327f7c7d0ec9a081abb4/base/test/launcher/test_launcher.cc
[modify] https://crrev.com/f72be4473accc49df61f327f7c7d0ec9a081abb4/base/test/launcher/test_result.cc
[modify] https://crrev.com/f72be4473accc49df61f327f7c7d0ec9a081abb4/base/test/launcher/test_result.h
[modify] https://crrev.com/f72be4473accc49df61f327f7c7d0ec9a081abb4/base/test/launcher/test_results_tracker.cc
[modify] https://crrev.com/f72be4473accc49df61f327f7c7d0ec9a081abb4/base/test/launcher/test_results_tracker.h
Project Member

Comment 2 by bugdroid1@chromium.org, Dec 18 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8f267191a3486923b03d16793bfdf877c61c8f8

commit c8f267191a3486923b03d16793bfdf877c61c8f8
Author: erikchen <erikchen@chromium.org>
Date: Tue Dec 18 14:53:02 2018

Do not emit UNRELIABLE_RESULTS global tag for incomplete results.

TestLauncher previously emitted the UNRELIABLE_RESULTS global tag when it had
incomplete results. This would cause the chromium_tests recipe to throw away the
data from all shards and then retry all tests. Instead, TestLauncher should emit
incomplete results. The placeholder iteration ensures that no data is lost. This
allows the chromium_tests recipe to correctly recover from the failure and only
retry the tests which had incomplete results.

Bug:  914929 
Change-Id: Ie1f6f64345e7d1874197acd1061e6d7804350886
Reviewed-on: https://chromium-review.googlesource.com/c/1377104
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#617488}
[modify] https://crrev.com/c8f267191a3486923b03d16793bfdf877c61c8f8/base/test/launcher/test_launcher.cc

Comment 3 by erikc...@chromium.org, Dec 19

Status: Fixed (was: Assigned)
This should be fixed. Closing for now.
►

Sign in to add a comment