New issue
Advanced search Search tips

Issue 833875 link

Starred by 1 user
Status: Assigned
Owner:
gbeaty@chromium.org
Cc:
bpastene@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug


Show other hotlists

Hotlists containing this issue:
chrome-client-infra-backlog


Sign in to add a comment

do not dump all failed tests to step text

Project Member Reported by no...@chromium.org, Apr 17 2018 
bug 833689 will enforce a limit of size of a build. If a large number of tests failed in a Chromium build and the recipe dumps ALL of them to step_text or step links, the build might violate the limit and will be marked as an infra failure (because of the violation) instead of the failure.

I believe a user would be still happy if the recipe dumps only first 100 tests to the step text, and dumps all tests into a step log.

example build that would violate the limit: https://ci.chromium.org/p/chromium/builds/b8949700709453540704
context: go/cit-pm-76

Comment 1 Deleted

Comment 2 by no...@chromium.org, Apr 17 2018

Description: Show this description

Comment 3 by sergeybe...@chromium.org, Apr 17 2018

Status: Available (was: Untriaged)
Added to chrome-client-infra-backlog for tracking.

Comment 4 by no...@chromium.org, May 13 2018

Cc: bpastene@chromium.org

Comment 5 by no...@chromium.org, May 13 2018

Summary: do not dump all failed tests to step text (was: do not dump all failed tests to step presentation)

Comment 6 by efoo@chromium.org, Jun 2 2018

Labels: cit-pm-76

Comment 7 by efoo@chromium.org, Jun 2 2018 

Friendly ping. This is a blocking bug for cit-pm-76. Please update pri and comment accordingly. Thanks!

Comment 8 by no...@chromium.org, Jun 2 2018 

this bug is related to cit-pm-76, but fixing this bug would not help preventing an outage like cit-pm-76. It does not block cit-pm-76.

Comment 9 by gbeaty@chromium.org, Jun 26 2018

Owner: gbeaty@chromium.org
Status: Assigned (was: Available)

Comment 10 by gbeaty@chromium.org, Jun 27 2018

Status: Started (was: Assigned)
Project Member

Comment 11 by bugdroid1@chromium.org, Jul 18 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/34094ab774eba332444bc31db99673ace272f93b

commit 34094ab774eba332444bc31db99673ace272f93b
Author: Garrett Beaty <gbeaty@chromium.org>
Date: Wed Jul 18 18:44:24 2018

Limit the number of failures reported when a gtest-based test fails.

Bug: 833875
Change-Id: Iab37988167627f8933c93198494d455ffdcdb674
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/1137252
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/test_utils/__init__.py
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipes/chromium_trybot.expected/invalid_results.json
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with details fail.json
[add] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/swarming/examples/full.expected/gtest_with_many_failures.json
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipes/findit/chromium/test.expected/none_swarming_tests.json
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/README.recipes.md
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipes/chromium_trybot.expected/compile_failure_without_patch_deapply_fn.json
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/test_utils/api.py
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/chromium_tests/steps.py
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.expected/basic fail.json
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/swarming/examples/full.py
[modify] https://crrev.com/34094ab774eba332444bc31db99673ace272f93b/scripts/slave/recipe_modules/chromium_android/api.py
Project Member

Comment 12 by bugdroid1@chromium.org, Jul 18 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/269ceebc62efc27ba20d6949d348115f6b75debe

commit 269ceebc62efc27ba20d6949d348115f6b75debe
Author: Garrett Beaty <gbeaty@chromium.org>
Date: Wed Jul 18 21:54:03 2018

Use present_gtest_failures consistently for gtest-based tests.

present_gtest_failures was updated to return the gtest_results so that
callers don't have to replicate the code to check for the presence of
the gtest_results if they have other operations to perform with it.

Bug: 833875
Change-Id: I7db2180bb032017926698b55e885dc0ea97c2a60
Reviewed-on: https://chromium-review.googlesource.com/1142472
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipes/chromium.expected/dynamic_junit_test.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_junit_test.expected/basic.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_test.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_tests/tests/steps/android_junit_test.expected/basic.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/README.recipes.md
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_tests/tests/steps/android_instrumentation_test.expected/basic.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_instrumentation_test.expected/basic.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_test_custom_name.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/test_utils/api.py
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_tests/steps.py
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.expected/basic fail.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipe_modules/chromium_android/api.py
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_nodefault_build.json
[modify] https://crrev.com/269ceebc62efc27ba20d6949d348115f6b75debe/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_nodefault_test.json
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 18 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/77991c350d8f82f8883cc08061d6539227a227ed

commit 77991c350d8f82f8883cc08061d6539227a227ed
Author: Garrett Beaty <gbeaty@chromium.org>
Date: Wed Jul 18 23:11:40 2018

Limit the number of reported test failures.

A general-purpose limit_failures method was added to the test_utils api
that can be used to limit the failures reported by a step.

Bug: 833875
Change-Id: Iae8eabbb8f354969eb54549b0444a676676ad263
Reviewed-on: https://chromium-review.googlesource.com/1142538
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/77991c350d8f82f8883cc08061d6539227a227ed/scripts/slave/README.recipes.md
[modify] https://crrev.com/77991c350d8f82f8883cc08061d6539227a227ed/scripts/slave/recipe_modules/test_utils/api.py
[modify] https://crrev.com/77991c350d8f82f8883cc08061d6539227a227ed/scripts/slave/recipe_modules/chromium_tests/steps.py

Comment 14 by gbeaty@google.com, Jul 19

Status: Fixed (was: Started)
I believe all test types that report a variable number of failures in their step text are now using some limiting mechanism.

Comment 15 by no...@chromium.org, Jan 10

Status: Assigned (was: Fixed)
"base_unittests (retry summary)" step in https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_asan_rel_ng/177500
still dumps too many tests
►

Sign in to add a comment