New issue
Advanced search Search tips

Issue 810210 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Aug 3
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

[Test Results] Add target_name field to hrd:test_results table

Project Member Reported by wylieb@chromium.org, Feb 8 2018

Issue description

(target_name = the gn label of the test binary)

In order to uniquely identify a test, a test_name and target_name is required since there are duplicate tests in different targets.

Chromium-try-flakes didn't use test/target to uniquely identify, but the new flake detection will since the amount of duplicate tests is 200+ (may grow in the future).

Any chance on getting this field added to the test results table?
 
Cc: st...@chromium.org
what is a "target_name" and where does it come from?
(target_name = the gn label of the test binary)
+stgao@ could you provide some context on where this is generated from & how Sean might go about accessing it?

Comment 5 by st...@chromium.org, Feb 9 2018

Cc: dpranke@chromium.org
Currently, we are using the test_type derived from the step name on test-results and chromium-try-flakes. And such test_type names include base_unittests, browser_tests, site_per_process_browser_tests, etc.

In fact, site_per_process_browser_tests and browser_tests run the same test binary but just with different command-line flags. The goal here is to use target name (either the Ninja one or its GN label) instead so that the same test name site_per_process_browser_tests and browser_tests could be identified by (target_name, test_name). As Brandon mentioned, currently chromium-try-flakes just uses test_name.

IMO, Ninja target name is good to go with.
+dpranke what do you think of this?

https://cs.chromium.org/chromium/src/testing/buildbot/gn_isolate_map.pyl
I believe we should have the equivalent info of this file in recipe especially chromium_tests recipe module. But I don't know for sure.
Just the ninja target name is not sufficient, because you might run the same binary in multiple configurations. You need to use the step name. See bit.ly/chromium-build-naming.

Comment 7 by st...@chromium.org, Feb 12 2018

step_name is already available in test-results table, but the target name is not added yet and it seems impossible to derive the target name from the step name, e.g. "site_per_process_browser_tests on Mac" and "browser_tests on Mac" both use the same binary -- browser_tests.

The gn label of the target name seems too long, so I was thinking that the ninja target name is better. It is still unique, because ninja requires compile targets to be unique in its build graph.
wdyt?
Ah. Why would you want the compile target instead of the step name? Is it specifically so you can determine whether two steps might have the same compile target?

Comment 9 by st...@chromium.org, Feb 12 2018

Nope :)

As we discussed early, we'd want to identify a test by <target-name, test-name> and group test flake occurrences by that instead of just <test-name> like chromium-try-flakes, because tests of the same name could be in two different test targets or two different test steps using the same test target.

Grouping by the step name only seems too fine-grained and might result in filing too many bugs for the same test definition.
I see. I'd probably start w/ step name and only change it if we need to.
Would this be an opportunity to use https://bit.ly/chromium-build-naming ?
Status: Assigned (was: Untriaged)
This bug has an owner, thus, it's been triaged. Changing status to "assigned".
Cc: liaoyuke@chromium.org
Status: WontFix (was: Assigned)
Based on a recent discussion, Yuke would like to use test binary name instead, and add it to step metadata. So this is not longer needed.

Sign in to add a comment