New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 717574 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Feature
okr



Sign in to add a comment

Need a tool to display builder test coverage in the lab

Reported by jrbarnette@chromium.org, May 2 2017

Issue description

We need a tool that will help in evaluating current HWTest
coverage in the lab, and finding gaps.  BVT test coverage
requirements for release builders is relatively consistent
and well understood, so the tool should focus on non-release
builders, especially the CQ and PFQ.

The tool needs to be able to gather and report the following
information about every non-release builder that is configured
to run tests:
  * From the chromite config:  The board the builder builds for.
  * From the chromite config:  All test suites the builder will run.
  * From the chromite config:  The pool the suites will run against.
  * From the AFE database:  The number of DUTs for the selected pool
    and board.
  * From the AFE database:  The number of DUTs in the board's
    'suites' pool.
  * From CIDB:  various run time statistics for HWTest phases;
    at minimum, include average run time over the past month or so.
  * From CIDB:  for each HWTest phase statistic, the analogous
    statistic for the builder as a whole.
  * From GoldenEye:  The board family of the builder.

The tool should also gather information about coverage gaps by
listing the following information about all boards _not_ tested
in a non-release builder:
  * From Goldeneye:  The board family.
  * From the AFE database:  The number of DUTs in the board's
    'suites' pool.

Additional information that may be helpful:
  * The number of tests in each suite.

The board->family information from GoldenEye can be found
here (among other locations):
    gs://chromeos-build-release-console/boards.json
Each entry in the file has a "reference_group" field.

 
For reference, attached is a sample Python program that
reports the necessary data from Chromite and the AFE,
but not from CIDB or GoldenEye.

coverage.py
2.0 KB View Download

Comment 2 by aut...@google.com, May 5 2017

Labels: -current-issue
Owner: jrbarnette@chromium.org
Labels: -Type-Bug OKR Type-Feature
Owner: ----
Probably, I'll wind up doing this, because no one else will
care enough.  But it's not in my queue yet, and I want this
marked as "someone else should feel entitled to work on this."

Owner: jrbarnette@chromium.org
Status: Assigned (was: Available)
Hi, this bug has not been updated recently. Please acknowledge the bug and provide status within two weeks (6/22/2018), or the bug will be archived. Thank you.

Sign in to add a comment