Can you add a link to an example on goldeye that is missing some tests?

Here is an example: https://uberchromeos-server38.corp.google.com/new_tko/#tab_id=spreadsheet_view&row=job_name%252Ctest_name&column=status&show_incomplete=true&show_only_latest=false&show_invalid=true&condition=job_name%2520LIKE%2520%27daisy-release/R53-8530.86.0/bvt-cq/%2525%27%2520AND%2520test_name%2520%253C%253E%2520%27SERVER_JOB%27%2520AND%2520test_name%2520NOT%2520LIKE%2520%27CLIENT%255C_JOB%2525%27

And the GoldenEye record: https://cros-goldeneye.corp.google.com/chromeos/console/viewBuild?version=8530.86.0&channel=stable&type=release

For Daisy (e.g.) you can see we just got 24 results

Idea: have the suite job emit keyvals that list the number of scheduled child jobs, and perhaps also their identities.

Goldeneye can ingest the keyval for the suite job from tko, and use this to at least determine how many child jobs to expect.

Will discuss this with Charlene in our 1:1 tomorrow, I expect she will be the one to implement the lab site.

Issue 653683 has been merged into this issue.

 Issue 470365  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cd866f3f1e0366623349e9831d6d19eb3c27f6c4

commit cd866f3f1e0366623349e9831d6d19eb3c27f6c4
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Nov 30 04:14:34 2016

[autotest] emit scheduled tests names and number of them from suite

For a suite, write the number of scheduled tests and names of them into
keyval file. These keyvals can be inserted into tko database when the
suite finishes, which can help goldeneye team debug the missing tests.

BUG=chromium:661703
TEST=unittest, trybot and run tests with the test build.

Change-Id: I3cb194d4d89628943b8052db32ef4a44df51df5e
Reviewed-on: https://chromium-review.googlesource.com/414243
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/cd866f3f1e0366623349e9831d6d19eb3c27f6c4/server/cros/dynamic_suite/suite.py
[modify] https://crrev.com/cd866f3f1e0366623349e9831d6d19eb3c27f6c4/server/cros/dynamic_suite/suite_unittest.py
[modify] https://crrev.com/cd866f3f1e0366623349e9831d6d19eb3c27f6c4/server/cros/dynamic_suite/constants.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cd866f3f1e0366623349e9831d6d19eb3c27f6c4

commit cd866f3f1e0366623349e9831d6d19eb3c27f6c4
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Nov 30 04:14:34 2016

[autotest] emit scheduled tests names and number of them from suite

For a suite, write the number of scheduled tests and names of them into
keyval file. These keyvals can be inserted into tko database when the
suite finishes, which can help goldeneye team debug the missing tests.

BUG=chromium:661703
TEST=unittest, trybot and run tests with the test build.

Change-Id: I3cb194d4d89628943b8052db32ef4a44df51df5e
Reviewed-on: https://chromium-review.googlesource.com/414243
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/cd866f3f1e0366623349e9831d6d19eb3c27f6c4/server/cros/dynamic_suite/suite.py
[modify] https://crrev.com/cd866f3f1e0366623349e9831d6d19eb3c27f6c4/server/cros/dynamic_suite/suite_unittest.py
[modify] https://crrev.com/cd866f3f1e0366623349e9831d6d19eb3c27f6c4/server/cros/dynamic_suite/constants.py

I see the test name list is being sent in one single keyval, I'm guessing this will go to tko_job_keyvals which has a limit of 300 chars on the value, am I correct?

Don't want to get greedy but the names will get truncated, is there a way to get the whole list?

Maybe increase the limit to 2048?

I'm not an expert but it's worth mentioning using TEXT as that would not waste a lot of space for the other keyvals and would allow for very large ones like the test names, I'd worry about its performance but given the rows are small it may not be hurtful (see http://stackoverflow.com/questions/2023481/mysql-large-varchar-vs-text and https://www.percona.com/blog/2010/02/09/blob-storage-in-innodb/).

Most of the other keyvals are very short, I think using TEXT would waste the space....And if the test_names field will be frequently queried, I think varchar may be better. 

The following queries can also help to get the scheduled tests for a suite
SELECT DISTINCT label FROM tko_jobs WHERE afe_parent_job_id = <suite_afe_job_id>;

SELECT DISTINCT label FROM tko_jobs WHERE label LIKE <SUITE_NAME>
SELECT DISTINCT label FROM tko_jobs WHERE label LIKE 'quawks-release/R55-8872.44.0/paygen_au_beta%';

I'm trying to get this done for GoldenEye now and just noticed that the tko_job_keyvals table value column is still limited at 300 chars. Can we do this so I can actually, cheaply, query the scheduled test names? I want to provide names and not just numbers. Thanks!

Will check later this week.

Fixed?

I took a look at goldeneye. The result counts for suites on boards with the same SOC often vary quite a bit. Which means the problem is not fixed for bvt/goldeneye.

That said for arc-cts and the cts dashboard the problem of getting expected tests was fixed as suggested here. (E.g. ask tradefed to enumerate all its tests and use that to compare to actual runs.)

To return to GE, we need to continue with comment #18. It sounds like we are very close!

The GoldenEye side is not done yet but we still need the column change to make it happen.