New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 682496 link

Starred by 1 user

Issue metadata

Status: Archived
Owner: ----
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

Make Sheriff-o-Matic distinguish between different tests on the same binary

Project Member Reported by zhangtiff@chromium.org, Jan 19 2017

Issue description

This was a request from the sheriffing survey. 
 
Do you have more information about this? How is this different from the test results data we already have?
Unfortunately, that's all the information I can find on this from the survey response. We could followup with the sheriff who requested this if we want though. 
Please do follow up with this sheriff. There isn't enough here to know if we've fixed the problem or not.
I left this feedback. What I meant was that a bug in the bug queue has a summary like '"browser_tests" is flaky', but it doesn't say which particular test(s). What happens also, iirc, is that the same bug gets re-opened for a different failure/flakiness. I think this would be nice to get into separate bugs, but I realize it may not be straight-forward.
Owner: serg...@chromium.org
Status: Assigned (was: Untriaged)
Oh I see. It's an issue with a bug in the bug queue, not really an alert. Those bugs come from chromium-try-flakes. Assigning to the owner of that.
Status: WontFix (was: Assigned)
Unfortunately, it is infeasible for us to implement uploading for every single test suite as there are too many different test launchers and we are frequently unfamiliar with the code. Therefore we ask teams to find owners for their test suites and implement uploading a JSON with the test results to the test-results server, e.g. see https://bugs.chromium.org/p/chromium/issues/detail?id=658619#c29. The JSON is documented at https://www.chromium.org/developers/the-json-test-results-format.

Many test suites (including browser_tests) are already uploading to test-results server and chromium-try-flakes files bugs for individual tests, e.g.  http://crbug.com/683419 . Also, as many test launchers are generating GTest JSON output, we have written code to covert that format to our JSON format making it even easier to implement necessary changes to start uploading results.
Cc: smut@chromium.org
Do you have an example CL for implementing uploading for a test suite?
Owner: ----
Status: Available (was: WontFix)
Re-opening since some clients of SoM *do* upload their test results, and we can potentially parse them. This isn't really a sergiyb/test-results problem, but that's to be noted. We can't implement this for clients who don't upload their own test results.

Comment 9 by s...@google.com, Feb 2 2017

Sure, here's how to upload test results for a test suite from a recipe (assuming you already have the JSON file as documented in #6):
https://chromium.googlesource.com/chromium/tools/build/+/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f

It's as simple as calling test_results.upload.
Status: Archived (was: Available)

Sign in to add a comment