Make Sheriff-o-Matic distinguish between different tests on the same binary |
||||||
Issue descriptionThis was a request from the sheriffing survey.
,
Jan 19 2017
Unfortunately, that's all the information I can find on this from the survey response. We could followup with the sheriff who requested this if we want though.
,
Jan 19 2017
Please do follow up with this sheriff. There isn't enough here to know if we've fixed the problem or not.
,
Jan 20 2017
I left this feedback. What I meant was that a bug in the bug queue has a summary like '"browser_tests" is flaky', but it doesn't say which particular test(s). What happens also, iirc, is that the same bug gets re-opened for a different failure/flakiness. I think this would be nice to get into separate bugs, but I realize it may not be straight-forward.
,
Jan 20 2017
Oh I see. It's an issue with a bug in the bug queue, not really an alert. Those bugs come from chromium-try-flakes. Assigning to the owner of that.
,
Jan 23 2017
Unfortunately, it is infeasible for us to implement uploading for every single test suite as there are too many different test launchers and we are frequently unfamiliar with the code. Therefore we ask teams to find owners for their test suites and implement uploading a JSON with the test results to the test-results server, e.g. see https://bugs.chromium.org/p/chromium/issues/detail?id=658619#c29. The JSON is documented at https://www.chromium.org/developers/the-json-test-results-format. Many test suites (including browser_tests) are already uploading to test-results server and chromium-try-flakes files bugs for individual tests, e.g. http://crbug.com/683419 . Also, as many test launchers are generating GTest JSON output, we have written code to covert that format to our JSON format making it even easier to implement necessary changes to start uploading results.
,
Feb 2 2017
Do you have an example CL for implementing uploading for a test suite?
,
Feb 2 2017
Re-opening since some clients of SoM *do* upload their test results, and we can potentially parse them. This isn't really a sergiyb/test-results problem, but that's to be noted. We can't implement this for clients who don't upload their own test results.
,
Feb 2 2017
Sure, here's how to upload test results for a test suite from a recipe (assuming you already have the JSON file as documented in #6): https://chromium.googlesource.com/chromium/tools/build/+/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f It's as simple as calling test_results.upload.
,
Dec 15 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by martiniss@chromium.org
, Jan 19 2017