Upload iOS bot test results to the flakiness dashboard |
||||
Issue descriptionOnly relevant for public iOS bots. I'd like to add iOS test results to the existing flakiness dashboard. It sounds like this involves creating a "gtest json" file and then using something like https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/recipe_modules/test_results/resources/upload_test_results.py to upload the results.
,
Jan 6 2017
,
Jan 7 2017
@smut can you give me some pointers on where/how to generate this json file? Could we just modify the format of the existing test_summary.json file, or is that consumed by something else that relies on this specific format?
,
Jan 7 2017
test_summary.json is only consumed by the build/scripts/slave/recipe_modules/ios/api.py in one place to render the presentation layer (stuff like linking the logs of failed tests), so we can modify it if need be, and then we can update the presentation stuff to match. The file is generated by the test runner, located at src/ios/build/bots/scripts/test_runner.py. I am not 100% sure about the details of the format we need to adhere to to use the flakiness dashboard.
,
Jan 7 2017
Capturing notes from an email thread, so that I have them for later: You can use a recipe module to do the uploading: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_results/api.py?rcl=0&l=11 You'll still need to create the test results json or gtest json file. Both formats are supported. Test results json is simpler and it's mostly described here: https://www.chromium.org/developers/the-json-test-results-format The trie encoding is optional and only needed if the test names are redundant and verbose. You can look at some example files here: http://test-results.appspot.com/testfile?name=full_results.json
,
Jan 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8eb71cdc6259464bee1640d4acdefb9638088e40 commit 8eb71cdc6259464bee1640d4acdefb9638088e40 Author: rohitrao <rohitrao@chromium.org> Date: Tue Jan 10 20:15:41 2017 [ios] Updates test_runner.py to output a test results json file. BUG= 678828 Review-Url: https://codereview.chromium.org/2619813002 Cr-Commit-Position: refs/heads/master@{#442668} [modify] https://crrev.com/8eb71cdc6259464bee1640d4acdefb9638088e40/ios/build/bots/scripts/run.py [modify] https://crrev.com/8eb71cdc6259464bee1640d4acdefb9638088e40/ios/build/bots/scripts/test_runner.py
,
Jan 10 2017
I'm not sure how to handle all of the various platforms we run on. Right now the platform gets embedded as part of the test name ("ios_chrome_web_egtests (iPad Air 2 iOS 10.0)"), but that would create a ton of new entries in the test dropdown.
Maybe it would be better to embed the platform as part of the test name or as part of the builder name.
,
Jan 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f commit 9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f Author: smut <smut@google.com> Date: Thu Jan 26 01:44:07 2017 Upload test results to flakiness dashboard BUG= 678828 Change-Id: I3725f5ea58d7986cd98bade6f164ed0b87d3bf0e Reviewed-on: https://chromium-review.googlesource.com/431112 Commit-Queue: smut <smut@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> [modify] https://crrev.com/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f/scripts/slave/recipe_modules/ios/__init__.py [modify] https://crrev.com/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f/scripts/slave/recipe_modules/ios/api.py [modify] https://crrev.com/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f/scripts/slave/recipe_modules/ios/example.expected/basic.json [modify] https://crrev.com/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f/scripts/slave/recipe_modules/ios/example.py [modify] https://crrev.com/9fbcc9b10e7c93aa94187fdc99e3410f94fcac5f/scripts/slave/recipe_modules/test_results/api.py
,
Jan 27 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by olivierrobin@chromium.org
, Jan 6 2017Owner: smut@chromium.org
Status: Assigned (was: Untriaged)