New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 678828 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
User never visited
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 2
Type: Feature



Sign in to add a comment

Upload iOS bot test results to the flakiness dashboard

Project Member Reported by rohitrao@chromium.org, Jan 6 2017

Issue description

Only relevant for public iOS bots.

I'd like to add iOS test results to the existing flakiness dashboard.  It sounds like this involves creating a "gtest json" file and then using something like https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/recipe_modules/test_results/resources/upload_test_results.py to upload the results.
 
Cc: -smut@chromium.org
Owner: smut@chromium.org
Status: Assigned (was: Untriaged)

Comment 2 by s...@google.com, Jan 6 2017

Cc: sergeybe...@chromium.org smut@chromium.org
Owner: ----
Status: Available (was: Assigned)
@smut can you give me some pointers on where/how to generate this json file?  Could we just modify the format of the existing test_summary.json file, or is that consumed by something else that relies on this specific format?

Comment 4 by s...@google.com, Jan 7 2017

test_summary.json is only consumed by the build/scripts/slave/recipe_modules/ios/api.py in one place to render the presentation layer (stuff like linking the logs of failed tests), so we can modify it if need be, and then we can update the presentation stuff to match.

The file is generated by the test runner, located at src/ios/build/bots/scripts/test_runner.py.

I am not 100% sure about the details of the format we need to adhere to to use the flakiness dashboard.
Owner: rohitrao@chromium.org
Capturing notes from an email thread, so that I have them for later:

You can use a recipe module to do the uploading:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_results/api.py?rcl=0&l=11

You'll still need to create the test results json or gtest json file. Both formats are supported. Test results json is simpler and it's mostly described here:
https://www.chromium.org/developers/the-json-test-results-format

The trie encoding is optional and only needed if the test names are redundant and verbose.

You can look at some example files here:
http://test-results.appspot.com/testfile?name=full_results.json
Project Member

Comment 6 by bugdroid1@chromium.org, Jan 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8eb71cdc6259464bee1640d4acdefb9638088e40

commit 8eb71cdc6259464bee1640d4acdefb9638088e40
Author: rohitrao <rohitrao@chromium.org>
Date: Tue Jan 10 20:15:41 2017

[ios] Updates test_runner.py to output a test results json file.

BUG= 678828 

Review-Url: https://codereview.chromium.org/2619813002
Cr-Commit-Position: refs/heads/master@{#442668}

[modify] https://crrev.com/8eb71cdc6259464bee1640d4acdefb9638088e40/ios/build/bots/scripts/run.py
[modify] https://crrev.com/8eb71cdc6259464bee1640d4acdefb9638088e40/ios/build/bots/scripts/test_runner.py

I'm not sure how to handle all of the various platforms we run on.  Right now the platform gets embedded as part of the test name ("ios_chrome_web_egtests (iPad Air 2 iOS 10.0)"), but that would create a ton of new entries in the test dropdown.

Maybe it would be better to embed the platform as part of the test name or as part of the builder name.

Comment 9 by s...@google.com, Jan 27 2017

Cc: -smut@chromium.org
Owner: smut@chromium.org
Status: Fixed (was: Available)

Sign in to add a comment