Found issues with JSON test results for skipped tests |
||
Issue descriptionI found several issues in the JSON test results for both skipped unit tests and browser tests. Issues with test results reported for unit tests run through unittest_runner.py 1,This happens when no expectations file is provided. In the JSON results for a test where self.skipTest() is called, the expected field shows 'PASS'. I believe it should show as 'SKIP', since that is what the expected value is when a test run through browser_test_runner.py is skipped and no expectations file is provided. 2,This happens when an expectations file is passed. If a test calls self.skipTest and Skip is not in the expectations for that test in the file, then a regression is reported and a non-zero exit code is returned. 3, This happens when an expectations file is passed. If a test is skipped through the command line, i.e --skip=*pass_test, then the expectated field shows as 'SKIP' even though it does not have skip in its expectations in the file. 4, This happens when an expectations file is passed. If a test has Skip in its expectations in the file, the expected field shows as 'PASS'. Issues with test results reported for browser tests run through browser_test_runner.py All issues happen when a expectations file is passed 1, If a test has Skip in its expectation in the file, the expectation shows as PASS 2, If a test does not have Skip in its expectations in the file and calls self.skipTest(), a regression is reported. Because of that a non zero exit code is returned. 3, If a test does not have Skip in its expectations in the file, and a test is skipped via command line arguments, i.e --skip=*pass_test, then the expectations for the test do not show in the expected field in the results. It is set to SKIP.
,
Today
(10 hours ago)
As reference, this is the doc for the results format: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/json_test_results_format.md
,
Today
(9 hours ago)
+Stephen, could you please review Rakib's understanding of the json test results format and make sure it is correct?
,
Today
(8 hours ago)
I don't know how consistent the test results system is for skipped tests. I haven't really dealt with it before. The problems you're describing sound like they are issues though, although I'm not sure how much it matters. Are you seeing issues with some tool interacting with these test results in incorrect ways? For number 3, where they pass the skip info with the command line, I'm not sure if it should show up as PASS in the test results.
,
Today
(5 hours ago)
Hi Stephen I'm not seeing issues with a tool. I am working on moving GPU test expectations to the new expectations format. I noticed some inconsistencies in the JSON results when an expectations file is passed to Typ. The inconsistencies are in the expected field for a test that is skipped. When an expectations file is passed to Typ, I believe for a test that is skipped, the expectations for the test that is specified in the file passed should be in the expected field of the results for the test. But in some cases I am not seeing that. Regards Rakib
,
Today
(4 hours ago)
Hi I am not going to do anything about issue number 2 for both unittest_runner.py and browser_test_runner.py. It seems that is the desired outcome. I found a unit test which tests for a regression if self.skipTest() is called and 'Skip' is not in the expectations for the test in the passed expectations file. Rakib
,
Today
(4 hours ago)
,
Today
(4 hours ago)
I think that it sounds like there are a few bugs with how TYP interacts with skipped tests. I'm not an expert on typ, so you should get people more familiar with that to weigh in on that. kbr@ and dprank@ are who I would ask, although they're currently both OOO I think. |
||
►
Sign in to add a comment |
||
Comment 1 by crouleau@chromium.org
, Jan 18 (4 days ago)Cc: st...@chromium.org