New issue
Advanced search Search tips

Issue 916434 link

Starred by 1 user
Status: Duplicate
Merged: issue 698902
Owner:
lin....@intel.com
Closed: Jan 3
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Mark cases expected "FAIL" in WebGL CTS test result

Project Member Reported by yang...@intel.com, Dec 19 
content/test/gpu/run_gpu_integration_test.py is used to run WebGL CTS, and with option "--write-full-results-to" we can get all the test results.
However in this result file, cases expected to be "FAIL" marked in webgl(2)_conformance_expectations.py would still be logged as "PASS" like below.
"WebglConformance_deqp_functional_gles3_lifetime": {
  "expected": "PASS", 
  "actual": "PASS", 
  "times": [
    1.2977
  ]
},

Can we change "expected" to "FAIL" here, and have "actual" reflect the actual results, either "FAIL" (no change) or "PASS" (improvement)? In this way, we can better know all the "FAIL" cases, and have some script to check improvements automatically.

Comment 1 by kbr@chromium.org, Dec 19

Components: Internals>GPU>Testing
Owner: ----
Status: Available (was: Untriaged)
Sorry but we can't do that in the test harness itself. Chrome's automated testing infrastructure has too many dependencies on "FAIL" not showing up in those files unless the test failed unexpectedly.

However, we could write some code to post-process the JSON file. It would take into consideration webgl_conformance_expectations.py or webgl2_conformance_expectations.py, and use Telemetry to create a browser on the machine - because that's needed to figure out what expectations to apply.

Then we could change the "PASS" results to "FAIL" results for those expected failures.

parse_common_test_results in src/testing/scripts/common.py can parse the JSON file written by --write-full-results-to.

Would a member of your team be able to take on this task? We can integrate the result into webgl_conformance_integration_test.py.

Comment 2 by yang...@intel.com, Dec 21

Owner: yang...@intel.com
I will look into this.

Comment 3 by kbr@chromium.org, Dec 21 

Thank you Yang.

Comment 4 by lin....@intel.com, Jan 3 

Found a similar issue crbug.com/698902

Comment 5 by kainino@chromium.org, Jan 3

Mergedinto: 698902
Status: Duplicate (was: Available)
Thanks for finding that! It looks like the same issue to me. Please reopen if not.

Comment 6 by lin....@intel.com, Jan 7

Owner: lin....@intel.com
►

Sign in to add a comment