graphics_WebGLPerformance: report sumOfFailed instead zero-to-nonzero |
||
Issue descriptionWhile investigating crbug.com/787087 , we noticed that the graphics_WebGLPerformance test outputs non-intuitive graphs. For example, this graph reports a zero-to-nonzero regression: https://chromeperf.appspot.com/group_report?bug_id=787087 In actuality, the test consists of 67 micro-benchmarks: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/client/site_tests/graphics_WebGLPerformance/graphics_WebGLPerformance.py For crbug.com/787087 , the number of failing tests jumped from 2 to 67 on Nyan. The zero to non-zero metric apparently is based on the median of previous test results. On Intel, the number of failing tests are zero -- so some added granularity would be useful (i.e, reporting sumOfFailed).
,
Nov 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/197c03aedee027bc2ba8d6f4c438da9daf903b09 commit 197c03aedee027bc2ba8d6f4c438da9daf903b09 Author: Po-Hsien Wang <pwang@chromium.org> Date: Tue Nov 21 07:25:48 2017 graphics_WebGLPerformance: report number of failures We moved the code to be calculated in python. So, lets report the actual number of failures in this test so that the graph should be clearer to understand. BUG= chromium:787199 TEST=test_that Change-Id: I1fd745ac6f756e5415f4e3d1b31f42b41a191afd Reviewed-on: https://chromium-review.googlesource.com/780390 Commit-Ready: Po-Hsien Wang <pwang@chromium.org> Tested-by: Po-Hsien Wang <pwang@chromium.org> Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org> [modify] https://crrev.com/197c03aedee027bc2ba8d6f4c438da9daf903b09/client/site_tests/graphics_WebGLPerformance/graphics_WebGLPerformance.py
,
Nov 22 2017
Test now reports 65 failures. https://chromeperf.appspot.com/report?sid=0b6c02013c34f644da423737b4ef429c9114069b2b7e9a8584382cb60778703b&start_rev=31970000988400000&end_rev=32740001015100000 |
||
►
Sign in to add a comment |
||
Comment 1 by pwang@chromium.org
, Nov 21 2017