Failure count in chromeperf claims eve have zero failure on gles3 but stainless shows 2 failure https://chromeperf.appspot.com/report?sid=ad58bf7cbc1c08a1a43933b63f740a77559a197b04f364f37a2edd0e43250e17 https://stainless.corp.google.com/search?exclude_retried=true&first_date=2018-04-05&master_builder_name=&builder_name_number=&shard=&exclude_acts=true&builder_name=&master_builder_name_number=&owner=&retry=&exclude_cts=false&exclude_non_production=false&hostname=&board=&test=graphics_dEQP.gles3-master&exclude_not_run=false&build=%5ER67%5C-10564%5C.0%5C.0%24&status=FAIL&status=ERROR&status=ABORT&reason=&waterfall=&suite=&last_date=2018-04-11&exclude_non_release=true&exclude_au=true&model=%5Ecave%24&view=list It may due to different result parsing mechanics we use in the test.
Might be that the dEQP versions are different.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/beb0b8be5f89583f2b99b7067de3b36a6a847d65 commit beb0b8be5f89583f2b99b7067de3b36a6a847d65 Author: Po-Hsien Wang <pwang@chromium.org> Date: Wed Apr 11 23:18:28 2018 graphics_dEQP: fix chromeperf failure inconsistency Failure count is different on chromeperf than on stainless since 'Unknown' failures are not counted as failure. This CL, count it failure if no log file exist. BUG= chromium:831845 TEST=None Change-Id: Idd32dc10bdac71589e47446cfc8f2ea00179ddd4 Reviewed-on: https://chromium-review.googlesource.com/1008840 Commit-Queue: Po-Hsien Wang <ddmail@google.com> Tested-by: Po-Hsien Wang <ddmail@google.com> Trybot-Ready: Po-Hsien Wang <ddmail@google.com> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/beb0b8be5f89583f2b99b7067de3b36a6a847d65/client/site_tests/graphics_dEQP/graphics_dEQP.py
Checked some data point and verified that its consistent now. https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/191701758-chromeos-test/chromeos4-row3-rack2-host17/graphics_dEQP.gles31-master
Comment 1 by ihf@chromium.org
, Apr 11 2018Labels: M-67 OS-Chrome