crosperf reporting status of failed tests as pass |
||
Issue descriptionSmoothness.tough_webgl_cases fails sometimes but crosperf reports the test status as pass. =========================================== Results report for 'toolchain_experiment.txt' =========================================== ------------------------------------------- Summary ------------------------------------------- Benchmark: smoothness.tough_webgl_cases; Iterations: 3 keys official_image (pass:0 fail:3) master_tot_afdo (pass:0 fail:3) Keys Amean StdDev StdDev/Mean GmeanSpeedup p-value Amean StdDev StdDev/Mean GmeanSpeedup p-value Results Fail ? ? Fail ? ? ------------------------------------------- Benchmark Run Status ------------------------------------------- official_image: smoothness.tough_webgl_cases (1) SUCCEEDED official_image: smoothness.tough_webgl_cases (2) SUCCEEDED official_image: smoothness.tough_webgl_cases (3) SUCCEEDED The generated json file also has the test status as 'pass': true:
,
Jun 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/toolchain-utils/+/92a823f8fa2e20a695e02f7aaa7687f57ed9f223 commit 92a823f8fa2e20a695e02f7aaa7687f57ed9f223 Author: Manoj Gupta <manojgupta@google.com> Date: Tue Jun 06 20:18:17 2017 Crosperf: Mark benchmark run as failed if it contains no results. Sometimes telemetry tests do not fail but results are empty. In that case, mark the run as failed explicitly. BUG= chromium:729634 TEST=Tested manually on failing lummpy nightly tests. The generated report shows tests as failed as expected. Change-Id: Ia52af887e81ed630d7578f9abe93f9656c70512b Reviewed-on: https://chromium-review.googlesource.com/526392 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> Commit-Queue: Manoj Gupta <manojgupta@chromium.org> Trybot-Ready: Manoj Gupta <manojgupta@chromium.org> [modify] https://crrev.com/92a823f8fa2e20a695e02f7aaa7687f57ed9f223/crosperf/results_organizer.py
,
Jun 6 2017
,
Jun 6 2017
Note that this only impacts the status in the generated report file, and does not change the stdout or the benchmark's result cache. |
||
►
Sign in to add a comment |
||
Comment 1 by manojgupta@chromium.org
, Jun 6 2017