New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 613044 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
OOO until 2019-01-24
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 352807
issue 636153

Blocking:
issue 545408



Sign in to add a comment

webgl_conformance_tests is not uploading results to test-results

Project Member Reported by serg...@chromium.org, May 19 2016

Issue description

From Pawel's email:

https://bugs.chromium.org/p/chromium/issues/detail?id=596622 is an interesting example because it's reported as flaky step bug, whereas looking at flake occurrences (https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM) reveals specific flaky tests, e.g.:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/223551 - WebglConformance.conformance_ogles_GL_atan_atan_009_to_012
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/221910 - WebglConformance.conformance_attribs_gl_vertexattribpointer
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/221792 - WebglConformance.conformance_extensions_angle_instanced_arrays_out_of_bounds
 
Blocking: 545408
This specific step is not split into tests because the step is not uploading results to test-results when run on swarming. This is a prerequisite for tests to be identified. I'm going to have a look and see if I can add such support, but generally this is responsibility of the step owners.
Cc: phajdan.jr@chromium.org
Owner: kbr@chromium.org
Summary: webgl_conformance_tests is not uploading results to test-results (was: Flakiness pipeline reports test flakes as step flakes)
Here is an example output of the test from swarming:

{"failures": ["WebglConformance.conformance_textures_image_bitmap_from_image_bitmap_tex_2d_rgb_rgb_unsigned_short_5_6_5"], "valid": true}

This is not test-results-compatible, but changing it would be trivial:

{
  "tests": {
    "WebglConformance.conformance_textures_image_bitmap_from_image_bitmap_tex_2d_rgb_rgb_unsigned_short_5_6_5": {
      "expected": "PASS",
      "actual": "FAIL"
    }
  },
  "interrupted": false,
  "path_delimiter": ".",
  "version": 3,
  "seconds_since_epoch": 1406662283.764424,
  "num_failures_by_type": {
    "FAIL": 1
  }
}

I am going to assign this issue to Ken, who added the test description to the https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/chromium.gpu.json. Please triage this to the test launcher author and have them write supported GTest output, after which it can be uploaded to test-results. The latter needs to be enabled in the steps.py similarly to how SwarmingGTestTest does it (see https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/recipe_modules/chromium_tests/steps.py#905).

Comment 4 by kbr@chromium.org, May 19 2016

Blockedon: 352807
Cc: nedngu...@google.com
Components: Internals>GPU>Testing Tests>Telemetry
I'm going to block this on changing over the GPU tests to Ned's simplified unit test runner. We don't want to do any more work on the old harness. Once this cutover is done we can revisit the issue. In the meantime, if anyone wants to write a simple JSON transformer in src/testing/scripts/run_telemetry_benchmark_as_googletest.py please feel free.

Comment 5 by kbr@chromium.org, May 19 2016

Labels: -Pri-1 Pri-2
Downgrading to P2 as this is not as urgent for our team as some other tasks like this harness cutover, and sharding our Telemetry-based tests. If someone else considers this higher priority feel free to take and reprioritize.

Just for reference test-results JSON format is described here: https://www.chromium.org/developers/the-json-test-results-format
Components: Infra>Flakiness>Pipeline
Components: -Infra>Flakiness>Pipeline Infra>Flakiness>Dashboard

Comment 9 by kbr@chromium.org, Feb 6 2017

Blockedon: 636153
Blocking this on the switch of the browser_test_runner harness to typ.

Components: Test>Telemetry
Components: -Tests>Telemetry

Sign in to add a comment