webgl_conformance_tests is not uploading results to test-results |
||||||||
Issue descriptionFrom Pawel's email: https://bugs.chromium.org/p/chromium/issues/detail?id=596622 is an interesting example because it's reported as flaky step bug, whereas looking at flake occurrences (https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM) reveals specific flaky tests, e.g.: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/223551 - WebglConformance.conformance_ogles_GL_atan_atan_009_to_012 https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/221910 - WebglConformance.conformance_attribs_gl_vertexattribpointer https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/221792 - WebglConformance.conformance_extensions_angle_instanced_arrays_out_of_bounds
,
May 19 2016
This specific step is not split into tests because the step is not uploading results to test-results when run on swarming. This is a prerequisite for tests to be identified. I'm going to have a look and see if I can add such support, but generally this is responsibility of the step owners.
,
May 19 2016
Here is an example output of the test from swarming:
{"failures": ["WebglConformance.conformance_textures_image_bitmap_from_image_bitmap_tex_2d_rgb_rgb_unsigned_short_5_6_5"], "valid": true}
This is not test-results-compatible, but changing it would be trivial:
{
"tests": {
"WebglConformance.conformance_textures_image_bitmap_from_image_bitmap_tex_2d_rgb_rgb_unsigned_short_5_6_5": {
"expected": "PASS",
"actual": "FAIL"
}
},
"interrupted": false,
"path_delimiter": ".",
"version": 3,
"seconds_since_epoch": 1406662283.764424,
"num_failures_by_type": {
"FAIL": 1
}
}
I am going to assign this issue to Ken, who added the test description to the https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/chromium.gpu.json. Please triage this to the test launcher author and have them write supported GTest output, after which it can be uploaded to test-results. The latter needs to be enabled in the steps.py similarly to how SwarmingGTestTest does it (see https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/recipe_modules/chromium_tests/steps.py#905).
,
May 19 2016
I'm going to block this on changing over the GPU tests to Ned's simplified unit test runner. We don't want to do any more work on the old harness. Once this cutover is done we can revisit the issue. In the meantime, if anyone wants to write a simple JSON transformer in src/testing/scripts/run_telemetry_benchmark_as_googletest.py please feel free.
,
May 19 2016
Downgrading to P2 as this is not as urgent for our team as some other tasks like this harness cutover, and sharding our Telemetry-based tests. If someone else considers this higher priority feel free to take and reprioritize.
,
May 20 2016
Just for reference test-results JSON format is described here: https://www.chromium.org/developers/the-json-test-results-format
,
Jun 29 2016
,
Dec 2 2016
,
Feb 6 2017
,
Jan 16
,
Jan 16
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by serg...@chromium.org
, May 19 2016