Make collisions in GPU test expectations fatal |
|||
Issue descriptionCurrently collisions in the GPU test expectations files are non-fatal, which has led to errors in the files where it's ambiguous which expectation will apply. This is especially problematic during pixel test rebaselines. Collisions need to be made fatal.
,
Apr 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/16ad333175b42977d1d6d7a4a7c29bf04607d927 commit 16ad333175b42977d1d6d7a4a7c29bf04607d927 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Apr 13 01:47:48 2018 Make collisions in text expectations fatal. If two different test expectations for the same test apply on the current browser configuration, fail hard. This has happened a couple of times already in the pixel test expectations and it has led to confusion. Added verify_all_expectations_unittest which checks for collisions in all of the GPU test expectations subclasses in this directory, taking into account configurations on the waterfall. This is automatically run during presubmit, along with all of the other GPU unittests, to make life easier on developers. Removed expectations for Pixel_CSS3DBlueBox, which should be passing on macOS reliably now. Bug: 533690 , 706016 , 796558, 830228 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I64a604d38a4b21729cf0e5b969328d2b051bddbb Reviewed-on: https://chromium-review.googlesource.com/1002032 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kai Ninomiya <kainino@chromium.org> Cr-Commit-Position: refs/heads/master@{#550464} [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/pixel_expectations.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/test_expectations.py [add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations_unittest.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py [add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_test_util.py
,
Apr 13 2018
,
Apr 13 2018
Found a collision: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/1468 Exception: FATAL ERROR: the following tests had colliding expectations in the test expectations: Pixel_Video_VP9
,
Apr 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d16476d6ca785934db7b86e58fe8285b7015bd8c commit d16476d6ca785934db7b86e58fe8285b7015bd8c Author: Kenneth Russell <kbr@chromium.org> Date: Fri Apr 13 18:53:34 2018 Verify test expectation collisions on High Sierra. Fix a collision of Pixel_Video_VP9 on the Mac Intel 10.13 bot. Tbr: ynovikov@chromium.org No-Try: True Bug: 830228 Change-Id: Ib074b0a8adc5961faea3341c082dcdbd2b7ff19e Reviewed-on: https://chromium-review.googlesource.com/1012463 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#550701} [modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/pixel_expectations.py [modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py
,
Apr 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/16ad333175b42977d1d6d7a4a7c29bf04607d927 commit 16ad333175b42977d1d6d7a4a7c29bf04607d927 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Apr 13 01:47:48 2018 Make collisions in text expectations fatal. If two different test expectations for the same test apply on the current browser configuration, fail hard. This has happened a couple of times already in the pixel test expectations and it has led to confusion. Added verify_all_expectations_unittest which checks for collisions in all of the GPU test expectations subclasses in this directory, taking into account configurations on the waterfall. This is automatically run during presubmit, along with all of the other GPU unittests, to make life easier on developers. Removed expectations for Pixel_CSS3DBlueBox, which should be passing on macOS reliably now. Bug: 533690 , 706016 , 796558, 830228 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I64a604d38a4b21729cf0e5b969328d2b051bddbb Reviewed-on: https://chromium-review.googlesource.com/1002032 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kai Ninomiya <kainino@chromium.org> Cr-Commit-Position: refs/heads/master@{#550464} [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/pixel_expectations.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/test_expectations.py [add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations_unittest.py [modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py [add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_test_util.py
,
Apr 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d16476d6ca785934db7b86e58fe8285b7015bd8c commit d16476d6ca785934db7b86e58fe8285b7015bd8c Author: Kenneth Russell <kbr@chromium.org> Date: Fri Apr 13 18:53:34 2018 Verify test expectation collisions on High Sierra. Fix a collision of Pixel_Video_VP9 on the Mac Intel 10.13 bot. Tbr: ynovikov@chromium.org No-Try: True Bug: 830228 Change-Id: Ib074b0a8adc5961faea3341c082dcdbd2b7ff19e Reviewed-on: https://chromium-review.googlesource.com/1012463 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#550701} [modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/pixel_expectations.py [modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py |
|||
►
Sign in to add a comment |
|||
Comment 1 by kbr@chromium.org
, Apr 8 2018