New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 830228 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
OOO until 2019-01-24
Closed: Apr 2018
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 575305



Sign in to add a comment

Make collisions in GPU test expectations fatal

Project Member Reported by kbr@chromium.org, Apr 8 2018

Issue description

Currently collisions in the GPU test expectations files are non-fatal, which has led to errors in the files where it's ambiguous which expectation will apply. This is especially problematic during pixel test rebaselines. Collisions need to be made fatal.

 

Comment 1 by kbr@chromium.org, Apr 8 2018

Blocking: 575305

Comment 2 by kbr@chromium.org, Apr 13 2018

Blockedon: 575305
Blocking: -575305
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/16ad333175b42977d1d6d7a4a7c29bf04607d927

commit 16ad333175b42977d1d6d7a4a7c29bf04607d927
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Apr 13 01:47:48 2018

Make collisions in text expectations fatal.

If two different test expectations for the same test apply on the
current browser configuration, fail hard. This has happened a couple
of times already in the pixel test expectations and it has led to
confusion.

Added verify_all_expectations_unittest which checks for collisions in
all of the GPU test expectations subclasses in this directory, taking
into account configurations on the waterfall. This is automatically
run during presubmit, along with all of the other GPU unittests, to
make life easier on developers.

Removed expectations for Pixel_CSS3DBlueBox, which should be passing
on macOS reliably now.

Bug:  533690 ,  706016 , 796558,  830228 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I64a604d38a4b21729cf0e5b969328d2b051bddbb
Reviewed-on: https://chromium-review.googlesource.com/1002032
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kai Ninomiya <kainino@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550464}
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/test_expectations.py
[add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations_unittest.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py
[add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_test_util.py

Comment 4 by kbr@chromium.org, Apr 13 2018

Status: Fixed (was: Assigned)
Found a collision:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/1468
  Exception: FATAL ERROR: the following tests had colliding expectations in the test expectations: Pixel_Video_VP9
Project Member

Comment 6 by bugdroid1@chromium.org, Apr 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d16476d6ca785934db7b86e58fe8285b7015bd8c

commit d16476d6ca785934db7b86e58fe8285b7015bd8c
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Apr 13 18:53:34 2018

Verify test expectation collisions on High Sierra.

Fix a collision of Pixel_Video_VP9 on the Mac Intel 10.13 bot.

Tbr: ynovikov@chromium.org
No-Try: True
Bug:  830228 
Change-Id: Ib074b0a8adc5961faea3341c082dcdbd2b7ff19e
Reviewed-on: https://chromium-review.googlesource.com/1012463
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550701}
[modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py

Project Member

Comment 7 by bugdroid1@chromium.org, Apr 17 2018

Labels: merge-merged-testbranch
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/16ad333175b42977d1d6d7a4a7c29bf04607d927

commit 16ad333175b42977d1d6d7a4a7c29bf04607d927
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Apr 13 01:47:48 2018

Make collisions in text expectations fatal.

If two different test expectations for the same test apply on the
current browser configuration, fail hard. This has happened a couple
of times already in the pixel test expectations and it has led to
confusion.

Added verify_all_expectations_unittest which checks for collisions in
all of the GPU test expectations subclasses in this directory, taking
into account configurations on the waterfall. This is automatically
run during presubmit, along with all of the other GPU unittests, to
make life easier on developers.

Removed expectations for Pixel_CSS3DBlueBox, which should be passing
on macOS reliably now.

Bug:  533690 ,  706016 , 796558,  830228 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I64a604d38a4b21729cf0e5b969328d2b051bddbb
Reviewed-on: https://chromium-review.googlesource.com/1002032
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kai Ninomiya <kainino@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550464}
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/test_expectations.py
[add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_expectations_unittest.py
[modify] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py
[add] https://crrev.com/16ad333175b42977d1d6d7a4a7c29bf04607d927/content/test/gpu/gpu_tests/webgl_test_util.py

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d16476d6ca785934db7b86e58fe8285b7015bd8c

commit d16476d6ca785934db7b86e58fe8285b7015bd8c
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Apr 13 18:53:34 2018

Verify test expectation collisions on High Sierra.

Fix a collision of Pixel_Video_VP9 on the Mac Intel 10.13 bot.

Tbr: ynovikov@chromium.org
No-Try: True
Bug:  830228 
Change-Id: Ib074b0a8adc5961faea3341c082dcdbd2b7ff19e
Reviewed-on: https://chromium-review.googlesource.com/1012463
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550701}
[modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/d16476d6ca785934db7b86e58fe8285b7015bd8c/content/test/gpu/gpu_tests/verify_all_expectations_unittest.py

Sign in to add a comment