New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 914200 link

Starred by 1 user
Status: Available
Owner: ----
Cc:
█ kbr@chromium.org
bsheedy@chromium.org
crouleau@chromium.org
zmo@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug

Blocking:
issue 850107



Sign in to add a comment

GPU pixel tests don't highlight differing pixels in bright red

Project Member Reported by sunn...@chromium.org, Dec 12 
See for example this run: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20FYI%20Release%20%28Nexus%205%29/12423 

The failing test is gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_CanvasLowLatency2D

Note: The run is marked as "Success" even though there's a failing pixel test. That's been fixed.

Swarming Logs: https://chromium-swarm.appspot.com/task?id=41b7f2a206890110&refresh=10&show_raw=1

Test results: http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?6eaa783f6d885acd1fab3c2d83466ef47026815f_Android_FYI_Release_Nexus_5__telemetry

The diff image is pure black, but producing a diff locally using imagemagick shows a difference:

compare -compose src Pixel_CanvasLowLatency2D_gen.png Pixel_CanvasLowLatency2D_ref.png canvas_diff.png

I noticed the same thing on VP9 pixel test a few days back.  Attached images for both cases.

zmo@ asked me to look for numpy in the logs and it looks like there's a numpy exception in another test that's marked as failing.  I suspect the same happens for the above test, but the exception doesn't get printed because there's one already (images not matching).

Expected exception while running Pixel_SolidColorBackground

Traceback (most recent call last):
  _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:155
    self.RunActualGpuTest(url, *args)
  RunActualGpuTest at content/test/gpu/gpu_tests/pixel_integration_test.py:150
    int(page.test_rect[3] * dpr))
  Crop at third_party/catapult/telemetry/telemetry/util/image_util.py:113
    return impl.Crop(image, left, top, width, height)
  Crop at third_party/catapult/telemetry/telemetry/internal/image_processing/image_util_numpy_impl.py:155
    raise ValueError('Invalid dimensions')
ValueError: Invalid dimensions

Locals:
  height     : 300
  image      : array([[[  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        ..., 
        [  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0]],

       [[  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        ..., 
        [  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0]],

       [[  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        ..., 
        [  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0]],

       ..., 
       [[  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        ..., 
        [  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0]],

       [[  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        ..., 
        [  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0]],

       [[  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0],
        ..., 
        [  0, 128,   0],
        [  0, 128,   0],
        [  0, 128,   0]]], dtype=uint8)
  img_height : 1533
  img_width  : 1080
  left       : 1500
  top        : 1500
  width      : 300
Pixel_CanvasLowLatency2D_gen.png
4.6 KB View Download
Pixel_CanvasLowLatency2D_ref.png
4.3 KB View Download
Pixel_Video_VP9_gen.png
127 KB View Download
Pixel_Video_VP9_ref.png
127 KB View Download
Pixel_Video_VP9_diff.png
6.9 KB View Download

Comment 1 by sunn...@chromium.org, Dec 12 

(attached black diff by accident earlier)
Pixel_CanvasLowLatency2D_diff.png
402 bytes View Download

Comment 2 by kbr@chromium.org, Dec 12

Blocking: 850107
Cc: bsheedy@chromium.org
It's a longstanding problem that Chromium's GPU pixel tests don't highlight differing pixels in bright red. This is a problem with the numpy and cv2 operators we're using to produce the diffs.

If there is a quick fix, we should do it, but at this point bsheedy@ is working on migrating the GPU pixel tests to use Skia's Gold tool. This should make rebaselining a lot easier and less fragile. Perhaps we should invest our development time on Issue 850107.

Comment 3 by crouleau@chromium.org, Dec 17

Blockedon: 850107
Blocking: -850107
Labels: -Pri-1 Pri-2
Summary: GPU pixel tests don't highlight differing pixels in bright red (was: Pixel test diffs are always black)
Should this be Blocked on 850107 rather than Blocking 850107?

If this is a longstanding problem, then I imagine it doesn't require P1 priority either?

This is within Telemetry, so I guess I could take a look if no one else can, but I'm not familiar with the code at all. I wonder if Brian might be a better owner?

Comment 4 by crouleau@chromium.org, Dec 17

Cc: crouleau@chromium.org
Owner: bsheedy@chromium.org
Please assign back to me if you think I should own this as part of Telemetry purview.

Comment 5 by kbr@chromium.org, Dec 17

Blockedon: -850107
Blocking: 850107
Owner: ----
Status: Available (was: Assigned)
Sunny was potentially interested in helping improve this logic, and we were wondering whether a quick fix to Telemetry's image_util might be possible to make these image diffs more visible; that's why it blocked Issue 850107, since switching to Gold will solve this categorically.

The code for the numpy backend is here:

https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/image_processing/image_util_numpy_impl.py?q=image_util_numpy&sq=package:chromium&g=0&l=105

and I think at this point that it should always be going down the cv2 code path, at least on the bots. (Developers should hopefully be using vpython for this suite.) Not sure which image operators would provide the most visible diffs.

bsheedy's going full steam on the Gold integration so I'd rather not ask him to task switch to work on this.

Comment 6 by crouleau@chromium.org, Dec 17 

Sounds good. I'm happy to review code if someone takes this on. However, I have no idea whether a quick fix is possible or not.

Comment 7 by benhenry@google.com, Jan 16 (6 days ago)

Components: Test>Telemetry

Comment 8 by benhenry@google.com, Jan 16 (6 days ago)

Components: -Tests>Telemetry
►

Sign in to add a comment