New issue
Advanced search Search tips

Issue 872733 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 13
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

cc_unittests (LayerTreeHostBlendingPixelTest) failing flakily on chromium.memory/Linux CFI

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Aug 9

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of vitaliii@google.com

cc_unittests failing on chromium.memory/Linux CFI

Builders failed on: 
- Linux CFI: 
  https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI


 
Out of last 25 builds, 6 failed.

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9454
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMask_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrix_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9465

LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
 
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9471
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL


https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9472
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL_TextureRect
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9477
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9478
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
After counting

      6 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
      4 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
      3 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL
      3 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
      3 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
      2 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
      2 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
      2 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL
      1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrix_GL
      1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMask_GL
      1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL_TextureRect
      1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersAA_GL
      1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassAA_GL

Components: Internals>GPU>Testing
Owner: kbr@chromium.org
Status: Assigned (was: Available)
These tests flake even on passing runs, eg https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9476 has:

[ RUN      ] LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
../../cc/test/layer_tree_test.cc:1014: Failure
Failed
Test timed out
Stack trace:
#0 0x00000188eca1 testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop()
#1 0x00000188e88d testing::internal::AssertHelper::operator=()
#2 0x000001ab2dfe cc::LayerTreeTest::RunTest()
#3 0x000001aaeb35 cc::LayerTreePixelTest::RunPixelTest()
#4 0x000001aad52a cc::LayerTreeHostPixelResourceTest::RunPixelResourceTest()
#5 0x0000012a88f0 cc::(anonymous namespace)::LayerTreeHostBlendingPixelTest::RunBlendingWithRenderPass()
[  FAILED  ] LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL (8520 ms)
[3211/3220] LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL (8520 ms)

but then it passes on the retry.

Assigning to kbr for gpu testing.
Disabling the tests with more than one occurrence in https://chromium-review.googlesource.com/c/chromium/src/+/1169204.
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 9

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd08401c92882ed1c69503217a67b17274439621

commit bd08401c92882ed1c69503217a67b17274439621
Author: vitaliii <vitaliii@chromium.org>
Date: Thu Aug 09 14:31:51 2018

Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky.

Disable on Linux

LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL

because flaky.

TBR=ajuma@chromium.org

Bug:872733
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I61a0b1cc99172da0437eec8f10149f0c46c966eb
Reviewed-on: https://chromium-review.googlesource.com/1169204
Reviewed-by: vitaliii <vitaliii@chromium.org>
Commit-Queue: vitaliii <vitaliii@chromium.org>

[modify] https://crrev.com/bd08401c92882ed1c69503217a67b17274439621/cc/trees/layer_tree_host_pixeltest_blending.cc

Labels: -Sheriff-Chromium
Has an owner + most common flakiness is disabled -> out of sheriff queue for now
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 9

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/edb50c1c51ee6f2e5af7c5e893712a42039f0fc2

commit edb50c1c51ee6f2e5af7c5e893712a42039f0fc2
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 19:17:53 2018

Revert "Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky."

This reverts commit bd08401c92882ed1c69503217a67b17274439621.

Reason for revert: Due to Gerrit outage  http://crbug.com/872722 , we are reverting this CL. Please, re-land it after all clear is given. If you have questions, please ask on the bug. Sorry for the inconvenience.

Original change's description:
> Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky.
> 
> Disable on Linux
> 
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL
> 
> because flaky.
> 
> TBR=ajuma@chromium.org
> 
> Bug:872733
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
> Change-Id: I61a0b1cc99172da0437eec8f10149f0c46c966eb
> Reviewed-on: https://chromium-review.googlesource.com/1169204
> Reviewed-by: vitaliii <vitaliii@chromium.org>
> Commit-Queue: vitaliii <vitaliii@chromium.org>

TBR=vitaliii@chromium.org

Change-Id: I05f9111829ee71617ab929cf4e2ed1d4982675d5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  872733 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Reviewed-on: https://chromium-review.googlesource.com/1169850
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/edb50c1c51ee6f2e5af7c5e893712a42039f0fc2/cc/trees/layer_tree_host_pixeltest_blending.cc

Components: -Internals>GPU>Testing Internals>Compositing
Labels: OS-Linux Type-Bug
Owner: enne@chromium.org
These are actually a different set of tests than the pixel tests which run on physical hardware. enne@, could you please triage this?

Cc: p...@chromium.org
Labels: -Pri-1 Pri-2
I suspect these tests are flakily timing out because they're doing rasterization with swift shader and are on memory bots with extra CFI instrumentation.  This is quite similar to  issue 430400  which had the same problem on DrMemory.  I don't think this is some sort of deadlock here, and think these tests would pass if given more time.

Maybe we just need to increase the timeout here?

cc-ing pcc here.  I'm curious if this has been a problem with other tests on the CFI bots.
It looks like this CL (bd08401c92882ed1c69503217a67b17274439621) and its revert (bd08401c92882ed1c69503217a67b17274439621) landed during the Great Gerrit Outage and were never assigned a Cr-Commit-Position, and were subsequently reset out of existence.

So this has not actually been disabled. Attempting to reland bd08401c92882ed1c69503217a67b17274439621 now.
Project Member

Comment 12 by bugdroid1@chromium.org, Aug 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0e3bef3eb7adc9b040795aff375e37b1439bcf4f

commit 0e3bef3eb7adc9b040795aff375e37b1439bcf4f
Author: vitaliii <vitaliii@chromium.org>
Date: Fri Aug 10 07:30:10 2018

Reland "Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky."

This is a reland of bd08401c92882ed1c69503217a67b17274439621
(Note that the original CL is no longer in the tree; it landed during yesterday's
Gerrit outage and was reset out of existence. So this is the first time it has
"officially" landed in the tree.)

Original change's description:
> Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky.
>
> Disable on Linux
>
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
> LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL
>
> because flaky.
>
> TBR=ajuma@chromium.org
>
> Bug:872733
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
> Change-Id: I61a0b1cc99172da0437eec8f10149f0c46c966eb
> Reviewed-on: https://chromium-review.googlesource.com/1169204
> Reviewed-by: vitaliii <vitaliii@chromium.org>
> Commit-Queue: vitaliii <vitaliii@chromium.org>

TBR=ajuma@chromium.org

Bug:  872733 
Change-Id: I044022d2db22b1cccb20692cf7300c3784ba2859
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Reviewed-on: https://chromium-review.googlesource.com/1170543
Reviewed-by: Matt Giuca <mgiuca@chromium.org>
Commit-Queue: Matt Giuca <mgiuca@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582085}
[modify] https://crrev.com/0e3bef3eb7adc9b040795aff375e37b1439bcf4f/cc/trees/layer_tree_host_pixeltest_blending.cc

Project Member

Comment 13 by bugdroid1@chromium.org, Aug 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d3df98f1195e5772ddbb2e369527e7fa3a920ce9

commit d3df98f1195e5772ddbb2e369527e7fa3a920ce9
Author: Adrienne Walker <enne@chromium.org>
Date: Fri Aug 10 22:44:54 2018

cc: Increase layer tree test timeout for CFI

Also, generally increase the timeout for layer tree tests.  Most things
don't need this, however the GL versions of pixel tests end up spending
quite a bit of time to compile shaders.  When run in parallel, this
only exacerbates the problem and tests can time out.  Be more generous
so that folks stop turning off these tests.

Bug:  872733 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I7c08212d454bf0d9ae6ea445fcb46b48b73aea88
Reviewed-on: https://chromium-review.googlesource.com/1171377
Reviewed-by: weiliangc <weiliangc@chromium.org>
Commit-Queue: enne <enne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582367}
[modify] https://crrev.com/d3df98f1195e5772ddbb2e369527e7fa3a920ce9/cc/test/layer_tree_test.cc

Status: Fixed (was: Assigned)
Reverted test disabling in https://chromium-review.googlesource.com/c/chromium/src/+/1171425
Project Member

Comment 15 by bugdroid1@chromium.org, Aug 21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d1f9ed886de3930f68a8b042fda843ca214eac96

commit d1f9ed886de3930f68a8b042fda843ca214eac96
Author: Adrienne Walker <enne@chromium.org>
Date: Tue Aug 21 03:00:57 2018

cc: Split up blending tests to avoid timeouts

The current tests time out quite a bit when using swiftshader.  I
suspect that this is due to shader compilation on a per-blend mode
basis, as reducing pixels rastered did not seem to make any difference.
The goal here is to reduce the maximum length of any one test so that
fewer tests time out and these tests keep causing flakiness and getting
disabled.

This test breaks up each individual blend mode test into parameterizing
by blend mode and rastering what used to be a single column in a larger
image into its own image.  Instead of saving out a million data pngs,
instead this modifies the test framework to use expected pngs as well.
This assumes that Skia's blend modes work properly and verifies that
software and gl renderer match the output of those blend modes on an
SkCanvas.

In terms of total time of the test, splitting each test up into 16
different tests on Linux debug for me has each individual test
completing in 1/4 of the time of the original (1/4x * 16 = 4x increase
overall).  However, when running the entire test suite in parallel, this
patch speeds up running cc_unittests from ~70s to ~60s overall.  So,
I think this is overall a win, both in terms of individual and total
test time.

Bug:  870326 , 872733 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I4791a0a2369f9f17af7bb2a6ba95d7f29dee8b42
Reviewed-on: https://chromium-review.googlesource.com/1180570
Reviewed-by: weiliangc <weiliangc@chromium.org>
Commit-Queue: enne <enne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584616}
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_resource_test.cc
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_resource_test.h
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_test.cc
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_test.h
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/pixel_test_utils.cc
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/pixel_test_utils.h
[modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/trees/layer_tree_host_pixeltest_blending.cc
[delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_and_filter.png
[delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_render_pass.png
[delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_render_pass_mask.png
[delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_transparent.png
[delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_with_root.png

Sign in to add a comment