cc_unittests (LayerTreeHostBlendingPixelTest) failing flakily on chromium.memory/Linux CFI |
||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of vitaliii@google.com cc_unittests failing on chromium.memory/Linux CFI Builders failed on: - Linux CFI: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI
,
Aug 9
After counting
6 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL
4 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL
3 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL
3 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect
3 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL
2 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL
2 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL
2 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL
1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrix_GL
1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMask_GL
1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL_TextureRect
1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersAA_GL
1 LayerTreeHostBlendingPixelTest.BlendingWithRenderPassAA_GL
,
Aug 9
These tests flake even on passing runs, eg https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20CFI/9476 has: [ RUN ] LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL ../../cc/test/layer_tree_test.cc:1014: Failure Failed Test timed out Stack trace: #0 0x00000188eca1 testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop() #1 0x00000188e88d testing::internal::AssertHelper::operator=() #2 0x000001ab2dfe cc::LayerTreeTest::RunTest() #3 0x000001aaeb35 cc::LayerTreePixelTest::RunPixelTest() #4 0x000001aad52a cc::LayerTreeHostPixelResourceTest::RunPixelResourceTest() #5 0x0000012a88f0 cc::(anonymous namespace)::LayerTreeHostBlendingPixelTest::RunBlendingWithRenderPass() [ FAILED ] LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL (8520 ms) [3211/3220] LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL (8520 ms) but then it passes on the retry. Assigning to kbr for gpu testing.
,
Aug 9
Disabling the tests with more than one occurrence in https://chromium-review.googlesource.com/c/chromium/src/+/1169204.
,
Aug 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bd08401c92882ed1c69503217a67b17274439621 commit bd08401c92882ed1c69503217a67b17274439621 Author: vitaliii <vitaliii@chromium.org> Date: Thu Aug 09 14:31:51 2018 Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky. Disable on Linux LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL because flaky. TBR=ajuma@chromium.org Bug:872733 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel Change-Id: I61a0b1cc99172da0437eec8f10149f0c46c966eb Reviewed-on: https://chromium-review.googlesource.com/1169204 Reviewed-by: vitaliii <vitaliii@chromium.org> Commit-Queue: vitaliii <vitaliii@chromium.org> [modify] https://crrev.com/bd08401c92882ed1c69503217a67b17274439621/cc/trees/layer_tree_host_pixeltest_blending.cc
,
Aug 9
Has an owner + most common flakiness is disabled -> out of sheriff queue for now
,
Aug 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/edb50c1c51ee6f2e5af7c5e893712a42039f0fc2 commit edb50c1c51ee6f2e5af7c5e893712a42039f0fc2 Author: Andrii Shyshkalov <tandrii@chromium.org> Date: Thu Aug 09 19:17:53 2018 Revert "Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky." This reverts commit bd08401c92882ed1c69503217a67b17274439621. Reason for revert: Due to Gerrit outage http://crbug.com/872722 , we are reverting this CL. Please, re-land it after all clear is given. If you have questions, please ask on the bug. Sorry for the inconvenience. Original change's description: > Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky. > > Disable on Linux > > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL > > because flaky. > > TBR=ajuma@chromium.org > > Bug:872733 > Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel > Change-Id: I61a0b1cc99172da0437eec8f10149f0c46c966eb > Reviewed-on: https://chromium-review.googlesource.com/1169204 > Reviewed-by: vitaliii <vitaliii@chromium.org> > Commit-Queue: vitaliii <vitaliii@chromium.org> TBR=vitaliii@chromium.org Change-Id: I05f9111829ee71617ab929cf4e2ed1d4982675d5 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 872733 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel Reviewed-on: https://chromium-review.googlesource.com/1169850 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/edb50c1c51ee6f2e5af7c5e893712a42039f0fc2/cc/trees/layer_tree_host_pixeltest_blending.cc
,
Aug 9
These are actually a different set of tests than the pixel tests which run on physical hardware. enne@, could you please triage this?
,
Aug 9
I suspect these tests are flakily timing out because they're doing rasterization with swift shader and are on memory bots with extra CFI instrumentation. This is quite similar to issue 430400 which had the same problem on DrMemory. I don't think this is some sort of deadlock here, and think these tests would pass if given more time. Maybe we just need to increase the timeout here? cc-ing pcc here. I'm curious if this has been a problem with other tests on the CFI bots.
,
Aug 10
It looks like this CL (bd08401c92882ed1c69503217a67b17274439621) and its revert (bd08401c92882ed1c69503217a67b17274439621) landed during the Great Gerrit Outage and were never assigned a Cr-Commit-Position, and were subsequently reset out of existence. So this has not actually been disabled. Attempting to reland bd08401c92882ed1c69503217a67b17274439621 now.
,
Aug 10
,
Aug 10
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0e3bef3eb7adc9b040795aff375e37b1439bcf4f commit 0e3bef3eb7adc9b040795aff375e37b1439bcf4f Author: vitaliii <vitaliii@chromium.org> Date: Fri Aug 10 07:30:10 2018 Reland "Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky." This is a reland of bd08401c92882ed1c69503217a67b17274439621 (Note that the original CL is no longer in the tree; it landed during yesterday's Gerrit outage and was reset out of existence. So this is the first time it has "officially" landed in the tree.) Original change's description: > Disable some LayerTreeHostBlendingPixelTest on Linux, because flaky. > > Disable on Linux > > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrixAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskColorMatrixAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassWithMaskAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL_TextureRect > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersColorMatrixAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskColorMatrix_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassShadersWithMaskAA_GL > LayerTreeHostBlendingPixelTest.BlendingWithRenderPassColorMatrixAA_GL > > because flaky. > > TBR=ajuma@chromium.org > > Bug:872733 > Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel > Change-Id: I61a0b1cc99172da0437eec8f10149f0c46c966eb > Reviewed-on: https://chromium-review.googlesource.com/1169204 > Reviewed-by: vitaliii <vitaliii@chromium.org> > Commit-Queue: vitaliii <vitaliii@chromium.org> TBR=ajuma@chromium.org Bug: 872733 Change-Id: I044022d2db22b1cccb20692cf7300c3784ba2859 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel Reviewed-on: https://chromium-review.googlesource.com/1170543 Reviewed-by: Matt Giuca <mgiuca@chromium.org> Commit-Queue: Matt Giuca <mgiuca@chromium.org> Cr-Commit-Position: refs/heads/master@{#582085} [modify] https://crrev.com/0e3bef3eb7adc9b040795aff375e37b1439bcf4f/cc/trees/layer_tree_host_pixeltest_blending.cc
,
Aug 10
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d3df98f1195e5772ddbb2e369527e7fa3a920ce9 commit d3df98f1195e5772ddbb2e369527e7fa3a920ce9 Author: Adrienne Walker <enne@chromium.org> Date: Fri Aug 10 22:44:54 2018 cc: Increase layer tree test timeout for CFI Also, generally increase the timeout for layer tree tests. Most things don't need this, however the GL versions of pixel tests end up spending quite a bit of time to compile shaders. When run in parallel, this only exacerbates the problem and tests can time out. Be more generous so that folks stop turning off these tests. Bug: 872733 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel Change-Id: I7c08212d454bf0d9ae6ea445fcb46b48b73aea88 Reviewed-on: https://chromium-review.googlesource.com/1171377 Reviewed-by: weiliangc <weiliangc@chromium.org> Commit-Queue: enne <enne@chromium.org> Cr-Commit-Position: refs/heads/master@{#582367} [modify] https://crrev.com/d3df98f1195e5772ddbb2e369527e7fa3a920ce9/cc/test/layer_tree_test.cc
,
Aug 13
Reverted test disabling in https://chromium-review.googlesource.com/c/chromium/src/+/1171425
,
Aug 21
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d1f9ed886de3930f68a8b042fda843ca214eac96 commit d1f9ed886de3930f68a8b042fda843ca214eac96 Author: Adrienne Walker <enne@chromium.org> Date: Tue Aug 21 03:00:57 2018 cc: Split up blending tests to avoid timeouts The current tests time out quite a bit when using swiftshader. I suspect that this is due to shader compilation on a per-blend mode basis, as reducing pixels rastered did not seem to make any difference. The goal here is to reduce the maximum length of any one test so that fewer tests time out and these tests keep causing flakiness and getting disabled. This test breaks up each individual blend mode test into parameterizing by blend mode and rastering what used to be a single column in a larger image into its own image. Instead of saving out a million data pngs, instead this modifies the test framework to use expected pngs as well. This assumes that Skia's blend modes work properly and verifies that software and gl renderer match the output of those blend modes on an SkCanvas. In terms of total time of the test, splitting each test up into 16 different tests on Linux debug for me has each individual test completing in 1/4 of the time of the original (1/4x * 16 = 4x increase overall). However, when running the entire test suite in parallel, this patch speeds up running cc_unittests from ~70s to ~60s overall. So, I think this is overall a win, both in terms of individual and total test time. Bug: 870326 , 872733 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel Change-Id: I4791a0a2369f9f17af7bb2a6ba95d7f29dee8b42 Reviewed-on: https://chromium-review.googlesource.com/1180570 Reviewed-by: weiliangc <weiliangc@chromium.org> Commit-Queue: enne <enne@chromium.org> Cr-Commit-Position: refs/heads/master@{#584616} [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_resource_test.cc [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_resource_test.h [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_test.cc [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/layer_tree_pixel_test.h [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/pixel_test_utils.cc [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/test/pixel_test_utils.h [modify] https://crrev.com/d1f9ed886de3930f68a8b042fda843ca214eac96/cc/trees/layer_tree_host_pixeltest_blending.cc [delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_and_filter.png [delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_render_pass.png [delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_render_pass_mask.png [delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_transparent.png [delete] https://crrev.com/429c5dcef93750aae315c80cdbfaea74d43c965c/components/viz/test/data/blending_with_root.png |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by vitaliii@chromium.org
, Aug 9