webkit_tests failing >100 tests and aborting on linux_chromium_rel_ng |
||||
Issue descriptionhttps://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng?numbuilds=200 Many tryjobs are failing on linux_chromium_rel_ng where the first 100 layout tests appear to fail to launch content_shell. This behavior seems to have started today. It is badly affecting the Chromium commit queue. Here are jobs which failed: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338184 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338174 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338160 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338154 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338139 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338134 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338131 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338126 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338118 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338113 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338112 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338110 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338107 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338097 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338095 There were some flakes before, but not as bad: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338058 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338056 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/338045 Here's the list of tests from one of the failing runs. virtual/display_list_2d_canvas/fast/canvas/imagebitmap/transferFromImageBitmap-alpha.html virtual/display_list_2d_canvas/fast/canvas/2d.text.draw.fill.maxWidth.veryLarge.html http/tests/appcache/abort-cache-ondownloading-resource-404.html http/tests/pointer-lock/iframe-sandboxed-nested-disallow-then-allow-pointer-lock.html http/tests/pointer-lock/iframe-sandboxed.html virtual/display_list_2d_canvas/fast/canvas/imagebitmap/bitmaprenderer-as-imagesource.html virtual/gpu/fast/canvas/2d.fillText.gradient.html virtual/display_list_2d_canvas/fast/canvas/2d.text.draw.fill.maxWidth.verySmall.html virtual/display_list_2d_canvas/fast/canvas/imagebitmap/transferFromImageBitmap.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-imageSmoothing.html virtual/display_list_2d_canvas/fast/canvas/2d.fillText.gradient.html http/tests/appcache/abort-cache-onchecking-resource-404.html virtual/gpu/fast/canvas/webgl/canvas-to-data-url.html virtual/display_list_2d_canvas/fast/canvas/webgl/offscreenCanvas-transferToImageBitmap-texImage2D.html http/tests/appcache/abort-cache-onchecking-manifest-404.html virtual/android/fullscreen/full-screen-cancel-nested.html virtual/gpu/fast/canvas/OffscreenCanvas-2d-gradients-in-worker.html virtual/gpu/fast/canvas/2d.composite.globalAlpha.fillPath.html virtual/display_list_2d_canvas/fast/canvas/webgl/offscreenCanvas-APIs-NOT-alter-webgl-states.html virtual/android/fullscreen/full-screen-child-not-allowed-crash.html virtual/gpu/fast/canvas/2d.text.draw.fill.maxWidth.negative.html http/tests/accessibility/slow-document-load.html virtual/display_list_2d_canvas/fast/canvas/webgl/context-gc-custom-properties.html virtual/display_list_2d_canvas/fast/canvas/imagebitmap/transferFromImageBitmap-nullability.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-drawImage.html http/tests/pointer-lock/iframe-sandboxed-allow-pointer-lock.html http/tests/appcache/404-resource.html virtual/gpu/fast/canvas/webgl/canvas-getContext-crash.html virtual/display_list_2d_canvas/fast/canvas/2d.text.draw.fill.maxWidth.gradient.html virtual/android/fullscreen/enter-exit-full-screen-hover.html virtual/android/fullscreen/anonymous-block-merge-crash.html virtual/display_list_2d_canvas/fast/canvas/webgl/context-destroyed-crash.html virtual/android/fullscreen/full-screen-ancestor-shadow.html virtual/android/fullscreen/compositor-touch-hit-rects-fullscreen-video-controls.html http/tests/appcache/abort-cache-ondownloading-manifest-404.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-pattern.html virtual/gpu/fast/canvas/imagebitmap/transferFromImageBitmap-drawImage.html virtual/gpu/fast/canvas/CanvasRendering2D-prototype-chain.html virtual/display_list_2d_canvas/fast/canvas/webgl/canvas-getContext-crash.html virtual/display_list_2d_canvas/fast/canvas/webgl/offscreenCanvas-transferToImageBitmap-invalid-mailbox.html virtual/android/fullscreen/model/fully-exit-fullscreen-single.html virtual/gpu-rasterization/images/12-55.html virtual/display_list_2d_canvas/fast/canvas/2d.composite.globalAlpha.fillPath.html virtual/display_list_2d_canvas/fast/canvas/CanvasRendering2D-prototype-chain.html virtual/gpu/fast/canvas/imagebitmap/transferFromImageBitmap-nullability.html virtual/gpu-rasterization/images/alt-text-wrapping.html http/tests/appcache/404-resource-cross-origin.html virtual/android/media/mediadocument/media-document-with-download-button.html virtual/gpu/fast/canvas/webgl/canvas-resize-crash.html virtual/gpu-rasterization/images/2-comp.html virtual/display_list_2d_canvas/fast/canvas/webgl/compressed-tex-image.html virtual/android/fullscreen/empty-anonymous-block-continuation-crash.html virtual/gpu/fast/canvas/webgl/compressed-tex-image.html http/tests/activedomobject/media.html http/tests/preload/delaying_onload_link_preload_after_discovery.html virtual/display_list_2d_canvas/fast/canvas/imagebitmap/transferFromImageBitmap-no-alpha.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-pattern-in-worker.html virtual/gpu/fast/canvas/imagebitmap/transferFromImageBitmap-no-alpha.html virtual/android/fullscreen/exit-full-screen-iframe.html virtual/android/fullscreen/api/document-fullscreen-element.html virtual/gpu/fast/canvas/imagebitmap/bitmaprenderer-as-imagesource.html http/tests/appcache/abort-cache-ondownloading.html virtual/gpu/fast/canvas/2d.text.draw.fill.maxWidth.gradient.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-imageSmoothing-in-worker.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-gradients-in-worker.html http/tests/preload/delaying_onload_preloader_after_discovery.html virtual/display_list_2d_canvas/fast/canvas/webgl/draw-webgl-to-canvas-2d-after-to-data-url-without-context.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-imageData-in-worker.html http/tests/pointer-lock/iframe-sandboxed-nested-allow-pointer-lock.html virtual/gpu/fast/canvas/OffscreenCanvas-2d-filter.html virtual/android/fullscreen/model/fully-exit-fullscreen-nested.html virtual/gpu/fast/canvas/OffscreenCanvas-2d-drawImage-in-worker.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-drawImage-in-worker.html http/tests/preload/avoid_delaying_onload_link_preload.html virtual/display_list_2d_canvas/fast/canvas/2d.text.draw.fill.maxWidth.negative.html virtual/gpu-rasterization/images/182.html virtual/display_list_2d_canvas/fast/canvas/OffscreenCanvas-2d-filter.html virtual/gpu/fast/canvas/2d.text.draw.fill.maxWidth.veryLarge.html http/tests/pointer-lock/pointerlockelement-different-origin.html virtual/gpu/fast/canvas/2d.text.draw.fill.maxWidth.verySmall.html virtual/display_list_2d_canvas/fast/canvas/imagebitmap/transferFromImageBitmap-drawImage.html virtual/gpu/fast/canvas/imagebitmap/transferFromImageBitmap-alpha.html virtual/android/fullscreen/model/fully-exit-fullscreen-nested-iframe.html virtual/gpu-rasterization/images/png-suite/test.html http/tests/pointer-lock/requestPointerLock-can-not-transfer-between-documents.html virtual/gpu-rasterization/images/2-dht.html virtual/android/fullscreen/full-screen-contentEditable-crash.html virtual/gpu-rasterization/images/23-55.html virtual/display_list_2d_canvas/fast/canvas/webgl/draw-webgl-to-canvas-2d.html http/tests/appcache/404-manifest.html virtual/display_list_2d_canvas/fast/canvas/webgl/canvas-to-data-url.html virtual/android/fullscreen/full-screen-api.html http/tests/preload/download_resources.html virtual/display_list_2d_canvas/fast/canvas/webgl/canvas-resize-crash.html http/tests/pointer-lock/pointerlockelement-same-origin.html virtual/gpu/fast/canvas/imagebitmap/transferFromImageBitmap.html http/tests/appcache/abort-cache-onchecking.html virtual/gpu-rasterization/images/55.html virtual/android/fullscreen/full-screen-cancel.html virtual/gpu/fast/canvas/OffscreenCanvas-2d-drawImage.html Unfortunately there are no crash stacks in the logs. I suspect something was broken in sub-process launching. This needs to be investigated immediately. It's blocking jobs from landing via the CQ. Marking P0.
,
Nov 16 2016
Some plausible causes based on the blamelist: https://chromium.googlesource.com/chromium/src/+/a15698acc363af3e3b28543135e765019ef709a1 https://chromium.googlesource.com/chromium/src/+/3da850c3bfadcf3d83407bb4aa9b1e047cbd44a8 https://chromium.googlesource.com/chromium/src/+/72b077630315b35652e85865064748e0bbc0c43e https://chromium.googlesource.com/chromium/src/+/b902fd366ba91757dbee99d5f821f00d2de1d181
,
Nov 16 2016
I would think that https://chromium.googlesource.com/chromium/src/+/3da850c3bfadcf3d83407bb4aa9b1e047cbd44a8 can only affect android.
,
Nov 16 2016
timav@: I would think so too, but some of the failing tests are in the "virtual/android" test suite and it wasn't obvious to me how much of your code could possibly run in that configuration.
,
Nov 16 2016
One thing I'm worried about is that in this CL I set explicitly initialize some global pointer (the Screen) before the first use and I might have missed some places. I would think then again, only virtual/android tests will fail, but can you just revert and see whether it fixes? Are there any logs? I could not figure out the actual reason.
,
Nov 16 2016
http/tests/accessibility/slow-document-load.html started crashing after #432306. So I'm reverting https://codereview.chromium.org/2500063003/.
,
Nov 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/233d31c7c642db349b8d127d333c9672acc52eca commit 233d31c7c642db349b8d127d333c9672acc52eca Author: horo <horo@chromium.org> Date: Wed Nov 16 03:43:00 2016 Revert of Replace kochi-*.ttf with IPA Fonts. (patchset #3 id:40001 of https://codereview.chromium.org/2500063003/ ) Reason for revert: Looks like caused many failures in webkit_tests. BUG= 665693 Original issue's description: > Replace kochi-*.ttf with IPA Fonts. > > kochi-gothic.ttf and kochi-mincho.ttf were deprecated [1] > and dropped from Debian and Ubuntu [2]. > They were replaced with ipag.ttf and ipam.ttf (fonts-ipafont package). > > [1] https://packages.debian.org/wheezy/ttf-kochi-gothic > [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726382#33 > > BUG= 665515 > > Committed: https://crrev.com/a15698acc363af3e3b28543135e765019ef709a1 > Cr-Commit-Position: refs/heads/master@{#432306} TBR=thomasanderson@chromium.org,asvitkine@chromium.org,msw@chromium.org,derat@chromium.org,thomasanderson@google.com,thakis@chromium.org,jack@nottheoilrig.com # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 665515 Review-Url: https://codereview.chromium.org/2504863002 Cr-Commit-Position: refs/heads/master@{#432371} [modify] https://crrev.com/233d31c7c642db349b8d127d333c9672acc52eca/AUTHORS [modify] https://crrev.com/233d31c7c642db349b8d127d333c9672acc52eca/build/experimental/install-build-deps.py [modify] https://crrev.com/233d31c7c642db349b8d127d333c9672acc52eca/build/install-build-deps.sh [modify] https://crrev.com/233d31c7c642db349b8d127d333c9672acc52eca/docs/layout_tests_linux.md [modify] https://crrev.com/233d31c7c642db349b8d127d333c9672acc52eca/ui/gfx/test/fontconfig_util_linux.cc
,
Nov 16 2016
horo's revert fixed the problem. This build: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty/builds/19992 unfortunately failed one test, but the huge number of failures is gone. Thank you horo@ for the quick action!
,
Nov 21 2016
|
||||
►
Sign in to add a comment |
||||
Comment 1 by kbr@chromium.org
, Nov 16 2016