False rejects caused by pixel_tests on macOS |
|||||
Issue descriptionCL: https://chromium-review.googlesource.com/c/chromium/src/+/1382670/3 Failing build: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212638 Build succeeds on retry: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212654 Failing task: https://chromium-swarm.appspot.com/task?id=41e391142db38810&refresh=10&show_raw=1 There appears to have been several reference image mismatches, and later there was a crash. """ 9238:775:1220/003743.727815:INFO:CONSOLE(14)] "undefined", source: http://127.0.0.1:51970/content/test/data/gpu/pixel_webgl_sad_canvas.html (14) Received signal 11 SEGV_MAPERR 000000000000 DevTools console [network]: Failed to load resource: the server responded with a status of 404 (File not found) http://127.0.0.1:51970/favicon.ico 0 Chromium Framework 0x00000001233c7f5f base::debug::StackTrace::StackTrace(unsigned long) + 31 1 Chromium Framework 0x00000001233c7db1 base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 2385 2 libsystem_platform.dylib 0x00007fff4fd70f5a _sigtramp + 26 3 ??? 0x00007fbbf0809208 0x0 + 140445170569736 4 Chromium Framework 0x000000012545853c gpu::GpuChannel::OnControlMessageReceived(IPC::Message const&) + 236 5 Chromium Framework 0x000000012545986c gpu::GpuChannel::HandleMessageHelper(IPC::Message const&) + 140 6 Chromium Framework 0x000000012545a970 base::internal::Invoker<base::internal::BindState<void (gpu::GpuChannel::*)(IPC::Message const&), base::WeakPtr<gpu::GpuChannel>, IPC::Message>, void ()>::RunOnce(base::internal::BindStateBase*) + 192 7 Chromium Framework 0x00000001232adee1 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) + 305 8 Chromium Framework 0x000000012335d0fa base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::TimeTicks*) + 650 9 Chromium Framework 0x00000001232e1bf3 base::MessagePumpCFRunLoopBase::RunWork() + 51 10 Chromium Framework 0x00000001232c76da base::mac::CallWithEHFrame(void () block_pointer) + 10 11 Chromium Framework 0x00000001232e13cf base::MessagePumpCFRunLoopBase::RunWorkSource(void*) + 63 12 CoreFoundation 0x00007fff27c43a11 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17 13 CoreFoundation 0x00007fff27cfd42c __CFRunLoopDoSource0 + 108 14 CoreFoundation 0x00007fff27c26470 __CFRunLoopDoSources0 + 208 15 CoreFoundation 0x00007fff27c258ed __CFRunLoopRun + 1293 16 CoreFoundation 0x00007fff27c25153 CFRunLoopRunSpecific + 483 17 Foundation 0x00007fff29d21f26 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 277 18 Chromium Framework 0x00000001232e2461 base::MessagePumpNSRunLoop::DoRun(base::MessagePump::Delegate*) + 113 19 Chromium Framework 0x00000001232e0d3f base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*) + 127 20 Chromium Framework 0x000000012335d86b base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool) + 155 21 Chromium Framework 0x000000012331a1b9 base::RunLoop::Run() + 249 22 Chromium Framework 0x00000001294f3374 content::GpuMain(content::MainFunctionParams const&) + 1412 23 Chromium Framework 0x0000000122c90503 content::ContentMainRunnerImpl::Run(bool) + 435 24 Chromium Framework 0x0000000126291a55 service_manager::Main(service_manager::MainParams const&) + 3173 25 Chromium Framework 0x0000000122c8f614 content::ContentMain(content::ContentMainParams const&) + 68 26 Chromium Framework 0x000000011ec2715f ChromeMain + 175 27 Chromium Helper 0x000000010efd272e main + 494 28 libdyld.dylib 0x00007fff4fa62015 start + 1 [end of stack trace] [9238:775:1220/003743.930106:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://127.0.0.1:51970/content/test/data/gpu/pixel_webgl_sad_canvas.html (0) """ This test was run on 12/20 3:30AM EST, which was after this revert landed [from a previous problem]: https://chromium-review.googlesource.com/c/chromium/src/+/1283010
,
Dec 20
The crash is deliberate - caused by the Pixel_WebGLSadCanvas test, which crashes the GPU process and ensures that the "sad canvas" is drawn. The other pixel test failures are real, and mysterious - looking at the image diffs (from this log line): "See http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?f181416280cf39f3e7782dee044d78adf89b3dfc_mac_chromium_rel_ng_telemetry for this run's test results" the failing tests: Pixel_OffscreenCanvasTransferToImageBitmap Pixel_WebGLGreenTriangle_AA_Alpha show ~8 little red dots in the image diffs. I might chalk this up to broken hardware but need to investigate more. There are some other recent failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212890 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212889 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212845 Need to look into these to see if there's a pattern.
,
Dec 21
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212845 https://chromium-swarm.appspot.com/task?id=41e524915c23e310&refresh=10&show_raw=1 Pixel_WebGLGreenTriangle_NonChromiumImage_AA_Alpha failed. In the archived pixel results: http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?7eae7e7cbdc73a802048566702531860feb60635_mac_chromium_rel_ng_telemetry there's a small cluster of blue pixels off to the side of the triangle. From the failure above: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212638 https://chromium-swarm.appspot.com/task?id=41e391142db38810&refresh=10&show_raw=1 ran on build439-m4: https://chromium-swarm.appspot.com/bot?id=build439-m4&sort_stats=total%3Adesc It's failing all of its pixel_test runs. https://chromium-swarm.appspot.com/task?id=41e524915c23e310&refresh=10&show_raw=1 ran on build397-m4: https://chromium-swarm.appspot.com/bot?id=build397-m4&sort_stats=total%3Adesc It's similarly failing all pixel_test runs. I've spot-checked adjacent bots and they seem to be running pixel_tests cleanly, so have taken these two bots out of the Swarming pool.
,
Dec 21
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212890 https://chromium-swarm.appspot.com/task?id=41e553969d041e10&refresh=10&show_raw=1 Pixel_OffscreenCanvasWebGLSoftwareCompositing failed http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?6c224c93f54e09a7f409de3ac96229b65ecc6705_mac_chromium_rel_ng_telemetry Similar failure mode, cluster of garbage pixels off to the side of the rendered image. Was bot 397-m4: https://chromium-swarm.appspot.com/bot?id=build397-m4&sort_stats=total%3Adesc https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/212889 https://chromium-swarm.appspot.com/task?id=41e550c383733910&refresh=10&show_raw=1 Pixel_OffscreenCanvasTransferToImageBitmap ; similar failure mode. http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?fb9e65b1547f9135e2dee472b5f8c02d5ae8b248_mac_chromium_rel_ng_telemetry Happened on: https://chromium-swarm.appspot.com/bot?id=build404-m4&sort_stats=total%3Adesc That bot's run pixel_tests successfully since, but there's also another failure: https://chromium-swarm.appspot.com/task?id=41e6b7acc9201310&refresh=10&show_raw=1 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/213268 Pixel_OffscreenCanvasWebGLDefaultWorker failed http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?370bd1fbfe08db1498ba360cbc4650df90d8dbfe_mac_chromium_rel_ng_telemetry Same failure mode, garbage pixels off to the side. Have taken this bot out of the Swarming pool too.
,
Dec 21
Found additional failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/213262 Pixel_OffscreenCanvasTransferToImageBitmap failed on build433-m4: https://chromium-swarm.appspot.com/task?id=41e6b7afefb43d10&refresh=10&show_raw=1 https://chromium-swarm.appspot.com/bot?id=build433-m4&sort_stats=total%3Adesc http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?b6acd4574e64fc9acb473d178527847178e318cb_mac_chromium_rel_ng_telemetry Took out of the Swarming pool. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/213234 https://chromium-swarm.appspot.com/task?id=41e69cea9aecb810&refresh=10&show_raw=1 Pixel_OffscreenCanvasAccelerated2DWorker Pixel_OffscreenCanvasTransferBeforeStyleResize Pixel_OffscreenCanvasWebglResizeOnWorker https://chromium-swarm.appspot.com/task?id=41e69cea9aecb810&refresh=10&show_raw=1 http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?ffa2a7a745891c34d2cab726430d20002ce59b51_mac_chromium_rel_ng_telemetry Was build439-m4 again. So far, have taken these bots out of the Swarming pool: https://chromium-swarm.appspot.com/bot?id=build397-m4&sort_stats=total%3Adesc https://chromium-swarm.appspot.com/bot?id=build404-m4&sort_stats=total%3Adesc https://chromium-swarm.appspot.com/bot?id=build433-m4&sort_stats=total%3Adesc https://chromium-swarm.appspot.com/bot?id=build439-m4&sort_stats=total%3Adesc Will see whether things stabilize or if it looks like there are bugs in Chrome that are causing these sorts of failures.
,
Dec 21
,
Dec 21
,
Jan 8
,
Jan 8
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by kbr@chromium.org
, Dec 20