v8 check failure on Android Release Nexus 9 gpu.fyi bot |
|||||
Issue descriptionhttps://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%209%29/builds/6655 Don't have a symbolicated stack trace but the failure was logged. (kbr@ is symbolication supposed to work on this bot?) 05-23 17:03:28.770 28554 28818 E v8 : 05-23 17:03:28.770 28554 28818 E v8 : 05-23 17:03:28.770 28554 28818 E v8 : # 05-23 17:03:28.770 28554 28818 E v8 : # Fatal error in ../../v8/src/compiler/register-allocator-verifier.cc, line 41 05-23 17:03:28.770 28554 28818 E v8 : # 05-23 17:03:28.771 28554 28818 E v8 : Check failed: move->source().IsAllocated() || move->source().IsConstant(). 05-23 17:03:28.771 28554 28818 E v8 : 05-23 17:03:28.771 28554 28818 E v8 : # 05-23 17:03:28.784 28554 28818 E chromium: #00 0x0000007f8a2d24f7 /data/app/org.chromium.chrome-1/base.apk+0x00000000028e24f7 05-23 17:03:28.784 28554 28818 E chromium: #01 0x0000007f8aa9b4bf /data/app/org.chromium.chrome-1/base.apk+0x00000000030ab4bf 05-23 17:03:28.784 28554 28818 E chromium: #02 0x0000007f8a509baf /data/app/org.chromium.chrome-1/base.apk+0x0000000002b19baf 05-23 17:03:28.784 28554 28818 E chromium: #03 0x0000007f8a502873 /data/app/org.chromium.chrome-1/base.apk+0x0000000002b12873 05-23 17:03:28.784 28554 28818 E chromium: #04 0x0000007f8a5032bb /data/app/org.chromium.chrome-1/base.apk+0x0000000002b132bb 05-23 17:03:28.784 28554 28818 E chromium: #05 0x0000007f8a505a2b /data/app/org.chromium.chrome-1/base.apk+0x0000000002b15a2b 05-23 17:03:28.784 28554 28818 E chromium: #06 0x0000007f8a505d6f /data/app/org.chromium.chrome-1/base.apk+0x0000000002b15d6f 05-23 17:03:28.784 28554 28818 E chromium: #07 0x0000007f8a4054bb /data/app/org.chromium.chrome-1/base.apk+0x0000000002a154bb 05-23 17:03:28.784 28554 28818 E chromium: #08 0x0000007f8a403f03 /data/app/org.chromium.chrome-1/base.apk+0x0000000002a13f03 05-23 17:03:28.784 28554 28818 E chromium: #09 0x0000007f8a404ce3 /data/app/org.chromium.chrome-1/base.apk+0x0000000002a14ce3 05-23 17:03:28.784 28554 28818 E chromium: #10 0x0000007f88c95963 /data/app/org.chromium.chrome-1/base.apk+0x00000000012a5963 05-23 17:03:28.784 28554 28818 E chromium: #11 0x0000007f88d4c82f /data/app/org.chromium.chrome-1/base.apk+0x000000000135c82f 05-23 17:03:28.784 28554 28818 E chromium: #12 0x0000007f88d40943 /data/app/org.chromium.chrome-1/base.apk+0x0000000001350943 05-23 17:03:28.784 28554 28818 E chromium: #13 0x0000007facbb26d7 /system/lib64/libc.so+0x00000000000676d7 05-23 17:03:28.784 28554 28818 E chromium: #14 0x0000007facb675c7 /system/lib64/libc.so+0x000000000001c5c7 05-23 17:03:28.784 28554 28818 E chromium: #15 0xffffffffffffffff <unknown> --------- beginning of crash 05-23 17:03:28.784 28554 28818 F libc : Fatal signal 5 (SIGTRAP), code 1 in tid 28818 (WorkerPool/2881) 05-23 17:03:28.886 228 228 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** 05-23 17:03:28.886 228 228 F DEBUG : Build fingerprint: 'google/volantis/flounder:6.0.1/MOB31L/3383703:userdebug/dev-keys' 05-23 17:03:28.886 228 228 F DEBUG : Revision: '0' 05-23 17:03:28.886 228 228 F DEBUG : ABI: 'arm64' 05-23 17:03:28.886 228 228 F DEBUG : pid: 28554, tid: 28818, name: WorkerPool/2881 >>> org.chromium.chrome:sandboxed_process0 <<< 05-23 17:03:28.886 228 228 F DEBUG : signal 5 (SIGTRAP), code 1 (TRAP_BRKPT), fault addr 0x182ea00006f8a 05-23 17:03:28.898 228 228 F DEBUG : x0 0000000000000001 x1 0000007facc00ad0 x2 0000000000004001 x3 0000007f9337f450 05-23 17:03:28.899 228 228 F DEBUG : x4 0000000000000000 x5 0000000000004001 x6 0000000000000000 x7 0000007facc00ad4 05-23 17:03:28.899 228 228 F DEBUG : x8 0000000000007092 x9 0000000000007092 x10 0000007facc00ad4 x11 0000000000004000 05-23 17:03:28.899 228 228 F DEBUG : x12 0000000000004001 x13 0000000000000000 x14 0000000000000001 x15 0000000000000004 05-23 17:03:28.899 228 228 F DEBUG : x16 0000007facbf2a58 x17 0000000000000000 x18 2800000000000000 x19 0000007f8eddb000 05-23 17:03:28.899 228 228 F DEBUG : x20 0000007facbf53e8 x21 0000007f8d5fe730 x22 0000007f8d633d60 x23 0000000000000140 05-23 17:03:28.899 228 228 F DEBUG : x24 00000055768b7050 x25 000000557689f4e0 x26 0000000000000001 x27 00000055768b5c90 05-23 17:03:28.899 228 228 F DEBUG : x28 000000557688d258 x29 0000007f9337e4d0 x30 0000007f8aa9b4d0 05-23 17:03:28.900 228 228 F DEBUG : sp 0000007f9337e4d0 pc 0000007f8aa9d180 pstate 0000000080000000 05-23 17:03:28.905 228 228 F DEBUG : 05-23 17:03:28.905 228 228 F DEBUG : backtrace: 05-23 17:03:28.905 228 228 F DEBUG : #00 pc 0000000001f8f180 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.905 228 228 F DEBUG : #01 pc 0000000001f8d4cc /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.905 228 228 F DEBUG : #02 pc 00000000019fbbac /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #03 pc 00000000019f4870 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #04 pc 00000000019f52b8 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #05 pc 00000000019f7a28 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #06 pc 00000000019f7d6c /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #07 pc 00000000018f74b8 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #08 pc 00000000018f5f00 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #09 pc 00000000018f6ce0 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #10 pc 0000000000187960 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #11 pc 000000000023e82c /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.906 228 228 F DEBUG : #12 pc 0000000000232940 /data/app/org.chromium.chrome-1/base.apk (offset 0x111e000) 05-23 17:03:28.907 228 228 F DEBUG : #13 pc 00000000000676d4 /system/lib64/libc.so (_ZL15__pthread_startPv+52) 05-23 17:03:28.906 228 228 W debuggerd64: type=1400 audit(0.0:1338): avc: denied { search } for name="org.chromium.chrome" dev="dm-2" ino=2510951 scontext=u:r:debuggerd:s0 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0 05-23 17:03:28.907 228 228 F DEBUG : #14 pc 000000000001c5c4 /system/lib64/libc.so (__start_thread+16)
,
May 23 2017
Sorry, I'm not sure what the stack symbolization situation is on the (non-Swarmed) Android bots with Telemetry. jbudorick@ would know more. The situation is going to change as these bots move into the Swarming pool, too.
,
May 24 2017
+mtrofin@, please have a look at this CHECK failure in the register allocator.
,
May 24 2017
#2: nonswarmed bots symbolize stacks in the stack_tool_* steps. We have yet to work out how symbolization works in swarmed telemetry tests on android.
,
May 25 2017
,
May 31 2017
For a brief investigation report up to this moment: This problem appears to have been going on for a while, see for example https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%209%29/builds/6590 (May 20). It is flaky, and so far wasn't able to repro it at all on chorme x64 running an emulated arm64 v8 (which, as it appears to be a register allocation issue, should repro). Also wasn't able to consistently repro on a Nexus 9 (no real surprise, the bots also seem to have the same problem). Still investigating.
,
May 31 2017
Just for the record, in case the logs expire: From this run: https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Android%20Release%20%28Nexus%209%29/6590 The failing tests were (redundant information eliminated): WebglConformance_conformance_ogles_GL_exp_exp_009_to_012 WebglConformance_conformance_textures_misc_tex_video_using_tex_unit_non_zero WebglConformance_conformance_textures_misc_texture_formats_test WebglConformance_conformance_textures_misc_texture_hd_dpi WebglConformance_deqp_data_gles2_shaders_keywords WebglConformance_deqp_data_gles2_shaders_linkage WebglConformance_deqp_data_gles2_shaders_preprocessor WebglConformance_deqp_data_gles2_shaders_qualification_order WebglConformance_deqp_data_gles2_shaders_reserved_operators It might be the case that this can be reproduced with (not sure about the --browser arg): run_gpu_integration_test.py webgl_conformance --browser=android-chromium --test-filter=WebglConformance_deqp_data_gles2_shaders Just to make reproduction faster, since it sounds like from offline conversation with mtrofin@ that reproduction is difficult and the problem occurs only on this device. It sounds like a pretty deep bug, possibly in management of TLB entries if a thread switches from one core to the other.
,
May 31 2017
gn args: dcheck_always_on = true ffmpeg_branding = "Chrome" is_component_build = false is_debug = false proprietary_codecs = true strip_absolute_paths_from_debug_symbols = true symbol_level = 1 target_cpu = "arm64" target_os = "android" use_goma = true Run: (from output dir, e.g. out/arm64.release) ../../content/test/gpu/run_gpu_integration_test.py webgl_conformance --s\ how-stdout --browser=android-chromium --passthrough -v --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc"
,
Jun 1 2017
Oh, and the bots build these targets: angle_unittests gl_tests gl_unittests telemetry_gpu_integration_test
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/631aba17c5a5519302a5756a25bea9d2d2b67b99 commit 631aba17c5a5519302a5756a25bea9d2d2b67b99 Author: Mircea Trofin <mtrofin@chromium.org> Date: Thu Jun 01 22:24:53 2017 [turbofan] Move some register allocation validation earlier. This is in an effort to narrow down the root cause of the referenced, very flaky, bug. Bug: chromium:725559 Change-Id: I8235983fea6f579d228e81177fdcac9ef6115d1f Reviewed-on: https://chromium-review.googlesource.com/521923 Reviewed-by: Bill Budge <bbudge@chromium.org> Commit-Queue: Mircea Trofin <mtrofin@chromium.org> Cr-Commit-Position: refs/heads/master@{#45668} [modify] https://crrev.com/631aba17c5a5519302a5756a25bea9d2d2b67b99/src/compiler/pipeline.cc [modify] https://crrev.com/631aba17c5a5519302a5756a25bea9d2d2b67b99/src/compiler/register-allocator.cc
,
Jun 5 2017
For tracking: the above change made it in Chrome as part of the v8-autoroll @588f75c366e41cda310d4a2da20eee27d671c041.
,
Jun 5 2017
Since then, we had 2 failures. One is in GC: https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%209%29/builds/6841 06-03 18:51:02.032 15672 15686 E v8 : # Fatal error in ../../v8/src/heap/incremental-marking.cc, line 510 06-03 18:51:02.032 15672 15686 E v8 : # 06-03 18:51:02.032 15672 15686 E v8 : Check failed: heap_->gc_state() == Heap::NOT_IN_GC. The other one is in the verifier. Since the stack traces are illegible, we can't determine where the validator failed. Sending a patch to allow for that.
,
Jun 5 2017
FWIW, "heap_->gc_state() == Heap::NOT_IN_GC" is appearing in the GPU FYI as well for NVIDIA Shield TV: https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28NVIDIA%20Shield%20TV%29/builds/1492
,
Jun 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/e5e55c61c7b9c37444980fa6620a1a2c36f66656 commit e5e55c61c7b9c37444980fa6620a1a2c36f66656 Author: Mircea Trofin <mtrofin@chromium.org> Date: Mon Jun 05 19:36:48 2017 [turbofan] Flaky regalloc verifier investigation: capture caller. Capture the place we call the verifier as part of the investigation into the referenced bug. Bug: chromium:725559 Change-Id: I08fa91636f73994f8d77ac6ab66aa7165a12ef0b Reviewed-on: https://chromium-review.googlesource.com/524266 Commit-Queue: Brad Nelson <bradnelson@chromium.org> Reviewed-by: Brad Nelson <bradnelson@chromium.org> Cr-Commit-Position: refs/heads/master@{#45712} [modify] https://crrev.com/e5e55c61c7b9c37444980fa6620a1a2c36f66656/src/compiler/pipeline.cc [modify] https://crrev.com/e5e55c61c7b9c37444980fa6620a1a2c36f66656/src/compiler/register-allocator-verifier.cc [modify] https://crrev.com/e5e55c61c7b9c37444980fa6620a1a2c36f66656/src/compiler/register-allocator-verifier.h
,
Jul 14 2017
I reproduced an instance of this crash locally running webgl tests on Nexus9: https://bugs.chromium.org/p/chromium/issues/detail?id=740769#c12 Let me know if I can help with debugging.
,
Jul 14 2017
I was, too, but then it stopped reproing. Are you able to get a failure consistently?
,
Aug 7 2017
Looks like it doesn't repro on the bots either. Closing as no repro. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by sunn...@chromium.org
, May 23 2017