Intermittent v8 assertions maps_pixel_tests on debug GPU bots |
|||||||||||||||
Issue descriptionFailures have been going on intermittently since the GPU bots cleared up https://build.chromium.org/p/chromium.gpu/builders/Mac%20Retina%20Debug https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29 (also has presumably unrelated WebGL failures) https://build.chromium.org/p/chromium.gpu/waterfall?builder=Linux%20Debug%20(NVIDIA) Example failure: # # Fatal error in ../../v8/src/heap/incremental-marking.cc, line 53 # Check failed: !Marking::IsImpossible(value_bit). # ==== C stack trace =============================== 1: V8_Fatal 2: v8::internal::IncrementalMarking::BaseRecordWrite(v8::internal::HeapObject*, v8::internal::Object*) 3: v8::internal::IncrementalMarking::RecordWriteSlow(v8::internal::HeapObject*, v8::internal::Object**, v8::internal::Object*) 4: v8::internal::JSObject::set_elements(v8::internal::FixedArrayBase*, v8::internal::WriteBarrierMode) 5: v8::internal::(anonymous namespace)::FastSmiOrObjectElementsAccessor<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::MoveElements(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSArray>, v8::internal::Handle<v8::internal::FixedArrayBase>, int, int, int, int, int) 6: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::Splice(v8::internal::Handle<v8::internal::JSArray>, unsigned int, unsigned int, v8::internal::Arguments*, unsigned int) 7: v8::internal::Builtin_Impl_ArraySplice(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)0>, v8::internal::Isolate*) 8: v8::internal::Builtin_ArraySplice(int, v8::internal::Object**, v8::internal::Isolate*) 9: 0x3461a2f06187 10: 0x3461a338d082 11: 0x3461a332c5b0 12: 0x3461a34464fa 13: 0x3461a332c315 Assigning to v8 sheriffs.
,
Mar 15 2016
Issue 595101 has been merged into this issue.
,
Mar 15 2016
,
Mar 15 2016
Please see the related bugs for more details on other crashes, all similar. Once the bug is fixed (if there was a V8 roll, it should be reverted until the bug's fixed), https://codereview.chromium.org/1800103002/ should be undone.
,
Mar 15 2016
Issue 594980 has been merged into this issue.
,
Mar 15 2016
I should point out that since this affects the tryservers, the top priority is to get them back to a stable state. If that means undoing a recent V8 roll or reverting more Blink patches, that is the correct thing to do. Assigning to one of the Chromium sheriffs and raising to P0. This must be investigated immediately.
,
Mar 15 2016
#6: I appreciate your ringing the bell on this. Now that crbug.com/594974 is under control I am focusing on this.
,
Mar 15 2016
My best guess is that the revert in 594974 should have cleared this up. I'm still keeping an eye on it. Since it looks like a few v8 folks are already cc'd here, any information on what that DCHECK might signify would be helpful for me if the problem is not in fact resolved.
,
Mar 15 2016
I wouldn't be so optimistic about revert from 594974. It has a completely different stack, and all other failures had inspector call there. This seems to be more of an internal v8 issue.
,
Mar 15 2016
Agreed. Most probably what is necessary is to look at the log for src/DEPS for the last day or so and start reverting V8 rolls.
,
Mar 15 2016
I think https://chromium.googlesource.com/v8/v8/+/3617612f8ba0a3123f813ac95070fdc61579caa0 and https://chromium.googlesource.com/v8/v8/+/0eef12e02cb275aa2b4f53242d3f7f3ce632cf69 may be the problem (judging by stack trace). Got rolled in https://codereview.chromium.org/1799403002/. @adamk: do you think these may be related to the crash?
,
Mar 15 2016
Seems very plausible. Please follow the instructions in the CL description of https://codereview.chromium.org/1799403002/ ; stop V8 rolls, and revert the V8 roll.
,
Mar 15 2016
OK, I'm going to revert DEPS to 'v8_revision': '2b8fdcf7910bee068f4735015e8e122dc8e56abd'
,
Mar 15 2016
v8 rolls are closed
,
Mar 15 2016
,
Mar 16 2016
Very good. Thank you! Let's downgrade to P2 now that the revert has landed, assuming it will clear up all of the flakiness. Please raise the priority again if this is not the case.
,
Mar 16 2016
,
Mar 16 2016
,
Mar 16 2016
Hannes PTAL. We probably need to disable black allocation again to get a roll through soon.
,
Mar 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/2d9c29cc463374d09184007a780f30ac9aca24d9 commit 2d9c29cc463374d09184007a780f30ac9aca24d9 Author: Hannes Payer <hpayer@chromium.org> Date: Wed Mar 16 07:40:18 2016 Disable black allocation. BUG= chromium:595092 LOG=n R=hablich@chromium.org Review URL: https://codereview.chromium.org/1803313002 . Cr-Commit-Position: refs/heads/master@{#34805} [modify] https://crrev.com/2d9c29cc463374d09184007a780f30ac9aca24d9/src/flag-definitions.h
,
Mar 16 2016
,
Mar 17 2016
Looks like this is happening again, after hpayer re-landed the flag flip in https://codereview.chromium.org/1809983002. Looking at: https://build.chromium.org/p/chromium.gpu/builders/Mac%20Retina%20Debug?numbuilds=100 Those builds are all green until two runs past the v8 roll: https://build.chromium.org/p/chromium.gpu/builders/Mac%20Retina%20Debug/builds/46008 Since then, the test has failed 4 of 19 times. https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29?numbuilds=100 also shows a few recent failures. It's no longer failing on the Linux bot, but I ran into a failure on a windows trybot: https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/190661 I'm inclined to revert the flag flip again.
,
Mar 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/434d660102f2b1154cbeeba955a8bc333aa24710 commit 434d660102f2b1154cbeeba955a8bc333aa24710 Author: adamk <adamk@chromium.org> Date: Thu Mar 17 23:38:51 2016 Revert of [heap] Enable black allocation. (patchset #1 id:1 of https://codereview.chromium.org/1809983002/ ) Reason for revert: Continues to cause flaky GPU test failures on Chromium waterfall. See details at http://crbug.com/595092#c22 Original issue's description: > [heap] Enable black allocation. > > BUG= > > Committed: https://crrev.com/447b1156d3bb4aa693175b74780104329ccd41ea > Cr-Commit-Position: refs/heads/master@{#34847} TBR=mlippautz@chromium.org,hpayer@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= chromium:595092 Review URL: https://codereview.chromium.org/1807393002 Cr-Commit-Position: refs/heads/master@{#34877} [modify] https://crrev.com/434d660102f2b1154cbeeba955a8bc333aa24710/src/flag-definitions.h
,
Mar 18 2016
Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
Mar 19 2016
Detected 7 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
Mar 21 2016
No more flakes the past couple of days, assuming the immediate flake problem is fixed.
,
Mar 21 2016
Detected 4 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
Mar 21 2016
These three failures: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/199661 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/199660 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/199628 look like a reoccurrence of Issue 586183 , where Image objects were being GC'd too early. Filed Issue about this. https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/191984 is an assertion failure related to hardware accelerated video decoding: [ RUN ] WebglConformance.conformance_textures_video_tex_image_and_sub_image_2d_with_video_rgba_rgba_unsigned_byte [4636:3428:0321/060355:FATAL:gl_fence_nv.cc(52)] Check failed: glIsFenceNV(fence_). Backtrace: GetHandleVerifier [0x65664997+181575] .... There's only one instance of this (looks like Issue 548383 ) so I'm not filing it for the moment. The maps_pixel_tests assertion failures are fixed at this point so I'm closing this as WontFix so new bugs get filed about any new flakiness. |
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by kbr@chromium.org
, Mar 15 2016