layout test shards timing out on mac_chromium_rel_ng |
|||||||||
Issue descriptionMany layout test shards are timing out on the mac CQ bot. The CQ bot is significantly backlogged as a result. This isn't happening on the CI bot, leading me to believe that it may be a DCHECK failure. Seems to have started around 12:30pm PST, with the initial wave on picking up chromium/src somewhere around https://chromium.googlesource.com/chromium/src/+log/9da9604039b057b10c25909ebea068f0d8116221
,
Nov 22
Issue 907672 has been merged into this issue.
,
Nov 22
,
Nov 22
https://chromium-review.googlesource.com/c/chromium/src/+/1347692 appears to have been responsible for the layout test failures on CI and was reverted; I'm wondering if it was also responsible for this. The timing isn't quite right, though...
,
Nov 22
+Blink>Layout as this appears to be related to a LayoutNG code path? I'm not sure how or why we're hitting that here, as it looks like LayoutNG is disabled by default and we don't set enable-layout-ng...
,
Nov 22
Chromium sheriff here. Let me cc to eae@chromium.org, kojii@chromium.org
,
Nov 22
If we can't identify a culprit, it's probably worth temporarily removing the suite from the bot.
,
Nov 22
Which tests are we talking about? Does anyone have a URL of failing build log? "layout tests" I assume "webkit_layout_tests" but it shouldn't hit any LayoutNG code path except linux_layout_tests_layout_ng. If it's hitting, we need to investigate but again, any failing build log would help. If "layout tests" mean "webkit_unit_tests", this might be it. https://chromium-review.googlesource.com/c/chromium/src/+/1344950 we enabled knowing some tests still fail, as enabling helps us to point out regressions. If it's troubling CQ, we'll need a different approach.
,
Nov 22
layout tests => webkit_layout_tests. There are a *ton* of infra-failed builds on mac_chromium_rel_ng, e.g. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/191227. Looking at the timed-out shards (not the "expired, not enough capacity" shards) shows a bunch of DCHECK failures, many with stacks like 20:01:01.106 29464 [31093:775:1121/200100.107542:FATAL:ng_paint_fragment.cc(564)] Check failed: container. 20:01:01.106 29464 0 Content Shell Framework 0x000000011cf3702f base::debug::StackTrace::StackTrace(unsigned long) + 31 20:01:01.106 29464 1 Content Shell Framework 0x000000011ce37b3f logging::LogMessage::~LogMessage() + 223 20:01:01.106 29464 2 Content Shell Framework 0x00000001206902dd blink::NGPaintFragment::FlippedLocalVisualRectFor(blink::LayoutObject const*, blink::LayoutRect*) + 589 20:01:01.106 29464 3 Content Shell Framework 0x00000001203e0ef2 blink::LayoutText::LocalVisualRectIgnoringVisibility() const + 82 and 20:01:07.773 29466 [31101:775:1121/200106.487394:FATAL:layout_box.cc(5727)] Check failed: !IsLayoutBlock(). 20:01:07.773 29466 0 Content Shell Framework 0x000000011354502f base::debug::StackTrace::StackTrace(unsigned long) + 31 20:01:07.773 29466 1 Content Shell Framework 0x0000000113445b3f logging::LogMessage::~LogMessage() + 223 20:01:07.773 29466 2 Content Shell Framework 0x000000011692b345 blink::LayoutBox::OffsetFromLogicalTopOfFirstPage() const + 149 20:01:07.773 29466 3 Content Shell Framework 0x0000000116928383 blink::LayoutBox::PageRemainingLogicalHeightForOffset(blink::LayoutUnit, blink::LayoutBox::PageBoundaryRule) const + 179 20:01:07.773 29466 4 Content Shell Framework 0x00000001169281bc blink::LayoutBox::UpdateFragmentationInfoForChild(blink::LayoutBox&) + 252 20:01:07.773 29466 5 Content Shell Framework 0x00000001168da90c blink::LayoutBlock::LayoutPositionedObject(blink::LayoutBox*, bool, blink::LayoutBlock::PositionedLayoutBehavior) + 1196 20:01:07.773 29466 6 Content Shell Framework 0x00000001168da441 blink::LayoutBlock::LayoutPositionedObjects(bool, blink::LayoutBlock::PositionedLayoutBehavior) + 113 20:01:07.773 29466 7 Content Shell Framework 0x0000000116aad0a7 blink::NGBlockNode::CopyFragmentDataToLayoutBox(blink::NGConstraintSpace const&, blink::NGLayoutResult const&) + 2823 20:01:07.774 29466 8 Content Shell Framework 0x0000000116aac309 blink::NGBlockNode::FinishLayout(blink::LayoutBlockFlow*, blink::NGConstraintSpace const&, blink::NGBreakToken const*, scoped_refptr<blink::NGLayoutResult>) + 1161 20:01:07.774 29466 9 Content Shell Framework 0x0000000116aaa688 blink::NGBlockNode::Layout(blink::NGConstraintSpace const&, blink::NGBreakToken const*) + 616 20:01:07.774 29466 10 Content Shell Framework 0x0000000116abfae4 blink::NGLayoutInputNode::Layout(blink::NGConstraintSpace const&, blink::NGBreakToken const*, blink::NGInlineChildLayoutContext*) + 148 20:01:07.774 29466 11 Content Shell Framework 0x0000000116aa3084 blink::NGBlockLayoutAlgorithm::HandleInflow(blink::NGLayoutInputNode, blink::NGBreakToken const*, blink::NGPreviousInflowPosition*, scoped_refptr<blink::NGBreakToken const>*) + 1156 20:01:07.774 29466 12 Content Shell Framework 0x0000000116aa016a blink::NGBlockLayoutAlgorithm::Layout() + 1834 and more. It's possible that I'm mixing up paint-ng and layout-ng, though, as I'm not too familiar w/ blink.
,
Nov 22
Thank you, I see, layout tests in "virtual/layout_ng_experimental/" do run with LayoutNG enabled, I forgot that. We're currently focusing on close-to-ship version and not much on this experimental virtual directly. mstensho@, WDYT? It's probably fine to change [ Crash ] to [ Skip ] for "virtual/layout_ng_experimental/"?
,
Nov 22
You mean, skipping all tests currently expected to crash or time out (but running the others like before)? Sounds good to me.
,
Nov 22
,
Nov 22
That said, if I look at the DCHECK failures in #c9, there seems to be 101 DCHECK failures. That's not a lot, is it (although I do realize it slows things down somewhat)? We could skip the ones crashing in layout_ng_experimental/, but that's just about 40 tests or so, so it wouldn't make much of a difference. I'm just saying that I don't understand how that would cause the problem (nothing new about these crashes either).
,
Nov 23
That mac builder have no infra failures for a long time, and looks pretty green atm. Removes Sheriff label.
,
Nov 26
Mac triage: this is Fixed now, right? |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by jbudorick@chromium.org
, Nov 22