New issue
Advanced search Search tips

Issue 907689 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner: ----
Closed: Nov 26
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug



Sign in to add a comment

layout test shards timing out on mac_chromium_rel_ng

Project Member Reported by jbudorick@chromium.org, Nov 22

Issue description

Many layout test shards are timing out on the mac CQ bot. The CQ bot is significantly backlogged as a result.

This isn't happening on the CI bot, leading me to believe that it may be a DCHECK failure. Seems to have started around 12:30pm PST, with the initial wave on picking up chromium/src somewhere around https://chromium.googlesource.com/chromium/src/+log/9da9604039b057b10c25909ebea068f0d8116221
 
trooper incident: http://shortn/_p5rscgzyBE
Issue 907672 has been merged into this issue.
https://chromium-review.googlesource.com/c/chromium/src/+/1347692 appears to have been responsible for the layout test failures on CI and was reverted; I'm wondering if it was also responsible for this. The timing isn't quite right, though...
Components: Blink>Layout
+Blink>Layout as this appears to be related to a LayoutNG code path? I'm not sure how or why we're hitting that here, as it looks like LayoutNG is disabled by default and we don't set enable-layout-ng...
Cc: kojii@chromium.org e...@chromium.org
Chromium sheriff here.

Let me cc to eae@chromium.org, kojii@chromium.org
If we can't identify a culprit, it's probably worth temporarily removing the suite from the bot.
Cc: cbiesin...@chromium.org
Which tests are we talking about? Does anyone have a URL of failing build log?

"layout tests" I assume "webkit_layout_tests" but it shouldn't hit any LayoutNG code path except linux_layout_tests_layout_ng. If it's hitting, we need to investigate but again, any failing build log would help.

If "layout tests" mean "webkit_unit_tests", this might be it.
https://chromium-review.googlesource.com/c/chromium/src/+/1344950
we enabled knowing some tests still fail, as enabling helps us to point out regressions. If it's troubling CQ, we'll need a different approach.
layout tests => webkit_layout_tests.

There are a *ton* of infra-failed builds on mac_chromium_rel_ng, e.g. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/191227. Looking at the timed-out shards (not the "expired, not enough capacity" shards) shows a bunch of DCHECK failures, many with stacks like

20:01:01.106 29464   [31093:775:1121/200100.107542:FATAL:ng_paint_fragment.cc(564)] Check failed: container. 
20:01:01.106 29464   0   Content Shell Framework             0x000000011cf3702f base::debug::StackTrace::StackTrace(unsigned long) + 31
20:01:01.106 29464   1   Content Shell Framework             0x000000011ce37b3f logging::LogMessage::~LogMessage() + 223
20:01:01.106 29464   2   Content Shell Framework             0x00000001206902dd blink::NGPaintFragment::FlippedLocalVisualRectFor(blink::LayoutObject const*, blink::LayoutRect*) + 589
20:01:01.106 29464   3   Content Shell Framework             0x00000001203e0ef2 blink::LayoutText::LocalVisualRectIgnoringVisibility() const + 82

and 

20:01:07.773 29466   [31101:775:1121/200106.487394:FATAL:layout_box.cc(5727)] Check failed: !IsLayoutBlock(). 
20:01:07.773 29466   0   Content Shell Framework             0x000000011354502f base::debug::StackTrace::StackTrace(unsigned long) + 31
20:01:07.773 29466   1   Content Shell Framework             0x0000000113445b3f logging::LogMessage::~LogMessage() + 223
20:01:07.773 29466   2   Content Shell Framework             0x000000011692b345 blink::LayoutBox::OffsetFromLogicalTopOfFirstPage() const + 149
20:01:07.773 29466   3   Content Shell Framework             0x0000000116928383 blink::LayoutBox::PageRemainingLogicalHeightForOffset(blink::LayoutUnit, blink::LayoutBox::PageBoundaryRule) const + 179
20:01:07.773 29466   4   Content Shell Framework             0x00000001169281bc blink::LayoutBox::UpdateFragmentationInfoForChild(blink::LayoutBox&) + 252
20:01:07.773 29466   5   Content Shell Framework             0x00000001168da90c blink::LayoutBlock::LayoutPositionedObject(blink::LayoutBox*, bool, blink::LayoutBlock::PositionedLayoutBehavior) + 1196
20:01:07.773 29466   6   Content Shell Framework             0x00000001168da441 blink::LayoutBlock::LayoutPositionedObjects(bool, blink::LayoutBlock::PositionedLayoutBehavior) + 113
20:01:07.773 29466   7   Content Shell Framework             0x0000000116aad0a7 blink::NGBlockNode::CopyFragmentDataToLayoutBox(blink::NGConstraintSpace const&, blink::NGLayoutResult const&) + 2823
20:01:07.774 29466   8   Content Shell Framework             0x0000000116aac309 blink::NGBlockNode::FinishLayout(blink::LayoutBlockFlow*, blink::NGConstraintSpace const&, blink::NGBreakToken const*, scoped_refptr<blink::NGLayoutResult>) + 1161
20:01:07.774 29466   9   Content Shell Framework             0x0000000116aaa688 blink::NGBlockNode::Layout(blink::NGConstraintSpace const&, blink::NGBreakToken const*) + 616
20:01:07.774 29466   10  Content Shell Framework             0x0000000116abfae4 blink::NGLayoutInputNode::Layout(blink::NGConstraintSpace const&, blink::NGBreakToken const*, blink::NGInlineChildLayoutContext*) + 148
20:01:07.774 29466   11  Content Shell Framework             0x0000000116aa3084 blink::NGBlockLayoutAlgorithm::HandleInflow(blink::NGLayoutInputNode, blink::NGBreakToken const*, blink::NGPreviousInflowPosition*, scoped_refptr<blink::NGBreakToken const>*) + 1156
20:01:07.774 29466   12  Content Shell Framework             0x0000000116aa016a blink::NGBlockLayoutAlgorithm::Layout() + 1834

and more. It's possible that I'm mixing up paint-ng and layout-ng, though, as I'm not too familiar w/ blink.
Cc: mstensho@chromium.org
Thank you, I see, layout tests in "virtual/layout_ng_experimental/" do run with LayoutNG enabled, I forgot that. We're currently focusing on close-to-ship version and not much on this experimental virtual directly.

mstensho@, WDYT? It's probably fine to change [ Crash ] to [ Skip ] for "virtual/layout_ng_experimental/"?
You mean, skipping all tests currently expected to crash or time out (but running the others like before)? Sounds good to me.
Cc: andruud@chromium.org
Status: Available (was: Untriaged)
That said, if I look at the DCHECK failures in #c9, there seems to be 101 DCHECK failures. That's not a lot, is it (although I do realize it slows things down somewhat)? We could skip the ones crashing in layout_ng_experimental/, but that's just about 40 tests or so, so it wouldn't make much of a difference.

I'm just saying that I don't understand how that would cause the problem (nothing new about these crashes either).
Labels: -Sheriff-Chromium
That mac builder have no infra failures for a long time, and looks pretty green atm. Removes Sheriff label.

Status: Fixed (was: Available)
Mac triage: this is Fixed now, right?

Sign in to add a comment