After a little trial I found making all localOverflowRectForPaintInvalidation() in physical coordinates is not easy. The problem is about non-box objects which returns localOverflowRectForPaintInvalidation() in containing block's coordinates. If we also need them to return physical coordinates, we'll need to access containingBlock() in the functions which will degrade performance.

I'll upload a simple patch to fix the issue first, then look into the root cause.

Sorry about that. We had originally explored LayoutView-specific change but it ended up looking like it also applied to LayoutBox when mapping to itself, I thought I had confirmed with another test but I see only one on that change. Maybe I had confirmed through scrutiny of the test noted here:

https://codereview.chromium.org/2318033003#msg4

as:

fast/repaint/window-resize-vertical-writing-mode.html

and now relocated at:

paint/invalidation/window-resize-vertical-writing-mode.html

so if that's still ok with your eventual rework then it should be fine.

Your planned work to change so that the input rect to slowMapToVisualRectInAncestorSpace() is in non-flipped-block space would be good, I think it is basically addressing the TODO you already partially referenced:

https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/layout/PaintInvalidationState.cpp?dr=CSs&q=paintinvalidationstate&sq=package:chromium&l=418
https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/paint/PaintInvalidator.cpp?sq=package:chromium&dr=CSs&rcl=1474903310&l=42

Some notes:
- It's now *in*consistent about the coordinates space of the input rect of mapToVisualRectInAncestorSpace()
  - For a LayoutBox, the input rect is in local physical coordinate space;
  - For a LayoutInline/LayoutText, the input rect is in flipped physical coordinate space of the containing block.
  - (SVG is not applicable).

This has been the status after https://codereview.chromium.org/2073563002. Before that, all implementations of the function accept rect in flipped local physical coordinate space (for LayoutInline/LayoutText, local coordinate space is the containing block's space).

- IMHO, the "flipped physical coordinate space" concept is mainly for performance. It avoids some flip/unflip operations to/from physical coordinate space which requires asking writing-mode of the writing-mode container which needs traverse the layout tree up.

- LayoutBox::topLeftLocation() has been used several performance-critical places (e.g. in LayoutBox::mapToVisualRectInAncestorSpace(), PaintPropertyTreeBuilder). It can avoid using "flipped physical coordinate space" and make code clearer but may affect performance. We can improve performance by caching the writing-mode container during tree walk.

Re your last bullet in comment 3: how do you propose to improve performance 
exactly? Flipping has to happen at every containing block up to to the writing
mode root.

Re #4 it's not flippings themselves, but LayoutBox::topLeftLocation() which search up the tree for the containing block affects performance. One way to improve it is that during a tree walk (e.g. PrePaintTreeWalk) we cache the current containing block for normal, absolute and fixed position objects and pass it to LayoutBox::topLeftLocation().

Ah, I see. Yes it could be useful to cache containing blocks.

Note that in LayoutBox::mapToVisualRectInAncestorSpace() all calls to topLeftLocation() except the one for Ruby (which is rare and performance matters less) pass the known containing block along, to avoid need to walk up to find it.

I think pdr@ had planned to do something similar in PaintPropertyTreeBuilder, perhaps to cache during walk as you mention.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/049d91617706a0fd7f96e861364e4340c966d855

commit 049d91617706a0fd7f96e861364e4340c966d855
Author: wangxianzhu <wangxianzhu@chromium.org>
Date: Tue Sep 27 22:28:19 2016

Fix paint invalidation for composited vertical-rl having overflow

Previously we special cased in slowMapToVisualRectInAncestorSpace()
in PaintInvalidationState.cpp for LayoutView background invalidation
which was because LayoutView::localOverflowRectForPaintInvalidation()
was in physical coordinates while other
localOverflowRectForPaintInvalidation() implementations are in flipped
physical coordinates.

This CL removes the special case, but let
LayoutView::localOverflowRectForPaintInvalidation() returns a rect in
flipped physical coordinates.

BUG= 650417 
TEST=paint/invalidation/composited-vertical-rl-overflow.html
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2

Review-Url: https://codereview.chromium.org/2371933003
Cr-Commit-Position: refs/heads/master@{#421362}

[add] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/LayoutTests/paint/invalidation/composited-vertical-rl-overflow-expected.html
[add] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/LayoutTests/paint/invalidation/composited-vertical-rl-overflow-expected.txt
[add] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/LayoutTests/paint/invalidation/composited-vertical-rl-overflow.html
[add] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/LayoutTests/paint/invalidation/vertical-rl-overflow-expected.html
[add] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/LayoutTests/paint/invalidation/vertical-rl-overflow-expected.txt
[add] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/LayoutTests/paint/invalidation/vertical-rl-overflow.html
[modify] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/Source/core/layout/LayoutView.cpp
[modify] https://crrev.com/049d91617706a0fd7f96e861364e4340c966d855/third_party/WebKit/Source/core/layout/PaintInvalidationState.cpp