rebaseline-cl created bizarre baseline changes due to platform differences in global-interface-listing-expected.txt |
||
Issue descriptionIn https://chromium-review.googlesource.com/c/chromium/src/+/1223149/1 (PS1) I updated TestExpectations and ran `./third_party/blink/tools/blink_tool.py rebaseline-cl` The bots which ran can be seen in that CL, 9 different ones including linux-blink-rel, win7-blink-rel, and others. After they finished, I ran `./third_party/blink/tools/blink_tool.py rebaseline-cl`. The exact output of that was never uploaded to review 1223149, but I uploaded it to a separate CL now to show it: https://chromium-review.googlesource.com/c/chromium/src/+/1224055 There are a number of added baselines among the many modified ones: platform/fuchsia/http/tests/worklet/webexposed/global-interface-listing-paint-worklet-expected.txt platform/mac-retina/webexposed/global-interface-listing-expected.txt platform/mac/webexposed/global-interface-listing-expected.txt platform/win/webexposed/global-interface-listing-expected.txt virtual/origin-trials-runtimeflags-disabled/http/tests/origin_trials/webexposed/animationworklet-origin-trial-interfaces-worklet-scope-expected.txt virtual/threaded/http/tests/worklet/webexposed/global-interface-listing-paint-worklet-expected.txt It looks like the cause of the changes is that the bots had different global-interface-listing-expected.txt. I downloaded the "actual" from each bot and named it after the bot. There were exactly two forms. linux-blink-rel.txt matches what I got locally but came from the bot. mac10_11-blink-rel, mac10_12-blink-rel and mac10_13-blink-rel had different results from the others: --- linux-blink-rel.txt 2018-09-13 11:46:50.993033176 +0200 +++ mac10_11-blink-rel.txt 2018-09-13 11:47:45.389376379 +0200 @@ -4092,6 +4092,7 @@ getter root getter rootMargin getter thresholds + getter trackVisibility method constructor method disconnect method observe @@ -4103,6 +4104,7 @@ getter intersectionRatio getter intersectionRect getter isIntersecting + getter isVisible getter rootBounds getter target getter time These two attributes have [RuntimeEnabled=IntersectionObserverV2] in the IDL, and I really don't understand how they could have been enabled on these bots. szager@, is it expected that this flag should be enabled on just some mac platforms? I can't find any code to suggest that. robertma@, do you know of any other reason why the results could be different between bots? My only hypothesis is that the bots tested different commits and got different behavior, but comparing the commits for two of the bots with different results I don't spot any changes that seems like plausible culprits. For now, logging this issue to see if anyone has ideas of what might have gone wrong.
,
Sep 13
Philip, I think your investigation is on the right path.
rebaseline-cl (or more specifically, optimize-baselines in this case) is WAI. The weird placement of baselines is due to mac10_11-blink-rel, mac10_12-blink-rel and mac10_13-blink-rel being different from others.
This is what the fallback tree looks like. I use "*" and "+" to represent the two different baselines, and "|" means this platform does not have a baseline (i.e. falls back to its parent).
* fuschia
* win
| linux
+ mac
* mac-retina
| mac10.12
| mac10.11
* mac10.10
,
Sep 14
Yeah, I also suspsec that rebaseline-cl is correct here. Why some of the mac bots had different results is the mystery. szager@, any hypothesis of why that might be?
,
Sep 14
This might be related to an issue I brought up on blink-dev recently: https://groups.google.com/a/chromium.org/d/msg/blink-dev/ivkXlBB3-XY/XkGdH8HiBwAJ Basically, there are some tests in LayoutTests/intersection-observer/v2 that do this: if (window.internals && internals.runtimeFlags) { internals.runtimeFlags.intersectionObserverV2Enabled = true; } If that js runs before the IntersectionObserver and IntersectionObserverEntry binding initialize code runs, then those extra attributes are added to the type templates. Because the layout test runner re-uses content_shell instances to run multiple tests, it may be that the global namespace was polluted because one of the intersection-observer/v2 tests ran before the global-interface-listing test.
,
Sep 14
Here is the recent CL that added the tests: https://chromium-review.googlesource.com/c/chromium/src/+/1212452
,
Sep 14
Oh, that would make sense. I thought that settings were reset between tests, but maybe not? Can you add cleanup code that resets the state back to the original?
,
Sep 14
Yeah, I'm working on just such a patch.
,
Sep 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/be431e79c6ecdc6a097e90f9defa077a0af7e2af commit be431e79c6ecdc6a097e90f9defa077a0af7e2af Author: Stefan Zager <szager@chromium.org> Date: Fri Sep 14 21:39:33 2018 [IntersectionObserverV2] Move all tests to virtual test suite Previously, IOv2 layout tests would enable IOv2 in script via: if (window.internals && internals.runtimeFlags) { internals.runtimeFlags.intersectionObserverV2Enabled true; } There are two problems with this approach: 1. By the time the above code runs, it may be too late: the prototype for the IntersectionObserverEntry javascript type may already have been created *without* the isVisible attribute. This was causing flaky test failures, because the exact timing of when the prototype is created may vary between runs. 2. Because the layout test runner re-uses content_shell processes for multiple tests, the side effects from turning on the feature flag affect tests that later use the same renderer process (see bug). This CL also adds DisableIntersectionObserverV2ThrottleForTesting, which eliminates the need to add a bunch of 100ms timers to the tests. Slow tests are bad. Finally, this CL reorders the handshake for cross-origin tests that rely on postMessage, to ensure that both the embedding frame and the child frame are properly initialized before starting the test procedure. BUG= 882280 , 883676 R=chrishtr@chromium.org,foolip@chromium.org Change-Id: I18e62c2cf099bf74ef4afc43f46a794c7a322582 Reviewed-on: https://chromium-review.googlesource.com/1220044 Commit-Queue: Stefan Zager <szager@chromium.org> Reviewed-by: Chris Harrelson <chrishtr@chromium.org> Cr-Commit-Position: refs/heads/master@{#591483} [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/TestExpectations [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/VirtualTestSuites [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/http/tests/intersection-observer/resources/v2-subframe.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/http/tests/intersection-observer/v2/cross-origin-effects.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/http/tests/intersection-observer/v2/cross-origin-occlusion.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/resources/intersection-observer-test-utils.js [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/v2/animated-occlusion.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/v2/box-shadow.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/v2/iframe-target.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/v2/simple-effects.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/v2/simple-occlusion.html [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/intersection-observer/v2/text-shadow.html [rename] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/virtual/intersection-observer-v2/http/tests/intersection-observer/v2/README.txt [copy] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/WebKit/LayoutTests/virtual/intersection-observer-v2/intersection-observer/v2/README.txt [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/blink/renderer/core/testing/internals.cc [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/blink/renderer/core/testing/internals.h [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/blink/renderer/core/testing/internals.idl [modify] https://crrev.com/be431e79c6ecdc6a097e90f9defa077a0af7e2af/third_party/blink/renderer/platform/runtime_enabled_features.json5
,
Sep 25
rebaseline-cl was actually working as expected here, closing. |
||
►
Sign in to add a comment |
||
Comment 1 by foolip@chromium.org
, Sep 13