I found a couple of issue in regards how layout tests preferences are managed. I grouped them together in a single issue because it makes more sense to solve both at once.

The effects of them are subtle enough to make it difficult to understand what's going on. It was the case for me in recent changes around these settings.

#1: Default preferences differ between browser and renderer.

Layout test preferences default values are defined in two different places: TestPreferences::Reset [1], used at the renderer, and ApplyLayoutTestDefaultPreferences [2], used at the browser (kinda weird because it lives in a renderer dir).

#2: BlinkTestRunner::ApplyPreferences has more side effects than it should be expected.

When tests need to change configurations controlled by the TestRunner component they do this by first changing values in TestPreferences obtained from from WebTestDelegate::Preferences and then calling WebTestDelegate::ApplyPreferences. See TestRunner::OverridePreference [3] as an example.

But the implementation of BlinkTestRunner::ApplyPreferences [4] has to invoke ExportLayoutTestSpecificPreferences to apply the updated values but it also applies all other values from TestPreferences. And this might end up changing more preferences than intended.

For instance, when executing the layout test fast/parser/noscript-with-javascript-disabled.html [5], which is only meant to disable javascript, ends up changing other preferences when ApplyPreferences is called:

Before -> After:
 - javascript_enabled:             1 -> 0  expected
 - experimental_webgl_enabled:     1 -> 0  *unexpected*
 - allow_running_insecure_content: 0 -> 1  *unexpected*

Note that this effect is a consequence of #1.


[1] https://cs.chromium.org/chromium/src/components/test_runner/test_preferences.cc?type=cs&q="TestPreferences::Reset"

[2] https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_helpers.cc?type=cs&q="ApplyLayoutTestDefaultPreferences"+file:layout_test/blink_test_helpers.cc&sq=package:chromium

[3] https://cs.chromium.org/chromium/src/components/test_runner/test_runner.cc?type=cs&q="TestRunner::OverridePreference"

[4] https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs&q="BlinkTestRunner::ApplyPreferences"

[5] https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/fast/parser/noscript-with-javascript-disabled.html