This is actually a significant regression. We lost 83 (!) unit tests in one slew.

Removing 'chrome' since we have policy indicators in non CrOS also (I'm not sure why that browser test was CrOS only).

There is unit test coverage for the indicators here:
https://cs.chromium.org/chromium/src/chrome/test/data/webui/cr_elements/cr_policy_indicator_tests.js

I think we are missing coverage for 'recommended' which I will add when I address issue 841620.

I will also audit the current indicators test, and look into adding an integration test.

The unit test verifies that the indicator Polymer element functions correctly.

The tests that went missing actually went through all policies expected to trigger indicators and verified that each of them is present in the UI, and reacts to the policy correctly.

Good question why the test was Chrome OS only. I guess I just never got around to verifying it works correctly on desktop platforms. It should all work the same there.

FWIW, porting the logic at [1] to work in the new page is going to be challenging (and probably result in a flaky test).
 1) you will have to use a /deep/ CSS query to pierce Shadow DOM. 
 2) The new page has lot of lazy loading, idle loading, which will make the timing of when to run the query fairly fragile. 

[1] https://chromium-review.googlesource.com/c/chromium/src/+/583935/7/chrome/browser/policy/policy_prefs_browsertest.cc

I wouldn't want to try to restore the old tests, I agree that they would be much too fragile.

Unfortunately, maintaining any sort of integration tests for Settings has been challenging. Even our "basic page tests" suite has been disabled due to flakiness:
https://cs.chromium.org/chromium/src/chrome/test/data/webui/settings/basic_page_browsertest.js

When investigating this before, a large challenge is the length of time these integration tests take to run. Even with all of our optimizations, loading the Settings page is resource intensive, so on heavily loaded test machines they are prone to timeouts.

In addition to the cr_policy_indicator_tests.js, I remembered that we do have:
https://cs.chromium.org/chromium/src/chrome/test/data/webui/settings/controlled_button_tests.js
https://cs.chromium.org/chromium/src/chrome/test/data/webui/settings/controlled_radio_button_tests.js

These are probably the correct level of testing; the policy blobs are injected so it's not a full integration test, but we shouldn't necessarily be testing the mechanism for sending policy blobs to Settings in the indicator tests. That is covered by the settings private API tests:
https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/settings_private/settings_private_apitest.cc

The best thing to do here I think is to expand coverage in the controlled_button_tests (e.g. to include 'recommended' policy) and to cover all controls with policy indicators.

This would still be unit-testing the various control types only, not verifying that the individual prefs which should have indicators actually have them.

policy_test_cases.json already had support for per-policy setup steps, allowing you to simulate user actions that trigger hidden parts of the Settings page to load. Would this approach not work with the new Settings as well?

Note that as a result of migrating to Shadow DOM V1, using /deep/ to reach various parts of the UI is now not an option anymore (see issue 860069).

 Issue 870106  has been merged into this issue.

Should we at least update the comments in policy_test_cases.json? For example, it says the test URL defaults to "chrome://extensions-frame", which doesn't make senses (and the fields are unused anyway).

In the meantime the values used here by new policies might be arbitrary (because tests won't break).

SGTM. Moreover there are a few more references to chrome://extensions-frame across the codebase, and we should remove them as well, see [1].


[1] https://cs.chromium.org/search/?q=chrome://extensions-frame+-file:third_party+-file:infra+-file:out/Debug+-file:out/chromium-android/Debug+-file:out/android-Debug+-file:src/v8+-file:src/native_client-sdk+-file:out/chromeos-Debug+-file:out/win-Debug&type=cs