Windows browser_tests, chrome_elf_unittests tests failing e.g. with `error: Value of: PolicyServiceIsEmpty(..)` - leaked registry state between runs |
|||||||
Issue descriptionChrome Version : 57.0.2984.0 Failures appear on https://uberchromegw.corp.google.com/i/chromium.win/builders/Win10%20Tests%20x64 from https://uberchromegw.corp.google.com/i/chromium.win/builders/Win10%20Tests%20x64/builds/8137 E.g. output like ComponentCloudPolicyTest.SignOutAndBackIn (run #1): [ RUN ] ComponentCloudPolicyTest.SignOutAndBackIn sending server_data: {"host": "127.0.0.1", "port": 50365} (36 bytes) [6020:1352:0205/030741.381:WARNING:policy_test_utils.cc(52)] There are pre-existing policies in this machine: { "MetricsReportingEnabled": false, "UserDataDir": "111" } c:\c\win\src\chromerowser\policy\cloud\component_cloud_policy_browsertest.cc(125): error: Value of: PolicyServiceIsEmpty(g_browser_process->policy_service()) Actual: false Expected: true Pre-existing policies in this machine will make this test fail. [ FAILED ] ComponentCloudPolicyTest.SignOutAndBackIn, where TypeParam = and GetParam() = (849 ms) That "111" string seems to come from https://codereview.chromium.org/2487783002 and UserDataDir.RegistrySettingsInHKLMTakesPrecedenceOverHKCU - landed ~2 months ago. It appears as though it's leaking registry state between test runs. I'm disabling tests in chrome/install_static/user_data_dir_win_unittest.cc and I'll find a trooper to help clean up registry keys. I still need to hunt down "MetricsReportingEnabled": false, as well. Nothing stands out in any of the bisect changes. sheriff-o-matic/FindIt just reports "Findit (?) results: Flaky"
,
Feb 6 2017
,
Feb 6 2017
+grt for the ` "MetricsReportingEnabled": false,` bit, which seems to be persisting across test runs and causing failures. Looks like this was (also) added a couple of months ago, in r432270 -> https://codereview.chromium.org/2422643002
,
Feb 6 2017
Poking around some more, https://build.chromium.org/p/chromium.win/builders/Win10%20Tests%20x64/builds/8155/steps/chrome_elf_unittests%20on%20Windows-10-10586/logs/BlacklistTest.AddAndRemoveModules has BlacklistTest.AddAndRemoveModules (run #1): [ RUN ] BlacklistTest.AddAndRemoveModules c:\c\win\srcase est est_reg_util_win.cc(76): error: Value of: temp_key_.Create((( HKEY ) (ULONG_PTR)((LONG)0x80000001) ), key_path.c_str(), (((0x001F0000L) | (0x0001) | (0x0002) | (0x0004) | (0x0008) | (0x0010) | (0x0020)) & (~(0x00100000L)))) Actual: 1450 Expected: 0L Which is: 0 [ FAILED ] BlacklistTest.AddAndRemoveModules (0 ms) "1450" appears to be status code 1450 (ERROR_NO_SYSTEM_RESOURCES). This error translates to "Insufficient system resources exist to complete the requested service." I'll follow up in Issue 688884 - I suspect that's the root cause of these. But if registry operations in these tests makes things flaky elsewhere this might be something to address still.
,
Feb 6 2017
> `UserDataDir.RegistrySettingsInHKLMTakesPrecedenceOverHKCU` still looks a bit sus though - I think it's relying on another test to run after it to restore the registry value to something sane? registry_util::RegistryOverrideManager is a scoped thing and it's supposed to write to a private location and then clean up as it goes out of scope.
,
Feb 6 2017
It looks like 688884 might have been the root cause though: https://bugs.chromium.org/p/chromium/issues/detail?id=688884#c7
,
Feb 6 2017
I'm going to mark this fixed as two builds have been fine now. No tests got disbled, right?
,
Feb 7 2017
> No tests got disbled, right?
correct :)
But I think RegistryOverrideManager should probably fail much harder that it does. ScopedRegistryKeyOverride does the following:
RegistryOverrideManager::ScopedRegistryKeyOverride::ScopedRegistryKeyOverride(
HKEY override,
const base::string16& key_path)
: override_(override) {
EXPECT_EQ(
ERROR_SUCCESS,
temp_key_.Create(HKEY_CURRENT_USER, key_path.c_str(), KEY_ALL_ACCESS));
EXPECT_EQ(ERROR_SUCCESS,
::RegOverridePredefKey(override_, temp_key_.Handle()));
}
It just flags a failure but allows the test to continue.
If the test then writes registry keys, will it proceed to modify the *real* registry on the machine and leave a mess for anything running subsequently? (i.e. are we making a bad situation cascade into more failures by not killing the process here?)
Assigning to grt -> I think you may know RegistryOverrideManager best. (feel free to WontFix if you think there's nothing to do though - I don't really know much about how the Windows registry works).
,
Feb 7 2017
Yeah, I think it makes sense for tests to fail hard if the override fails. Did anyone look at the affected machine to see if the disk was full or if the registry was chock full of old test data or something? Perhaps there's another problem lurking that needs to be fixed. The RegistryOverrideManager takes steps to clean up data from previous runs. Maybe there's other code that isn't doing this? Regarding comment 3, MetricsReportingEnabled has been around for a long time. My CL was just a refactoring.
,
Feb 13 2017
This started happening again since https://build.chromium.org/p/chromium.win/builders/Win10%20Tests%20x64/builds/8294 https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=chrome_elf_unittests&tests=BlacklistTest Reopened bug 688884
,
Feb 13 2017
I have a fix to the override manager underway to have tests fail fast if the override fails.
,
Feb 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f6d7da245a926307dc9c59d35cb893d9b1d65d82 commit f6d7da245a926307dc9c59d35cb893d9b1d65d82 Author: grt <grt@chromium.org> Date: Tue Feb 14 07:14:47 2017 Fail tests fast if overriding the Windows registry fails. All calls to RegistryOverrideManager::OverrideRegistry are now wrapped in ASSERT_NO_FATAL_FAILURE, as are a few helper functions in RegistryOverrideManagerTest that should have been. BUG= 688878 TBR=brettw@chromium.org Review-Url: https://codereview.chromium.org/2692843002 Cr-Commit-Position: refs/heads/master@{#450288} [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/base/test/test_reg_util_win.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/base/test/test_reg_util_win.h [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/base/test/test_reg_util_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/downgrade/user_data_downgrade_browsertest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/extensions/api/messaging/native_messaging_test_util.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/google/google_update_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/media_galleries/media_galleries_test_util.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/profile_resetter/triggered_profile_resetter_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/safe_browsing/incident_reporting/environment_data_collection_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/safe_browsing/incident_reporting/incident_reporting_service_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/safe_browsing/incident_reporting/platform_state_store_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/safe_browsing/incident_reporting/state_store_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/browser/win/chrome_elf_init_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/install_static/install_util_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/install_static/product_install_details_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/install_static/user_data_dir_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/gcapi/BUILD.gn [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/gcapi/gcapi_last_run_test.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/gcapi/gcapi_omaha_experiment_test.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/gcapi/gcapi_reactivation_test.cc [delete] https://crrev.com/0deaad5f28e26bb3eddf708271dd244da8abff44/chrome/installer/gcapi/gcapi_test_registry_overrider.cc [delete] https://crrev.com/0deaad5f28e26bb3eddf708271dd244da8abff44/chrome/installer/gcapi/gcapi_test_registry_overrider.h [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/mini_installer/configuration_test.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/setup/installer_state_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/setup/setup_util_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/setup/update_active_setup_version_work_item_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/beacons_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/delete_reg_value_work_item_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/google_update_settings_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/install_util_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/product_state_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/product_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/scoped_user_protocol_entry_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/set_reg_value_work_item_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome/installer/util/shell_util_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome_elf/blacklist/test/blacklist_test.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome_elf/chrome_elf_util_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/chrome_elf/nt_registry/nt_registry_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/components/browser_watcher/exit_code_watcher_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/components/browser_watcher/watcher_metrics_provider_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/components/user_prefs/tracked/registry_hash_store_contents_win_unittest.cc [modify] https://crrev.com/f6d7da245a926307dc9c59d35cb893d9b1d65d82/rlz/test/rlz_test_helpers.cc
,
Feb 14 2017
The test fixes described in comment 8 have landed. Any further failures like this are a result of bot issues. it's worth checking into why the registry operations are failing. For example, low disk space? |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by tapted@chromium.org
, Feb 6 2017