Widespread Windows Registry problems causing flakes on many VMs (Multiple PrefHashBrowserTest, all install_static_unittests, MediaGalleries, chrome_elf_unittests) |
|||||
Issue description"PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label. We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyeQsSBUZsYWtlIm5QcmVmSGFzaEJyb3dzZXJUZXN0VW50cnVzdGVkSW5pdGlhbGl6ZWRJbnN0YW5jZS9QcmVmSGFzaEJyb3dzZXJUZXN0VW50cnVzdGVkSW5pdGlhbGl6ZWQuVW50cnVzdGVkSW5pdGlhbGl6ZWQvMQw. Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
,
May 15 2017
Issue 722082 has been merged into this issue.
,
May 15 2017
- PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1
- PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/1
- PrefHashBrowserTestChangedAtomicInstance/PrefHashBrowserTestChangedAtomic.ChangedAtomic/3
- PrefHashBrowserTestClearedAtomicInstance/PrefHashBrowserTestClearedAtomic.ClearedAtomic/3
all with stuff like
e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(849): error: Expected: protection_level_ > PROTECTION_DISABLED_ON_PLATFORM ? 1 : 0
Which is: 1
To be equal to: GetTrackedPrefHistogramCount( user_prefs::tracked::kTrackedPrefHistogramChanged, user_prefs::tracked::kTrackedPrefRegistryValidationSuffix, BEGIN_ALLOW_SINGLE_BUCKET + 4)
Which is: 0
,
May 15 2017
Issue 722076 has been merged into this issue.
,
May 15 2017
And others, which have perhaps not flaked enough to get reported from https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/441975 new: PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipeInstance/PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipe.UntrustedAdditionToPrefsAfterWipe/0 cache PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipeInstance/PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipe.UntrustedAdditionToPrefsAfterWipe/3 cache PrefHashBrowserTestClearedAtomicInstance/PrefHashBrowserTestClearedAtomic.ClearedAtomic/0 cache PrefHashBrowserTestUnchangedDefaultInstance/PrefHashBrowserTestUnchangedDefault.UnchangedDefault/0 cache PrefHashBrowserTestChangedAtomicInstance/PrefHashBrowserTestChangedAtomic.ChangedAtomic/1 cache PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/0 cache PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/2 cache (others reported in that run MediaGalleriesGalleryWatchApiTest.SetupGalleryWatchWithoutListeners cache ExtensionApiTest.UserLevelNativeMessaging cache MediaGalleriesGalleryWatchApiTest.RemoveListenerAndModifyGallery cache IEImporterBrowserTest.IE7ImporterPasswordsTest cache CloudPolicyManagerTest.RegisterFailsWithRetries cache some of these are Windows registry errors. Perhaps that's hinting at a root cause..
,
May 15 2017
Issue 722159 has been merged into this issue.
,
May 15 2017
Issue 722158 has been merged into this issue.
,
May 15 2017
Ah, OK. So these PrefHashBrowserTest seem to have a PRE_ test that involves Windows registry writes, and that's also failing. Sometimes.
I'm not sure pref_hash_browsertest.cc is using "base/test/test_reg_util_win.h" properly.. It #includes it, but doesn't reference anything in registry_util::
[595/600] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.PRE_RegistryValidationFailure/1 (332 ms)
[ RUN ] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.RegistryValidationFailure/1
e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(1150): error: Expected: 0L
Which is: 0
To be equal to: key.Open((( HKEY ) (ULONG_PTR)((LONG)0x80000001) ), registry_key.c_str(), (0x0002) | (0x0200))
Which is: 2
or
[596/606] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.PRE_RegistryValidationFailure/1 (484 ms)
[ RUN ] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.RegistryValidationFailure/1
e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(1153): error: Value of: key.WriteValue(L"homepage", base::string16(64, 'A').c_str())
Actual: 1450
Expected: 0L
Which is: 0
Other output in the test run that might be related
[ RUN ] CloudPolicyTest.InvalidatePolicy
sending server_data: {"host": "127.0.0.1", "port": 51194} (36 bytes)
[4552:648:0512/123544.366:ERROR:configuration_policy_handler_list.cc(92)] Unknown policy: StringPolicy
[4552:648:0512/123544.389:ERROR:configuration_policy_handler_list.cc(92)] Unknown policy: StringPolicy
[1076:3724:0512/123544.438:INFO:media_foundation_video_encode_accelerator_win.cc(329)] Windows versions earlier than 8 are not supported.
[4552:648:0512/123544.543:WARNING:policy_test_utils.cc(52)] There are pre-existing policies in this machine: {
"StringPolicy": "hklm"
}
e:\b\c\b\win\src\chrome\browser\policy\cloud\cloud_policy_browsertest.cc(202): error: Value of: PolicyServiceIsEmpty(g_browser_process->policy_service())
Actual: false
Expected: true
There isn't a particular bot where this is happening. Seems to be sharded out to a bunch of VMs, which all fail, and are reporting registry issues in multiple tests.
vm15-m4 - https://chromium-swarm.appspot.com/task?id=3621c3b313c9ef10&refresh=10&show_raw=1
vm229-m4 - https://chromium-swarm.appspot.com/task?id=361d1b5a8072f010&refresh=10&show_raw=1
vm229-m4 - https://chromium-swarm.appspot.com/task?id=361b4ffe2ccdbc10&refresh=10&show_raw=1
vm229-m4 - https://chromium-swarm.appspot.com/task?id=36124cbe365ccf10&refresh=10&show_raw=1
vm53-m4 - https://chromium-swarm.appspot.com/task?id=361a047e9c339d10&refresh=10&show_raw=1
vm1-m4 - https://chromium-swarm.appspot.com/task?id=36186b5e18a62910&refresh=10&show_raw=1
vm15-m4 - https://chromium-swarm.appspot.com/task?id=3616f32254562310&refresh=10&show_raw=1
vm15-m4 - https://chromium-swarm.appspot.com/task?id=3610f2d0b39f8710&refresh=10&show_raw=1
vm32-m4 - https://chromium-swarm.appspot.com/task?id=36144d51000a3810&refresh=10&show_raw=1
Nearly all the runs have this "Unknown policy: StringPolicy" error being emitted in earlier tests.
not this bot though
vm467-m4 - https://chromium-swarm.appspot.com/task?id=361252998d2ab510&refresh=10&show_raw=1
vm467-m4 - https://chromium-swarm.appspot.com/task?id=36111c12e31d8310&refresh=10&show_raw=1
oh, but it does in this run:
vm467-m4 - https://chromium-swarm.appspot.com/task?id=361252998d2ab510&refresh=10&show_raw=1
,
May 15 2017
"""RegOpenKeyEx returning status code 1450 (ERROR_NO_SYSTEM_RESOURCES). This error translates to "Insufficient system resources exist to complete the requested service."""
"""A single registry key can be opened only 65534 times. When attempting the 65535th open operation, this function fails with ERROR_NO_SYSTEM_RESOURCES."""
I don't see any recent changes mucking with registry keys :/
We don't check the result of RegCloseKey in
void RegKey::Close() {
if (key_) {
::RegCloseKey(key_);
That might be a red-herring though.
Note this seems to be widespread :/. So far the following are likely the same root cause
http://crbug.com/721983 - chrome_elf_unittests BlacklistTest.SetupFailed
http://crbug.com/722017 - install_static_unittests
http://crbug.com/722075 - MediaGalleriesGalleryWatchApiTest
http://crbug.com/722083 - (this bug)
,
May 15 2017
Issue 721609 has been merged into this issue.
,
May 15 2017
Issue 722017 has been merged into this issue.
,
May 15 2017
,
May 15 2017
,
May 15 2017
,
May 26 2017
did message go trough. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by tapted@chromium.org
, May 15 2017Components: UI>Browser>Preferences>Protector
Errors like [5052:4700:0514/145440.239:WARNING:chrome_browser_main_win.cc(466)] Command line too long for RegisterApplicationRestart: --brave-new-test-launcher --gtest_also_run_disabled_tests --gtest_filter=PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1 --single_process --test-launcher-bot-mode --test-launcher-summary-output="e:\b\swarm_slave\w\iosqgspi\output.json" --user-data-dir="C:\Users\CHROME~2\AppData\Local\Temp\scoped_dir3884_19973\d3884_16914" --disable-offline-auto-reload --force-fieldtrials=SettingsEnforcement/enforce_always/ --disable-features=NetworkPrediction --no-first-run --no-default-browser-check --enable-logging=stderr --log-level=0 --safebrowsing-disable-auto-update --disable-default-apps --wm-window-animations-disabled --disable-component-update --test-type=browser --disable-zero-browsers-open-for-tests --ipc-connection-timeout=30 --allow-file-access-from-files --dom-automation --log-gpu-control-list-decisions --disable-backgrounding-occluded-windows --disable-gl-drawing-for-tests --override-use-software-gl-for-tests --flag-switches-begin --flag-switches-end --restore-last-session about:blank e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(750): error: Expected: protection_level_ > PROTECTION_DISABLED_ON_PLATFORM ? num_tracked_prefs() : 0 Which is: 21 To be equal to: GetTrackedPrefHistogramCount( user_prefs::tracked::kTrackedPrefHistogramUnchanged, user_prefs::tracked::kTrackedPrefRegistryValidationSuffix, ALLOW_ANY) Which is: 0 [ FAILED ] PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1, where GetParam() = "enforce_always" (387 ms) [ RUN ] PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1 I think the command line too long thing is a red herring. Failing on win and win64. First flake is at https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/441819 -> r470907 Seems to always fail with PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/1 cache e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(517): error: Expected: protection_level_ > PROTECTION_DISABLED_ON_PLATFORM ? num_tracked_prefs() : 0 Which is: 21 To be equal to: GetTrackedPrefHistogramCount( user_prefs::tracked::kTrackedPrefHistogramUnchanged, user_prefs::tracked::kTrackedPrefRegistryValidationSuffix, ALLOW_ANY) Which is: 11 [ FAILED ] PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/1, where GetParam() = "enforce_always" (302 ms) Nothing seems to have changed recently around here. We might need to disable the tests - (maybe they need to wait for some histogram stuff to get flushed).