New issue
Advanced search Search tips

Issue 722083 link

Starred by 2 users

Issue metadata

Status: Duplicate
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

Widespread Windows Registry problems causing flakes on many VMs (Multiple PrefHashBrowserTest, all install_static_unittests, MediaGalleries, chrome_elf_unittests)

Project Member Reported by chromium...@appspot.gserviceaccount.com, May 14 2017

Issue description

"PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyeQsSBUZsYWtlIm5QcmVmSGFzaEJyb3dzZXJUZXN0VW50cnVzdGVkSW5pdGlhbGl6ZWRJbnN0YW5jZS9QcmVmSGFzaEJyb3dzZXJUZXN0VW50cnVzdGVkSW5pdGlhbGl6ZWQuVW50cnVzdGVkSW5pdGlhbGl6ZWQvMQw.

Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
 

Comment 1 by tapted@chromium.org, May 15 2017

Cc: gab@chromium.org
Components: UI>Browser>Preferences>Protector
Errors like

[5052:4700:0514/145440.239:WARNING:chrome_browser_main_win.cc(466)] Command line too long for RegisterApplicationRestart:  --brave-new-test-launcher --gtest_also_run_disabled_tests --gtest_filter=PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1 --single_process --test-launcher-bot-mode --test-launcher-summary-output="e:\b\swarm_slave\w\iosqgspi\output.json" --user-data-dir="C:\Users\CHROME~2\AppData\Local\Temp\scoped_dir3884_19973\d3884_16914" --disable-offline-auto-reload --force-fieldtrials=SettingsEnforcement/enforce_always/ --disable-features=NetworkPrediction --no-first-run --no-default-browser-check --enable-logging=stderr --log-level=0 --safebrowsing-disable-auto-update --disable-default-apps --wm-window-animations-disabled --disable-component-update --test-type=browser --disable-zero-browsers-open-for-tests --ipc-connection-timeout=30 --allow-file-access-from-files --dom-automation --log-gpu-control-list-decisions --disable-backgrounding-occluded-windows --disable-gl-drawing-for-tests --override-use-software-gl-for-tests --flag-switches-begin --flag-switches-end --restore-last-session about:blank
e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(750): error:       Expected: protection_level_ > PROTECTION_DISABLED_ON_PLATFORM ? num_tracked_prefs() : 0
      Which is: 21
To be equal to: GetTrackedPrefHistogramCount( user_prefs::tracked::kTrackedPrefHistogramUnchanged, user_prefs::tracked::kTrackedPrefRegistryValidationSuffix, ALLOW_ANY)
      Which is: 0
[  FAILED  ] PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1, where GetParam() = "enforce_always" (387 ms)
[ RUN      ] PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1


I think the command line too long thing is a red herring.

Failing on win and win64. First flake is at https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/441819 -> r470907

Seems to always fail with PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/1 cache

e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(517): error:       Expected: protection_level_ > PROTECTION_DISABLED_ON_PLATFORM ? num_tracked_prefs() : 0
      Which is: 21
To be equal to: GetTrackedPrefHistogramCount( user_prefs::tracked::kTrackedPrefHistogramUnchanged, user_prefs::tracked::kTrackedPrefRegistryValidationSuffix, ALLOW_ANY)
      Which is: 11
[  FAILED  ] PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/1, where GetParam() = "enforce_always" (302 ms)


Nothing seems to have changed recently around here. We might need to disable the tests - (maybe they need to wait for some histogram stuff to get flushed).

Comment 2 by tapted@chromium.org, May 15 2017

 Issue 722082  has been merged into this issue.

Comment 3 by tapted@chromium.org, May 15 2017

Summary: Multiple PrefHashBrowserTest flaky (was: "PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1" is flaky)
- PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/1
- PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/1
- PrefHashBrowserTestChangedAtomicInstance/PrefHashBrowserTestChangedAtomic.ChangedAtomic/3
- PrefHashBrowserTestClearedAtomicInstance/PrefHashBrowserTestClearedAtomic.ClearedAtomic/3

all with stuff like

e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(849): error:       Expected: protection_level_ > PROTECTION_DISABLED_ON_PLATFORM ? 1 : 0
      Which is: 1
To be equal to: GetTrackedPrefHistogramCount( user_prefs::tracked::kTrackedPrefHistogramChanged, user_prefs::tracked::kTrackedPrefRegistryValidationSuffix, BEGIN_ALLOW_SINGLE_BUCKET + 4)
      Which is: 0


Comment 4 by tapted@chromium.org, May 15 2017

 Issue 722076  has been merged into this issue.

Comment 5 by tapted@chromium.org, May 15 2017

Labels: OS-Windows
And others, which have perhaps not flaked enough to get reported

from https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/441975

new:
PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipeInstance/PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipe.UntrustedAdditionToPrefsAfterWipe/0 cache
PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipeInstance/PrefHashBrowserTestUntrustedAdditionToPrefsAfterWipe.UntrustedAdditionToPrefsAfterWipe/3 cache
PrefHashBrowserTestClearedAtomicInstance/PrefHashBrowserTestClearedAtomic.ClearedAtomic/0 cache
PrefHashBrowserTestUnchangedDefaultInstance/PrefHashBrowserTestUnchangedDefault.UnchangedDefault/0 cache
PrefHashBrowserTestChangedAtomicInstance/PrefHashBrowserTestChangedAtomic.ChangedAtomic/1 cache
PrefHashBrowserTestUnchangedCustomInstance/PrefHashBrowserTestUnchangedCustom.UnchangedCustom/0 cache
PrefHashBrowserTestUntrustedInitializedInstance/PrefHashBrowserTestUntrustedInitialized.UntrustedInitialized/2 cache


(others reported in that run
MediaGalleriesGalleryWatchApiTest.SetupGalleryWatchWithoutListeners cache
ExtensionApiTest.UserLevelNativeMessaging cache
MediaGalleriesGalleryWatchApiTest.RemoveListenerAndModifyGallery cache
IEImporterBrowserTest.IE7ImporterPasswordsTest cache
CloudPolicyManagerTest.RegisterFailsWithRetries cache


some of these are Windows registry errors. Perhaps that's hinting at a root cause..

Comment 6 by tapted@chromium.org, May 15 2017

 Issue 722159  has been merged into this issue.

Comment 7 by tapted@chromium.org, May 15 2017

 Issue 722158  has been merged into this issue.

Comment 8 by tapted@chromium.org, May 15 2017

Cc: grt@chromium.org
Summary: Multiple PrefHashBrowserTest flaky - Windows Registry problems? (was: Multiple PrefHashBrowserTest flaky)
Ah, OK. So these PrefHashBrowserTest seem to have a PRE_ test that involves Windows registry writes, and that's also failing. Sometimes.


I'm not sure pref_hash_browsertest.cc is using  "base/test/test_reg_util_win.h" properly.. It #includes it, but doesn't reference anything in registry_util::



[595/600] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.PRE_RegistryValidationFailure/1 (332 ms)
[ RUN      ] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.RegistryValidationFailure/1
e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(1150): error:       Expected: 0L
      Which is: 0
To be equal to: key.Open((( HKEY ) (ULONG_PTR)((LONG)0x80000001) ), registry_key.c_str(), (0x0002) | (0x0200))
      Which is: 2

or

[596/606] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.PRE_RegistryValidationFailure/1 (484 ms)
[ RUN      ] PrefHashBrowserTestRegistryValidationFailureInstance/PrefHashBrowserTestRegistryValidationFailure.RegistryValidationFailure/1
e:\b\c\b\win\src\chrome\browser\prefs\tracked\pref_hash_browsertest.cc(1153): error: Value of: key.WriteValue(L"homepage", base::string16(64, 'A').c_str())
  Actual: 1450
Expected: 0L
Which is: 0





Other output in the test run that might be related

[ RUN      ] CloudPolicyTest.InvalidatePolicy

sending server_data: {"host": "127.0.0.1", "port": 51194} (36 bytes)
[4552:648:0512/123544.366:ERROR:configuration_policy_handler_list.cc(92)] Unknown policy: StringPolicy
[4552:648:0512/123544.389:ERROR:configuration_policy_handler_list.cc(92)] Unknown policy: StringPolicy
[1076:3724:0512/123544.438:INFO:media_foundation_video_encode_accelerator_win.cc(329)] Windows versions earlier than 8 are not supported.
[4552:648:0512/123544.543:WARNING:policy_test_utils.cc(52)] There are pre-existing policies in this machine: {
   "StringPolicy": "hklm"
}
e:\b\c\b\win\src\chrome\browser\policy\cloud\cloud_policy_browsertest.cc(202): error: Value of: PolicyServiceIsEmpty(g_browser_process->policy_service())
  Actual: false
Expected: true




There isn't a particular bot where this is happening. Seems to be sharded out to a bunch of VMs, which all fail, and are reporting registry issues in multiple tests.

vm15-m4 - https://chromium-swarm.appspot.com/task?id=3621c3b313c9ef10&refresh=10&show_raw=1
vm229-m4 - https://chromium-swarm.appspot.com/task?id=361d1b5a8072f010&refresh=10&show_raw=1
vm229-m4 - https://chromium-swarm.appspot.com/task?id=361b4ffe2ccdbc10&refresh=10&show_raw=1
vm229-m4 - https://chromium-swarm.appspot.com/task?id=36124cbe365ccf10&refresh=10&show_raw=1
vm53-m4 - https://chromium-swarm.appspot.com/task?id=361a047e9c339d10&refresh=10&show_raw=1
vm1-m4 - https://chromium-swarm.appspot.com/task?id=36186b5e18a62910&refresh=10&show_raw=1
vm15-m4 - https://chromium-swarm.appspot.com/task?id=3616f32254562310&refresh=10&show_raw=1
vm15-m4 - https://chromium-swarm.appspot.com/task?id=3610f2d0b39f8710&refresh=10&show_raw=1
vm32-m4 - https://chromium-swarm.appspot.com/task?id=36144d51000a3810&refresh=10&show_raw=1


Nearly all the runs have this "Unknown policy: StringPolicy" error being emitted in earlier tests.

not this bot though 
vm467-m4 - https://chromium-swarm.appspot.com/task?id=361252998d2ab510&refresh=10&show_raw=1
vm467-m4 - https://chromium-swarm.appspot.com/task?id=36111c12e31d8310&refresh=10&show_raw=1

oh, but it does in this run:

vm467-m4 - https://chromium-swarm.appspot.com/task?id=361252998d2ab510&refresh=10&show_raw=1

Comment 9 by tapted@chromium.org, May 15 2017

"""RegOpenKeyEx returning status code 1450 (ERROR_NO_SYSTEM_RESOURCES). This error translates to "Insufficient system resources exist to complete the requested service."""

"""A single registry key can be opened only 65534 times. When attempting the 65535th open operation, this function fails with ERROR_NO_SYSTEM_RESOURCES."""

I don't see any recent changes mucking with registry keys :/

We don't check the result of RegCloseKey in

void RegKey::Close() {
  if (key_) {
    ::RegCloseKey(key_);

That might be a red-herring though.


Note this seems to be widespread :/. So far the following are likely the same root cause

  http://crbug.com/721983  - chrome_elf_unittests BlacklistTest.SetupFailed
  http://crbug.com/722017  - install_static_unittests 
  http://crbug.com/722075  - MediaGalleriesGalleryWatchApiTest
  http://crbug.com/722083  - (this bug)

 Issue 721609  has been merged into this issue.
 Issue 722017  has been merged into this issue.
Cc: reillyg@chromium.org
 Issue 722075  has been merged into this issue.
Components: Platform>Extensions>API Internals>Installer
Owner: grt@chromium.org
Status: Assigned (was: Untriaged)
Summary: Widespread Windows Registry problems causing flakes on many VMs (Multiple PrefHashBrowserTest, all install_static_unittests, MediaGalleries, chrome_elf_unittests) (was: Multiple PrefHashBrowserTest flaky - Windows Registry problems?)
Mergedinto: 721245
Status: Duplicate (was: Assigned)
ah, finally found Issue 721245. Gosh.
did message go trough.

Sign in to add a comment