Telemetry failures due to missing system salt file |
||||||||
Issue descriptionA large number of unit tests are failing on telemetry bots because /home/.shadow/salt is missing: Example: https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-telemetry/builds/9531 08/11 05:09:49.453 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 08/11 05:09:49.463 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/browser_test_case.py", line 89, in setUpClass 08/11 05:09:49.487 INFO |run_chromeos_tests:0052| current_browser = browser_to_create.Create(options) 08/11 05:09:49.488 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_finder.py", line 52, in Create 08/11 05:09:49.489 INFO |run_chromeos_tests:0052| browser_backend, self._platform_backend, self._credentials_path) 08/11 05:09:49.490 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 55, in __init__ 08/11 05:09:49.491 INFO |run_chromeos_tests:0052| self._browser_backend.Start() 08/11 05:09:49.491 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 148, in Start 08/11 05:09:49.496 INFO |run_chromeos_tests:0052| self._WaitForLogin() 08/11 05:09:49.501 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 244, in _WaitForLogin 08/11 05:09:49.502 INFO |run_chromeos_tests:0052| util.WaitFor(self._IsLoggedIn, 60) 08/11 05:09:49.503 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/util.py", line 86, in WaitFor 08/11 05:09:49.504 INFO |run_chromeos_tests:0052| res = condition() 08/11 05:09:49.505 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 240, in _IsLoggedIn 08/11 05:09:49.506 INFO |run_chromeos_tests:0052| return not self._GetLoginStatus() 08/11 05:09:49.506 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 229, in _GetLoginStatus 08/11 05:09:49.507 INFO |run_chromeos_tests:0052| if not self._IsCryptohomeMounted(): 08/11 05:09:49.508 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 224, in _IsCryptohomeMounted 08/11 05:09:49.509 INFO |run_chromeos_tests:0052| return self._cri.IsCryptohomeMounted(username, self._is_guest) 08/11 05:09:49.510 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/cros_interface.py", line 474, in IsCryptohomeMounted 08/11 05:09:49.512 INFO |run_chromeos_tests:0052| profile_path = self.CryptohomePath(username) 08/11 05:09:49.512 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/cros_interface.py", line 469, in CryptohomePath 08/11 05:09:49.513 INFO |run_chromeos_tests:0052| raise OSError('cryptohome-path failed: %s' % stderr) 08/11 05:09:49.514 INFO |run_chromeos_tests:0052| OSError: cryptohome-path failed: [0811/050949:ERROR:cryptohome.cc(39)] Could not get size of system salt: /home/.shadow/salt: No such file or directory
,
Aug 15 2016
Darren, Andrey - do you know of any reason why cryptohome-path might be failing?
,
Aug 15 2016
,
Aug 15 2016
Alex, why was this CL necessary: https://codereview.chromium.org/2220433002
,
Aug 15 2016
First failing build: https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-telemetry/builds/10182 Strongly suspect this CL: https://chromium-review.googlesource.com/#/c/363370/ because it matches the timeline of the failures. Second possibility: https://chromium-review.googlesource.com/#/c/360132/
,
Aug 15 2016
Gardeners - it may make sense to try to locally reproduce this and then try backing out the chromeos changes first, and if that doesn't yield anything, try a bisect on the chrome changes?
,
Aug 15 2016
Re #5 and https://chromium-review.googlesource.com/#/c/363370/: wasn't that change reverted already? See https://groups.google.com/a/google.com/d/msgid/chromeos-infra-discuss/CAASgrz18TKzV6LOiOxq5xrm3toKcyp7BkwDZ8fuTkddNrRZEzA%40mail.gmail.com
,
Aug 15 2016
> Alex, why was this CL necessary: https://codereview.chromium.org/2220433002 This was a my regression in M52. It returns back old behavior of "waiting for salt to load" before setting enterprise wallpaper. This should prevent faults on "CHECK" in wallpaper manager. I don't think that it should change any ChromeOS behavior.
,
Aug 15 2016
Yup, looks like 363370 was reverted here: https://chromium-review.googlesource.com/#/c/368130/
,
Aug 15 2016
Yep, that CL caused cryptohomed to not start on any environment that doesn't support the 'vpd' CLI (including VMs I assume). Just a heads up, it's re-landing with a fix: https://chromium-review.googlesource.com/#/c/370302/
,
Aug 16 2016
The last failing build is https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-telemetry/builds/10185 The above revert landed around that time and the issue disappeared in subsequent builds, so marking as fixed.
,
Aug 29 2016
,
Aug 29 2016
bulk verified
,
Aug 29 2016
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by domlasko...@chromium.org
, Aug 15 2016