testListSimpleBenchmarksDefinedInOneFileComplex test failure |
|||||||||
Issue description
Telemetry bots have been red since August 11 due to a unit test failure:
08/15 12:08:40.997 INFO |run_chromeos_tests:0052| [190/274] core.benchmark_finders_unittest.TestGetBenchmarkNamesForFile.testListSimpleBenchmarksDefinedInOneFileComplex failed unexpectedly 0.0080s:
08/15 12:08:40.998 INFO |run_chromeos_tests:0052| Traceback (most recent call last):
08/15 12:08:40.998 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/tools/perf/core/benchmark_finders_unittest.py", line 34, in testListSimpleBenchmarksDefinedInOneFileComplex
08/15 12:08:40.999 INFO |run_chromeos_tests:0052| 'test_benchmark_complex_subclass_from_other_module'])
08/15 12:08:41.000 INFO |run_chromeos_tests:0052| AssertionError: Lists differ: ['test_benchmark_1', 'test_ben... != ['test_benchmark_complex_1', '...
08/15 12:08:41.000 INFO |run_chromeos_tests:0052|
08/15 12:08:41.001 INFO |run_chromeos_tests:0052| First differing element 0:
08/15 12:08:41.001 INFO |run_chromeos_tests:0052| test_benchmark_1
08/15 12:08:41.002 INFO |run_chromeos_tests:0052| test_benchmark_complex_1
08/15 12:08:41.003 INFO |run_chromeos_tests:0052|
08/15 12:08:41.003 INFO |run_chromeos_tests:0052| First list contains 4 additional elements.
08/15 12:08:41.004 INFO |run_chromeos_tests:0052| First extra element 3:
08/15 12:08:41.005 INFO |run_chromeos_tests:0052| test_benchmark_complex_subclass
08/15 12:08:41.005 INFO |run_chromeos_tests:0052|
08/15 12:08:41.006 INFO |run_chromeos_tests:0052| - ['test_benchmark_1',
08/15 12:08:41.007 INFO |run_chromeos_tests:0052| - 'test_benchmark_2',
08/15 12:08:41.008 INFO |run_chromeos_tests:0052| - 'test_benchmark_complex_1',
08/15 12:08:41.008 INFO |run_chromeos_tests:0052| ? ^
08/15 12:08:41.009 INFO |run_chromeos_tests:0052|
08/15 12:08:41.009 INFO |run_chromeos_tests:0052| + ['test_benchmark_complex_1',
08/15 12:08:41.010 INFO |run_chromeos_tests:0052| ? ^
08/15 12:08:41.011 INFO |run_chromeos_tests:0052|
08/15 12:08:41.011 INFO |run_chromeos_tests:0052| 'test_benchmark_complex_subclass',
08/15 12:08:41.012 INFO |run_chromeos_tests:0052| - 'test_benchmark_complex_subclass_from_other_module',
08/15 12:08:41.013 INFO |run_chromeos_tests:0052| ? ^
08/15 12:08:41.013 INFO |run_chromeos_tests:0052|
08/15 12:08:41.014 INFO |run_chromeos_tests:0052| + 'test_benchmark_complex_subclass_from_other_module']
08/15 12:08:41.014 INFO |run_chromeos_tests:0052| ? ^
08/15 12:08:41.015 INFO |run_chromeos_tests:0052|
08/15 12:08:41.016 INFO |run_chromeos_tests:0052| - 'test_benchmark_subclass_1',
08/15 12:08:41.017 INFO |run_chromeos_tests:0052| - 'test_benchmark_subclass_2']
In certain runs, a large number of tests are also failing due to a missing salt file:
08/11 05:09:49.453 INFO |run_chromeos_tests:0052| Traceback (most recent call last):
08/11 05:09:49.463 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/browser_test_case.py", line 89, in setUpClass
08/11 05:09:49.487 INFO |run_chromeos_tests:0052| current_browser = browser_to_create.Create(options)
08/11 05:09:49.488 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_finder.py", line 52, in Create
08/11 05:09:49.489 INFO |run_chromeos_tests:0052| browser_backend, self._platform_backend, self._credentials_path)
08/11 05:09:49.490 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 55, in __init__
08/11 05:09:49.491 INFO |run_chromeos_tests:0052| self._browser_backend.Start()
08/11 05:09:49.491 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 148, in Start
08/11 05:09:49.496 INFO |run_chromeos_tests:0052| self._WaitForLogin()
08/11 05:09:49.501 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 244, in _WaitForLogin
08/11 05:09:49.502 INFO |run_chromeos_tests:0052| util.WaitFor(self._IsLoggedIn, 60)
08/11 05:09:49.503 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/util.py", line 86, in WaitFor
08/11 05:09:49.504 INFO |run_chromeos_tests:0052| res = condition()
08/11 05:09:49.505 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 240, in _IsLoggedIn
08/11 05:09:49.506 INFO |run_chromeos_tests:0052| return not self._GetLoginStatus()
08/11 05:09:49.506 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 229, in _GetLoginStatus
08/11 05:09:49.507 INFO |run_chromeos_tests:0052| if not self._IsCryptohomeMounted():
08/11 05:09:49.508 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 224, in _IsCryptohomeMounted
08/11 05:09:49.509 INFO |run_chromeos_tests:0052| return self._cri.IsCryptohomeMounted(username, self._is_guest)
08/11 05:09:49.510 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/cros_interface.py", line 474, in IsCryptohomeMounted
08/11 05:09:49.512 INFO |run_chromeos_tests:0052| profile_path = self.CryptohomePath(username)
08/11 05:09:49.512 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/cros_interface.py", line 469, in CryptohomePath
08/11 05:09:49.513 INFO |run_chromeos_tests:0052| raise OSError('cryptohome-path failed: %s' % stderr)
08/11 05:09:49.514 INFO |run_chromeos_tests:0052| OSError: cryptohome-path failed: [0811/050949:ERROR:cryptohome.cc(39)] Could not get size of system salt: /home/.shadow/salt: No such file or directory
,
Aug 15 2016
+nednguyen achuith, I am suspicious that this CL from 4 days ago might have caused the failure: https://codereview.chromium.org/2228413003 Does that CL look suspicious to you? Dominik, do you want to take this bug, since you've been investigating this issue so far? Example failure, for reference: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/10190 It looks like the earlier builds failed with the salt issue, then this started failing?
,
Aug 15 2016
The first issue (testListSimpleBenchmarksDefinedInOneFileComplex) could potentially affect cros performance tests. The salt issue will probably affect PFQ and CQ runs, and is quite serious.
,
Aug 15 2016
Filed separate bug for salt issue: https://bugs.chromium.org/p/chromium/issues/detail?id=637960
,
Aug 15 2016
This is a bug in [1] that was masked by the duplicate methods fixed by the above CL. According to the Python docs, the first argument of imp.load_source "is used to create or access a module object". Since the module name is constant, the loaded files are merged into a single cached module, so the test that runs second contains benchmarks from both files. [1] https://chromium.googlesource.com/chromium/src/+/master/tools/perf/core/benchmark_finders.py#31
,
Aug 16 2016
Reverting for now: https://codereview.chromium.org/2244373003/
,
Aug 16 2016
You can disable the test with @decorators.Disabled('chromeOs') when waiting for the actual fix. Please do not revert the CL.
,
Aug 16 2016
Good to know. Sorry about jumping the gun with the revert. Considering that the telemetry bots don't block the PFQ, this isn't really urgent, so reassigning to Ned for the real fix instead of disabling the test.
,
Aug 16 2016
Dominik, please disable the test asap - it is trivial to do so. I don't think we want the bots to stay red while this is sorted out.
,
Aug 16 2016
I will disable the test.
,
Aug 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7fe492ff89e7cf3b3ac5d0e1569ce98d66a9cc67 commit 7fe492ff89e7cf3b3ac5d0e1569ce98d66a9cc67 Author: jamescook <jamescook@chromium.org> Date: Tue Aug 16 17:49:43 2016 Disable TestGetBenchmarkNamesForFile.testListSimpleBenchmarksDefinedInOneFileComplex on chromeos It is causing chromeos telemetry bot failures: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/10190 BUG=637938 TEST=perf tests Review-Url: https://codereview.chromium.org/2254473002 Cr-Commit-Position: refs/heads/master@{#412282} [modify] https://crrev.com/7fe492ff89e7cf3b3ac5d0e1569ce98d66a9cc67/tools/perf/core/benchmark_finders_unittest.py
,
Aug 29 2016
This same issue seems to be failing android tests as well. I'm just going to mark disabled for all platforms as this doesn't seem to be os-specific. -- Matt, current clank build sheriff
,
Aug 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/40d72565df1f79563bb56dbfc51d10345ce0b598 commit 40d72565df1f79563bb56dbfc51d10345ce0b598 Author: Egor Pasko <pasko@chromium.org> Date: Mon Aug 29 10:59:27 2016 Disable testListSimpleBenchmarksDefinedInOneFileComplex. This is due to failures on AndroidTest. Coupled with existing failures on ChromeOS I suspect this is not a platform-specfic error and so should be disabled everywhere. See https://build.chromium.org/p/chromium.linux/builders/Android%20Tests/builds/31161 BUG=637938 TBR=mattcary@chromium.org,nednguyen@chromium.org patch from issue 2288143002 at patchset 1 (http://crrev.com/2288143002#ps1) (from mattcary@). patch from issue 2289633002 at patchset 1 (http://crrev.com/2289633002#ps1) Review URL: https://codereview.chromium.org/2291523002 . Cr-Commit-Position: refs/heads/master@{#414995} [modify] https://crrev.com/40d72565df1f79563bb56dbfc51d10345ce0b598/tools/perf/core/benchmark_finders_unittest.py
,
Dec 5 2016
Change title for this bug.
,
Dec 4 2017
,
Dec 4
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue. Sorry for the inconvenience if the bug really should have been left as Available. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Jan 16
,
Jan 16
|
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by achuith@chromium.org
, Aug 15 2016Owner: jamescook@chromium.org