telemetry_UnitTests failure in x86-generic-telemetry |
||||||||||||||
Issue descriptionFailing build: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/11222 Failure snippet: 12/14 07:35:19.114 INFO |run_chromeos_tests:0052| [648/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputFormatNegativeFilter queued 12/14 07:35:19.115 INFO |run_chromeos_tests:0052| [649/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputFormatNegativeFilter passed 0.0158s: 12/14 07:35:19.115 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.115 INFO |run_chromeos_tests:0052| FAIL: add_1_and_2 (browser_tests.simple_numeric_test.SimpleTest) 12/14 07:35:19.116 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.116 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.116 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/serially_executed_browser_test_case.py", line 155, in <lambda> 12/14 07:35:19.117 INFO |run_chromeos_tests:0052| return lambda self: based_method(self, *args) 12/14 07:35:19.117 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/simple_numeric_test.py", line 57, in AdderTest 12/14 07:35:19.117 INFO |run_chromeos_tests:0052| self.assertEqual(a + b, partial_sum) 12/14 07:35:19.118 INFO |run_chromeos_tests:0052| AssertionError: 3 != 5 12/14 07:35:19.118 INFO |run_chromeos_tests:0052| 12/14 07:35:19.118 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.118 INFO |run_chromeos_tests:0052| FAIL: add_7_and_3 (browser_tests.simple_numeric_test.SimpleTest) 12/14 07:35:19.119 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.119 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.119 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/serially_executed_browser_test_case.py", line 155, in <lambda> 12/14 07:35:19.120 INFO |run_chromeos_tests:0052| return lambda self: based_method(self, *args) 12/14 07:35:19.120 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/simple_numeric_test.py", line 57, in AdderTest 12/14 07:35:19.120 INFO |run_chromeos_tests:0052| self.assertEqual(a + b, partial_sum) 12/14 07:35:19.121 INFO |run_chromeos_tests:0052| AssertionError: 10 != 5 12/14 07:35:19.121 INFO |run_chromeos_tests:0052| 12/14 07:35:19.121 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.121 INFO |run_chromeos_tests:0052| FAIL: multiplier_simple_2 (browser_tests.simple_numeric_test.SimpleTest) 12/14 07:35:19.122 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.122 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.122 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/serially_executed_browser_test_case.py", line 155, in <lambda> 12/14 07:35:19.123 INFO |run_chromeos_tests:0052| return lambda self: based_method(self, *args) 12/14 07:35:19.123 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/simple_numeric_test.py", line 69, in MultiplierTest 12/14 07:35:19.123 INFO |run_chromeos_tests:0052| self.assertEqual(a * b, partial_sum * self.extra) 12/14 07:35:19.123 INFO |run_chromeos_tests:0052| AssertionError: 6 != 25 12/14 07:35:19.124 INFO |run_chromeos_tests:0052| 12/14 07:35:19.124 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.124 INFO |run_chromeos_tests:0052| Ran 6 tests in 0.001s 12/14 07:35:19.125 INFO |run_chromeos_tests:0052| 12/14 07:35:19.125 INFO |run_chromeos_tests:0052| FAILED (failures=3) 12/14 07:35:19.642 INFO |run_chromeos_tests:0052| [649/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputFormatPositiveFilter queued 12/14 07:35:19.642 INFO |run_chromeos_tests:0052| [650/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputFormatPositiveFilter passed 0.5166s: 12/14 07:35:19.643 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.643 INFO |run_chromeos_tests:0052| ERROR: TestException (browser_tests.simple_numeric_test.SimpleTest) 12/14 07:35:19.644 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.644 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.644 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/simple_numeric_test.py", line 76, in TestException 12/14 07:35:19.645 INFO |run_chromeos_tests:0052| raise Exception('Expected exception') 12/14 07:35:19.645 INFO |run_chromeos_tests:0052| Exception: Expected exception 12/14 07:35:19.646 INFO |run_chromeos_tests:0052| 12/14 07:35:19.646 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.647 INFO |run_chromeos_tests:0052| FAIL: TestSimple (browser_tests.simple_numeric_test.SimpleTest) 12/14 07:35:19.647 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.648 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.648 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/simple_numeric_test.py", line 73, in TestSimple 12/14 07:35:19.648 INFO |run_chromeos_tests:0052| self.assertEqual(1, self.extra) 12/14 07:35:19.649 INFO |run_chromeos_tests:0052| AssertionError: 1 != 5 12/14 07:35:19.649 INFO |run_chromeos_tests:0052| 12/14 07:35:19.649 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.650 INFO |run_chromeos_tests:0052| Ran 2 tests in 0.501s 12/14 07:35:19.650 INFO |run_chromeos_tests:0052| 12/14 07:35:19.650 INFO |run_chromeos_tests:0052| FAILED (failures=1, errors=1) 12/14 07:35:19.668 INFO |run_chromeos_tests:0052| [650/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputWhenSetupClassFailed queued 12/14 07:35:19.669 INFO |run_chromeos_tests:0052| [651/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputWhenSetupClassFailed passed 0.0175s: 12/14 07:35:19.669 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.670 INFO |run_chromeos_tests:0052| ERROR: setUpClass (browser_tests.failed_tests.SetUpClassFailedTest) 12/14 07:35:19.670 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.670 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.671 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/failed_tests.py", line 15, in setUpClass 12/14 07:35:19.671 INFO |run_chromeos_tests:0052| raise Exception 12/14 07:35:19.671 INFO |run_chromeos_tests:0052| Exception 12/14 07:35:19.672 INFO |run_chromeos_tests:0052| 12/14 07:35:19.672 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.673 INFO |run_chromeos_tests:0052| Ran 0 tests in 0.001s 12/14 07:35:19.673 INFO |run_chromeos_tests:0052| 12/14 07:35:19.673 INFO |run_chromeos_tests:0052| FAILED (errors=1) 12/14 07:35:19.694 INFO |run_chromeos_tests:0052| [651/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputWhenTearDownClassFailed queued 12/14 07:35:19.695 INFO |run_chromeos_tests:0052| [652/1070] telemetry.testing.browser_test_runner_unittest.BrowserTestRunnerTest.testJsonOutputWhenTearDownClassFailed passed 0.0205s: 12/14 07:35:19.695 INFO |run_chromeos_tests:0052| ====================================================================== 12/14 07:35:19.696 INFO |run_chromeos_tests:0052| ERROR: tearDownClass (browser_tests.failed_tests.TearDownClassFailedTest) 12/14 07:35:19.696 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.696 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/14 07:35:19.697 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/examples/browser_tests/failed_tests.py", line 32, in tearDownClass 12/14 07:35:19.697 INFO |run_chromeos_tests:0052| raise Exception 12/14 07:35:19.697 INFO |run_chromeos_tests:0052| Exception 12/14 07:35:19.698 INFO |run_chromeos_tests:0052| 12/14 07:35:19.698 INFO |run_chromeos_tests:0052| ---------------------------------------------------------------------- 12/14 07:35:19.698 INFO |run_chromeos_tests:0052| Ran 100 tests in 0.003s 12/14 07:35:19.699 INFO |run_chromeos_tests:0052| 12/14 07:35:19.699 INFO |run_chromeos_tests:0052| FAILED (errors=1)
,
Dec 14 2016
https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/11219 also contains the former failure (just once) https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/11216 and https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/11220 also contains the latter failure (just once) https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/11217 contains 12/13 12:36:55.327 INFO |run_chromeos_tests:0052| [175/1070] telemetry.internal.actions.key_event_unittest.KeyPressActionTest.testPressEndAndHome failed unexpectedly 22.5508s: 12/13 12:36:55.328 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/13 12:36:55.328 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/browser_test_case.py", line 41, in WrappedMethod 12/13 12:36:55.328 INFO |run_chromeos_tests:0052| method(self) 12/13 12:36:55.329 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/actions/key_event_unittest.py", line 50, in testPressEndAndHome 12/13 12:36:55.329 INFO |run_chromeos_tests:0052| delta=20) 12/13 12:36:55.329 INFO |run_chromeos_tests:0052| AssertionError: 894 != 0 within 20 delta so we were just unlucky that that particular run had two consecutive failures. We should still try to see if it's possible to deflake this test.
,
Dec 14 2016
The latter test is flaky in most platforms: https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/extension_unittest.py?q=testWebviewInExtension&sq=package:chromium&l=171 maybe we should add Chrome OS to the list?
,
Dec 14 2016
Adding myself in case it's some cros-specific flakiness.
,
Dec 14 2016
No, that test seems to be flaky across the board: https://bugs.chromium.org/p/chromium/issues/detail?id=661434 https://bugs.chromium.org/p/chromium/issues/detail?id=508325
,
Dec 14 2016
Thank you!
,
Dec 20 2016
Could you please tell us in which subdirectory in cloud storage the stdout from the Telemetry tests is? I find it difficult to believe that these unit tests are flaky. They're pure Python and don't even bring up a browser. I think the flakes see in Issue 661434 are another problem.
,
Dec 20 2016
,
Dec 20 2016
From this recent failure: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-telemetry/builds/11261 With this cloud storage output: https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-generic-telemetry/R57-9103.0.0-b11261/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-2-telemetry_UnitTests?pli=1 Messages from the client and server can be found in this directory (debug/): https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-generic-telemetry/R57-9103.0.0-b11261/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-2-telemetry_UnitTests/debug/?pli=1 And messages from the test can be found here (telemetry_UnitTests.user/debug): https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/x86-generic-telemetry/R57-9103.0.0-b11261/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-2-telemetry_UnitTests/telemetry_UnitTests.user/debug/?pli=1 If you search for 'FAIL:' in telemetry_UnitTests.user.INFO from that last link, you will see the output from comment #0.
,
Dec 20 2016
Thanks. It's awful that it doesn't re-report the failed test. It's this one which actually failed: 12/19 14:27:38.801 INFO |run_chromeos_tests:0052| [337/1070] telemetry.internal.browser.extension_unittest.WebviewInExtensionTest.testWebviewInExtension queued 12/19 14:27:38.801 INFO |run_chromeos_tests:0052| [338/1070] telemetry.internal.browser.extension_unittest.WebviewInExtensionTest.testWebviewInExtension failed unexpectedly 21.1069s: 12/19 14:27:38.802 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 12/19 14:27:38.802 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/extension_unittest.py", line 180, in testWebviewInExtension 12/19 14:27:38.802 INFO |run_chromeos_tests:0052| self.assertEquals(1, len(webview_contexts)) 12/19 14:27:38.803 INFO |run_chromeos_tests:0052| AssertionError: 1 != 0 That test (telemetry.internal.browser.extension_unittest.WebviewInExtensionTest.testWebviewInExtension) needs to be disabled on Chrome OS as well. Ned, can you do that or should I?
,
Dec 20 2016
,
Dec 20 2016
Disable cl in https://codereview.chromium.org/2593813002. Reassign to Ben for triaging this.
,
Dec 20 2016
Let's just close this as fixed. Chrome OS folks, please reopen if Ned's CL didn't fix it. Ned, thanks for the fix.
,
Dec 21 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/19755d02f501ad5f213a2aeae174e277f940d9cd commit 19755d02f501ad5f213a2aeae174e277f940d9cd Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Wed Dec 21 00:34:13 2016 Roll src/third_party/catapult/ 28346a164..f3dc14e52 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/28346a164856..f3dc14e52fae $ git log 28346a164..f3dc14e52 --date=short --no-merges --format='%ad %ae %s' 2016-12-20 nednguyen [Telemetry] Disable testWebviewInExtension on chromeos BUG= 674220 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2593843002 Cr-Commit-Position: refs/heads/master@{#439950} [modify] https://crrev.com/19755d02f501ad5f213a2aeae174e277f940d9cd/DEPS
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Aug 3 2017
Closing. Please reopen it if its not fixed. Thanks! |
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by lhchavez@chromium.org
, Dec 14 2016