telemetry_UnitTests failing on chrome os |
||||||||||
Issue descriptionFailing builder: https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-telemetry/builds/8973 Also failing on x86-generic. Snippet from client.0.DEBUG: 05/24 02:17:58.486 INFO |run_chromeos_tests:0052| [143/1027] telemetry.internal.actions.action_runner_unittest.ActionRunnerTest.testWaitForElement failed unexpectedly 27.1144s: 05/24 02:17:58.487 INFO |run_chromeos_tests:0052| Traceback (most recent call last): 05/24 02:17:58.487 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/tab_test_case.py", line 23, in setUp 05/24 02:17:58.488 INFO |run_chromeos_tests:0052| self._tab = self._browser.tabs[0] 05/24 02:17:58.489 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/tab_list.py", line 18, in __getitem__ 05/24 02:17:58.489 INFO |run_chromeos_tests:0052| return self._tab_list_backend.__getitem__(index) 05/24 02:17:58.490 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 64, in __getitem__ 05/24 02:17:58.490 INFO |run_chromeos_tests:0052| return self.GetBackendFromContextId(context_id) 05/24 02:17:58.491 INFO |run_chromeos_tests:0052| File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 78, in GetBackendFromContextId 05/24 02:17:58.491 INFO |run_chromeos_tests:0052| raise e 05/24 02:17:58.492 INFO |run_chromeos_tests:0052| DevtoolsTargetCrashException: Devtools target crashed 05/24 02:17:58.493 INFO |run_chromeos_tests:0052| ******************************************************************************** 05/24 02:17:58.493 INFO |run_chromeos_tests:0052| (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:341 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed. 05/24 02:17:58.494 INFO |run_chromeos_tests:0052| ******************************************************************************** 05/24 02:17:58.494 INFO |run_chromeos_tests:0052| (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:342 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:60880/devtools/page/039aa536-6488-43bd-98b1-12645f4a2b9f 05/24 02:17:58.495 INFO |run_chromeos_tests:0052| ******************************************************************************** 05/24 02:17:58.496 INFO |run_chromeos_tests:0052| (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py:110 _HandleDevToolsConnectionError) The browser exists and can be reached. The devtools target probably crashed. 05/24 02:17:58.496 INFO |run_chromeos_tests:0052| Found Minidump: False 05/24 02:17:58.497 INFO |run_chromeos_tests:0052| Stack Trace: 05/24 02:17:58.497 INFO |run_chromeos_tests:0052| ******************************************************************************** 05/24 02:17:58.498 INFO |run_chromeos_tests:0052| Cannot get stack trace on CrOS 05/24 02:17:58.498 INFO |run_chromeos_tests:0052| ******************************************************************************** 05/24 02:17:58.499 INFO |run_chromeos_tests:0052| Standard output: 05/24 02:17:58.500 INFO |run_chromeos_tests:0052| ******************************************************************************** 05/24 02:17:58.500 INFO |run_chromeos_tests:0052| Cannot get standard output on CrOS
,
May 24 2016
,
May 24 2016
,
May 24 2016
Going to attempt a bisect: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/telemetry/builds/0 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/telemetry/builds/1
,
May 24 2016
,
May 24 2016
There are no catapult rolls in that failing build (in comment 6), so this is probably a breakage in chrome. The mojo change (and perhaps the webview change), look suspicious. We have 23 failing telemetry tests: https://pantheon.corp.google.com/m/cloudstorage/b/chromeos-image-archive/o/amd64-generic-telemetry/R53-8365.0.0-b8973/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-1-telemetry_UnitTests/telemetry_UnitTests.user/debug/telemetry_UnitTests.user.DEBUG 5 failing perf tests: https://pantheon.corp.google.com/m/cloudstorage/b/chromeos-image-archive/o/amd64-generic-telemetry/R53-8365.0.0-b8973/vm_test_results_1/test_harness/all/SimpleTestVerify/1_autotest_tests/results-2-telemetry_UnitTests_perf/telemetry_UnitTests.perf/debug/telemetry_UnitTests.perf.DEBUG
,
May 24 2016
I don't think that's https://codereview.chromium.org/1997293002 unless some code parses console output produced from postMessage.
,
May 24 2016
I am working on reproducing locally now. The error looks to be "The devtools target probably crashed" so that CL caught my attention. Hopefully we will be able to bisect it quickly.
,
May 25 2016
I can't reproduce this locally, but cbuildbot is succeeding / failing as expected for first two builds, will start the next six now.
,
May 25 2016
I doubt the Mojo change is at fault. It only changes behavior on windows. And if it was broken, _everything_ would be failing. Also, the change has been reverted (due to Dr memory failures).
,
May 25 2016
Yup, the mojo change is not in the amd64 blame list, so it's not the culprit.
,
May 25 2016
I queued up a bisect but the tryserevrs were drafted for release builds so the bisect has not started yet :(
,
May 25 2016
(I did conform however that #395534 passes and #395540 failed, so a chrome change is definitely the culprit)
,
May 25 2016
This is affecting AFDO (profile based optimizations) since the telemetry test to captuire the profile is not working. see autoserv.DEBUG at https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/64535807-chromeos-test/chromeos2-row3-rack9-host11/debug/
,
May 25 2016
ATTN: changwan@ This is the failing commit: cbd74f1402d7f3d87b032ea9ee1d4d65f6fcde8c https://codereview.chromium.org/2001983002 I have no idea how/why this is affecting the telemetry tests. I would like to revert the CL because I have no idea how to debug it, but am re-assigning it first in case someone has any thoughts. Success: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/telemetry/builds/0 Fail: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/telemetry/builds/9
,
May 25 2016
No way my cl has to do with this. it's only affecting android, and not chromeos. it may just mean flakiness of the test.
,
May 26 2016
I do agree that it is surprising, but I don't know what all the telemetry / catapult system does so I wasn't sure. I went ahead and ran the test again @ a1a73208fade64c6326c70df5d93bb08753c583d (the previous commit and the first commit in the failing run) and this time it does in fact appear to be failing: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/telemetry/builds/12 So there does in fact appear to be some element of flakiness or other factor, sigh. I will try to dig a bit deeper tomorrow.
,
May 26 2016
This started passing again, so either it is very flakey, or was the side effect of another bug that got fixed (perhaps the one referenced in comment #15). I will keep an eye on it but if the builders continue to pass I will just be thankful and mark this closed.
,
May 26 2016
,
May 26 2016
,
May 26 2016
Last 3 builds on both telemetry builders passed, so marking WontFix.
,
May 26 2016
This is unlikely to just be flake since there were a large number of independent tests that were failing. We run well over 1000 tests with each test run so even minor flakes would cause consistent failures.
,
May 26 2016
Then the difference must have been the chromeos versions. The only real difference between the outputs of try run 0 and run 12 was: trybot-x86-generic-telemetry/R53-8368.0.0-b0 vs. trybot-x86-generic-telemetry/R53-8371.0.0-b12 (The slaves were also different but that seems less likely to be the difference) |
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by steve...@chromium.org
, May 24 2016Labels: -Pri-3 OS-Chrome Pri-1