Build failure: Android PFQ failure due to adb connection error |
||||
Issue descriptionLast known good pfq ARC build is 4854484. First known failure build is 4856391. Only three CL's included in ARC builds. Link to build or pfq page. https://uberchromegw.corp.google.com/i/chromeos/builders/master-nyc-android-pfq/builds/2479 build # for that buildbot. 2479 Snippet of log that contains the failure. Autotest instance created: cautotest-prod 06-21-2018 [22:32:25] Created suite job: http://cautotest-prod/afe/#tab_id=view_job&object_id=210677751 @@@STEP_LINK@Link to suite@http://cautotest-prod/afe/#tab_id=view_job&object_id=210677751@@@ The suite job has another 3:39:46.962955 till timeout. 06-21-2018 [23:03:59] Suite job is finished. 06-21-2018 [23:03:59] Start collecting test results and dump them to json. Suite job [ PASSED ] cheets_CTS_N.7.1_r18.x86.CtsDramTestCases [ INFO ] cheets_CTS_N.7.1_r18.x86.CtsDramTestCases TEST_NA: Skipping: test not supported on this board/pool. cheets_CTS_N.7.1_r18.x86.CtsAppUsageHostTestCases [ INFO ] cheets_CTS_N.7.1_r18.x86.CtsAppUsageHostTestCases TEST_NA: Skipping: test not supported on this board/pool. cheets_CTS_N.7.1_r18.x86.CtsAccountManagerTestCases [ INFO ] cheets_CTS_N.7.1_r18.x86.CtsAccountManagerTestCases TEST_NA: Skipping: test not supported on this board/pool. cheets_CTS_N.7.1_r18.x86.CtsAccelerationTestCases [ INFO ] cheets_CTS_N.7.1_r18.x86.CtsAccelerationTestCases TEST_NA: Skipping: test not supported on this board/pool. cheets_GTS.GtsAdminTestCases [ FAILED ] cheets_GTS.GtsAdminTestCases FAIL: Error: Failed to set up adb connection cheets_StartAndroid.stress [ PASSED ] cheets_CTS_N.CtsAccountManagerTestCases [ FAILED ] cheets_CTS_N.CtsAccountManagerTestCases FAIL: Error: Failed to set up adb connection cheets_CTS_N.CtsDramTestCases [ FAILED ] cheets_CTS_N.CtsDramTestCases FAIL: Error: Failed to set up adb connection cheets_CTS_N.CtsAppUsageHostTestCases [ FAILED ] cheets_CTS_N.CtsAppUsageHostTestCases FAIL: Error: Failed to set up adb connection cheets_ContainerSmokeTest [ FAILED ] cheets_ContainerSmokeTest FAIL: adb is not ready in 60 seconds. cheets_LoginScreen [ PASSED ] cheets_DownloadsFilesystem [ FAILED ] cheets_DownloadsFilesystem FAIL: adb is not ready in 60 seconds. cheets_ClobberStateful [ PASSED ] cheets_Midis [ PASSED ] security_NetworkListeners [ PASSED ] cheets_AndroidToChromeIntents [ FAILED ] cheets_AndroidToChromeIntents FAIL: adb is not ready in 60 seconds. cheets_MediaPlayerVideoHWDecodeUsed [ FAILED ] cheets_MediaPlayerVideoHWDecodeUsed FAIL: adb is not ready in 60 seconds. cheets_FileSystemPermissions [ FAILED ] cheets_FileSystemPermissions FAIL: adb is not ready in 60 seconds. telemetry_LoginTest.arc [ PASSED ] cheets_SELinuxTest [ FAILED ] cheets_SELinuxTest FAIL: adb is not ready in 60 seconds. cheets_KeyboardTest [ FAILED ] cheets_KeyboardTest FAIL: adb is not ready in 60 seconds. cheets_SettingsBridge [ FAILED ] cheets_SettingsBridge FAIL: adb is not ready in 60 seconds. graphics_Idle.arc [ PASSED ] cheets_FileSystemXattrs [ FAILED ] cheets_FileSystemXattrs FAIL: adb is not ready in 60 seconds. cheets_RemovableMedia [ FAILED ] cheets_RemovableMedia FAIL: adb is not ready in 60 seconds. cheets_ContainerMount [ FAILED ] cheets_ContainerMount FAIL: adb is not ready in 60 seconds. cheets_SysctlTest [ FAILED ] cheets_SysctlTest FAIL: adb is not ready in 60 seconds. desktopui_ExitOnSupervisedUserCrash.arc [ PASSED ] cheets_MountObbTest [ PASSED ] cheets_CameraOrientation [ FAILED ] cheets_CameraOrientation FAIL: adb is not ready in 60 seconds. cheets_ClipboardTest [ FAILED ] cheets_ClipboardTest FAIL: adb is not ready in 60 seconds. cheets_GTS.GtsPlacementTestCases [ FAILED ] cheets_GTS.GtsPlacementTestCases FAIL: Error: Failed to set up adb connection cheets_GTS.GtsPlacementTestCases retry_count: 1 cheets_CTS_N.CtsAccelerationTestCases [ FAILED ] cheets_CTS_N.CtsAccelerationTestCases FAIL: Error: Failed to set up adb connection cheets_CTS_N.CtsAccelerationTestCases retry_count: 1 cheets_NotificationTest [ FAILED ] cheets_NotificationTest FAIL: adb is not ready in 60 seconds. cheets_NotificationTest retry_count: 1 cheets_CleanShutDown [ FAILED ] cheets_CleanShutDown FAIL: adb is not ready in 60 seconds. cheets_CleanShutDown retry_count: 1 cheets_GTS.GtsNetTestCases [ FAILED ] cheets_GTS.GtsNetTestCases FAIL: Error: Failed to set up adb connection cheets_GTS.GtsNetTestCases retry_count: 1 Suite timings: Downloads started at 2018-06-21 22:32:21 Payload downloads ended at 2018-06-21 22:32:23 Suite started at 2018-06-21 22:32:38 Artifact downloads ended (at latest) at 2018-06-21 22:32:42 Testing started at 2018-06-21 22:32:44 Testing ended at 2018-06-21 23:01:36 Links to test logs: Suite job http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677751-chromeos-test/ cheets_CTS_N.7.1_r18.x86.CtsDramTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677751-chromeos-test/ cheets_CTS_N.7.1_r18.x86.CtsAppUsageHostTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677751-chromeos-test/ cheets_CTS_N.7.1_r18.x86.CtsAccountManagerTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677751-chromeos-test/ cheets_CTS_N.7.1_r18.x86.CtsAccelerationTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677751-chromeos-test/ cheets_GTS.GtsAdminTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677754-chromeos-test/ cheets_StartAndroid.stress http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677755-chromeos-test/ cheets_CTS_N.CtsAccountManagerTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677756-chromeos-test/ cheets_CTS_N.CtsDramTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677757-chromeos-test/ cheets_CTS_N.CtsAppUsageHostTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677759-chromeos-test/ cheets_ContainerSmokeTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677760-chromeos-test/ cheets_LoginScreen http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677761-chromeos-test/ cheets_DownloadsFilesystem http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677764-chromeos-test/ cheets_ClobberStateful http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677765-chromeos-test/ cheets_Midis http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677766-chromeos-test/ security_NetworkListeners http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677767-chromeos-test/ cheets_AndroidToChromeIntents http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677768-chromeos-test/ cheets_MediaPlayerVideoHWDecodeUsed http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677769-chromeos-test/ cheets_FileSystemPermissions http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677770-chromeos-test/ telemetry_LoginTest.arc http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677771-chromeos-test/ cheets_SELinuxTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677772-chromeos-test/ cheets_KeyboardTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677773-chromeos-test/ cheets_SettingsBridge http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677774-chromeos-test/ graphics_Idle.arc http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677775-chromeos-test/ cheets_FileSystemXattrs http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677776-chromeos-test/ cheets_RemovableMedia http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677777-chromeos-test/ cheets_ContainerMount http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677778-chromeos-test/ cheets_SysctlTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677779-chromeos-test/ desktopui_ExitOnSupervisedUserCrash.arc http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677780-chromeos-test/ cheets_MountObbTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677781-chromeos-test/ cheets_CameraOrientation http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677783-chromeos-test/ cheets_ClipboardTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210677784-chromeos-test/ cheets_GTS.GtsPlacementTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210678836-chromeos-test/ cheets_CTS_N.CtsAccelerationTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210679720-chromeos-test/ cheets_NotificationTest http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210679721-chromeos-test/ cheets_CleanShutDown http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210679722-chromeos-test/ cheets_GTS.GtsNetTestCases http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/210679996-chromeos-test/
,
Jun 22 2018
Included ARC changes looks nothing related to the error. The changes are: Display ag/4396080 WiFi Network ag/4012569 Pipe Reader ag/4399101 App Reinstall ag/4174872
,
Jun 22 2018
veyron_* and samus, they are android PFQs of 3.14 kernel boards failed.
,
Jun 22 2018
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1108562 This 4.4 change is suspicious to whitelist a syscall for adbd, maybe we need this change for 3.14 too ?
,
Jun 22 2018
around adbd and authentication, there's bunch of sshd logs. Something changed with sslh / adb signature ? 2018-06-22T05:49:35.972430+00:00 INFO sshd[12790]: Did not receive identification string from 127.0.0.1 port 45078 2018-06-22T05:49:39.359030+00:00 INFO sshd[12836]: Did not receive identification string from 127.0.0.1 port 45155 2018-06-22T05:49:42.773012+00:00 INFO sshd[12851]: Did not receive identification string from 127.0.0.1 port 45232 2018-06-22T05:49:46.176744+00:00 INFO sshd[12875]: Did not receive identification string from 127.0.0.1 port 45309 2018-06-22T05:49:49.564592+00:00 INFO sshd[12898]: Did not receive identification string from 127.0.0.1 port 45386 2018-06-22T05:49:52.989431+00:00 INFO sshd[12913]: Did not receive identification string from 127.0.0.1 port 45463 2018-06-22T05:49:56.375981+00:00 INFO sshd[12942]: Did not receive identification string from 127.0.0.1 port 45541 2018-06-22T05:49:57.973541+00:00 DEBUG kernel: [ 623.094948] SELinux: initialized (dev proc, type proc), uses genfs_contexts 2018-06-22T05:49:59.800682+00:00 INFO sshd[12972]: Did not receive identification string from 127.0.0.1 port 45618 2018-06-22T05:50:03.200357+00:00 INFO sshd[12988]: Did not receive identification string from 127.0.0.1 port 45695 2018-06-22T05:50:06.585067+00:00 INFO sshd[13014]: Did not receive identification string from 127.0.0.1 port 45772 2018-06-22T05:50:10.006016+00:00 INFO sshd[13029]: Did not receive identification string from 127.0.0.1 port 45849 2018-06-22T05:50:13.446523+00:00 INFO sshd[13044]: Did not receive identification string from 127.0.0.1 port 45926 2018-06-22T05:50:16.053473+00:00 DEBUG kernel: [ 641.176802] SELinux: initialized (dev proc, type proc), uses genfs_contexts 2018-06-22T05:50:16.825443+00:00 INFO sshd[13077]: Did not receive identification string from 127.0.0.1 port 46003 2018-06-22T05:50:20.250460+00:00 INFO sshd[13096]: Did not receive identification string from 127.0.0.1 port 46080 2018-06-22T05:50:23.669498+00:00 INFO sshd[13111]: Did not receive identification string from 127.0.0.1 port 46157
,
Jun 22 2018
I think this change from lhchavez changes the behavior of adb authentication https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1108246
,
Jun 22 2018
Not only PFQ but the release image of R69-10806.0.0 looks already hitting this failure: https://stainless.corp.google.com/search?status=GOOD&status=WARN&status=FAIL&status=ERROR&exclude_retried=true&exclude_cts=false&exclude_non_production=true&exclude_acts=true&exclude_non_release=true&exclude_au=true&test=cheets_CTS_N.7.1_r18.arm&exclude_not_run=false&row=build&col=test&view=matrix&first_date=2018-06-08&last_date=2018-06-22 Looking at this table, https://stainless.corp.google.com/search?view=list&first_date=2018-06-08&last_date=2018-06-22&test=%5Echeets_CTS_N%5C.7%5C.1_r18%5C.arm%5C.CtsAccelerationTestCases%24&build=%5ER69%5C-10806%5C.0%5C.0%24&status=GOOD&status=WARN&status=FAIL&status=ERROR&status=ABORT&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true precisely the 3.14 boards looks dying. 4.4, and 3.18 (e.g., Cyan) are ok
,
Jun 22 2018
The changes in comment #4 and comment #6 are unrelated. they only apply to adb-over-usb, and those devices do not use it: They don't have a /etc/arc/adbd.json or have the magic crossystem flag.
,
Jun 22 2018
It appears that for some reason, the network in the container is all messed up: # android-sh -c 'ifconfig arcb0' ifconfig: arc0: No such device And of course that causes adb to be not able to connect because it goes through that interface: sslh-fork v1.18 started forward to adb failed:connect: No route to host connect: No route to host
,
Jun 22 2018
Found it! https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1094840 Those modules are not present in 3.14.
,
Jun 22 2018
Fix is out: https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1112211/1/arc/network/init/arc-network.conf And with that, a rant: I *seriously* need to remove the string 'adb' from that error message. We tend to place too much blame into it and leads into wild goose chases every time :(
,
Jun 22 2018
,
Jun 22 2018
Also decided to fix the thing I complained about in the previous rant: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1112356
,
Jun 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/dc0265a1d77ac00dcc722f17809d5862f46fd011 commit dc0265a1d77ac00dcc722f17809d5862f46fd011 Author: Luis Hector Chavez <lhchavez@google.com> Date: Sat Jun 23 00:22:14 2018 arc: Be a bit more careful with loading modules on old kernels This change loads the nf_reject_ipv6 kernel module while on the 3.14 kernel. This makes it possible for arc-networkd to start again. BUG= chromium:855363 TEST=android-sh -c 'ifconfig arc0' TEST=adb shell # works again! Change-Id: I34706890c3a91611007399e395c8055cbf6257db Reviewed-on: https://chromium-review.googlesource.com/1112211 Commit-Ready: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Luis Hector Chavez <lhchavez@chromium.org> Reviewed-by: Yury Khmel <khmel@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/dc0265a1d77ac00dcc722f17809d5862f46fd011/arc/network/init/arc-network.conf
,
Jun 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4443ced11fa81f378534b0c3efeb8b6819f08481 commit 4443ced11fa81f378534b0c3efeb8b6819f08481 Author: Luis Hector Chavez <lhchavez@google.com> Date: Sat Jun 23 03:19:36 2018 arc: Try to better assign blame to adb connection failures This change tries to go through a laundry list of usual suspects before blaming adb for failing to connect. This makes debugging way easier since the granularity of 'adb not ready' is way too coarse. BUG= chromium:855363 TEST=cheets_ContainerSmokeTest # Failed with adbd TCP port is not reachable. Expecting # _is_tcp_port_reachable(_ADBD_ADDRESS) == True, seeing False Change-Id: I75ec3c799cb8ee5a2f942aa1b34393e02edf6d7d Reviewed-on: https://chromium-review.googlesource.com/1112356 Commit-Ready: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Luis Hector Chavez <lhchavez@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Victor Hsieh <victorhsieh@chromium.org> [modify] https://crrev.com/4443ced11fa81f378534b0c3efeb8b6819f08481/client/common_lib/cros/arc.py
,
Jun 23 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by yunfanc@chromium.org
, Jun 22 2018