servod: CPU console stuck in "flushing" state with servo_micro
Reported by
guillaum...@gmail.com,
Jul 19
|
||||
Issue descriptionUserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 Steps to reproduce the problem: 1. connect a gru-kevin with a servo_micro to a Linux PC running servod-tools 2. open the CPU console using `conserver` and `console` 3. after many reboots, the console gets stuck in a "flushing" state until the USB cable gets reconnected What is the expected behavior? The CPU console should keep on working, like with other devices using Servo v4. What went wrong? It's unclear what the root cause of the issue is, but it may be related to this issue with lost characters on the same set-up: https://bugs.chromium.org/p/chromium/issues/detail?id=865478 Did this work before? N/A Chrome version: Channel: n/a OS Version: Flash Version: Here are some LAVA jobs showing a working console, when it stops responding, and the next job stuck in "flushing" state: https://lava.collabora.co.uk/scheduler/job/1201015 https://lava.collabora.co.uk/scheduler/job/1201016 https://lava.collabora.co.uk/scheduler/job/1201017
,
Jul 20
guillaume.tucker@ Thanks for the issue. As this issue is out of scope of triaging at TE end, tentatively adding 'OS>Firmware>EC' and 'Internals>Core' components, 'TE-NeedsTriageHelp' label and requesting the appropriate team to look into the issue and help in further triaging. Thanks..
,
Jul 23
,
Jul 23
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734 commit a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734 Author: Mary Ruthven <mruthven@google.com> Date: Thu Jul 26 11:07:17 2018 Revert "servod: update pyusb to 1.0.2" This reverts commit 2c74796cd47d3d747baa2326a336c153cb933071. pyusb 1.0.2 fails under load and locks up the usb endpoint. BUG=b:111658037, chromium:865482 TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES}; done CQ-DEPEND=CL:1149509,CL:1149512 Change-Id: Ifa1b5f59f3e519318734ce67bae4c39b0c4c7a47 Signed-off-by: Mary Ruthven <mruthven@google.com> Reviewed-on: https://chromium-review.googlesource.com/1150863 Commit-Ready: Mary Ruthven <mruthven@chromium.org> Tested-by: Mary Ruthven <mruthven@chromium.org> Reviewed-by: Nick Sanders <nsanders@chromium.org> [modify] https://crrev.com/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734/servo/servo_postinit.py [modify] https://crrev.com/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734/servo/servo_server.py [modify] https://crrev.com/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734/servo/stm32usb.py
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/4acb45139aa2f001b7682b0b9181e77b23de418a commit 4acb45139aa2f001b7682b0b9181e77b23de418a Author: Nick Sanders <nsanders@chromium.org> Date: Thu Jul 26 11:07:15 2018 Revert "pyusb: upgraded package to upstream 1.0.2" This reverts commit f3ec249e1b315a98b648cc94c2cce03a0a1a80fc. pyusb 1.0.2 fails under load and locks up the usb endpoint. BUG=b:111658037, chromium:865482 TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES}; done CQ-DEPEND=CL:1149512 Change-Id: If90a06c5a11df55d95fe979eb5c6721e65df3f20 Reviewed-on: https://chromium-review.googlesource.com/1149509 Commit-Ready: Mary Ruthven <mruthven@chromium.org> Tested-by: Nick Sanders <nsanders@chromium.org> Reviewed-by: Mary Ruthven <mruthven@chromium.org> [add] https://crrev.com/4acb45139aa2f001b7682b0b9181e77b23de418a/metadata/md5-cache/dev-python/pyusb-1.0.0_beta1 [delete] https://crrev.com/7be412986d8e41c6f243a1c580b688d4e2057f80/dev-python/pyusb/pyusb-1.0.2.ebuild [modify] https://crrev.com/4acb45139aa2f001b7682b0b9181e77b23de418a/dev-python/pyusb/metadata.xml [modify] https://crrev.com/4acb45139aa2f001b7682b0b9181e77b23de418a/dev-python/pyusb/Manifest [add] https://crrev.com/4acb45139aa2f001b7682b0b9181e77b23de418a/dev-python/pyusb/pyusb-1.0.0_beta1.ebuild [delete] https://crrev.com/7be412986d8e41c6f243a1c580b688d4e2057f80/metadata/md5-cache/dev-python/pyusb-1.0.2
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/5ae6338b6a2666d1834998ddf56cfda7c1c53e84 commit 5ae6338b6a2666d1834998ddf56cfda7c1c53e84 Author: Nick Sanders <nsanders@chromium.org> Date: Thu Jul 26 11:07:15 2018 Revert "pyusb: add correct deps" This reverts commit 27d8955348f5424594e0e2bd0f1798cb794080af. pyusb 1.0.2 fails under load and locks up the usb endpoint. BUG=b:111658037, chromium:865482 TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES}; done CQ-DEPEND=CL:1149509 Change-Id: Icf5a0aa988ace556e7081e237a0cb1a651f15987 Reviewed-on: https://chromium-review.googlesource.com/1149512 Commit-Ready: Mary Ruthven <mruthven@chromium.org> Tested-by: Nick Sanders <nsanders@chromium.org> Reviewed-by: Mary Ruthven <mruthven@chromium.org> [modify] https://crrev.com/5ae6338b6a2666d1834998ddf56cfda7c1c53e84/dev-util/hdctools/hdctools-9999.ebuild [modify] https://crrev.com/5ae6338b6a2666d1834998ddf56cfda7c1c53e84/chromeos-base/ec-devutils/ec-devutils-9999.ebuild
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/ec/+/3793a83cbd7349b07453cc5a44674f9e7bc856f3 commit 3793a83cbd7349b07453cc5a44674f9e7bc856f3 Author: Nick Sanders <nsanders@chromium.org> Date: Thu Jul 26 11:07:16 2018 Revert "ec-devutils: update pyusb to 1.0.2" This reverts commit 7d26ae49bffcf269df82df7d0fee00e8c3341166. pyusb 1.0.2 fails under load and locks up the usb endpoint. BUG=b:111658037, chromium:865482 TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES}; done CQ-DEPEND=CL:1149509 Original change's description: > ec-devutils: update pyusb to 1.0.2 > > Pyusb interface has changed yet again. Patch calls to get_string > and usb.core.find. Modify ordering of detetch kernel driver > and avoid rereferencing endpoints. > > BRANCH=None > BUG=b:110907544 > TEST=emerge ec-devutils, run powerlog, usb_console, servo_updater > > CQ-DEPEND=CL:1119632 > > Signed-off-by: Nick Sanders <nsanders@chromium.org> > > Change-Id: I33df474ce3bb57d2e11c9f384ac013b51d210843 > Reviewed-on: https://chromium-review.googlesource.com/1125354 > Commit-Ready: Nick Sanders <nsanders@chromium.org> > Tested-by: Nick Sanders <nsanders@chromium.org> > Reviewed-by: Aseda Aboagye <aaboagye@chromium.org> Bug: b:110907544 Change-Id: I04f015b4924e7fee76ab0f1734267e8e2d14faaa Reviewed-on: https://chromium-review.googlesource.com/1150963 Commit-Ready: Nick Sanders <nsanders@chromium.org> Tested-by: Nick Sanders <nsanders@chromium.org> Reviewed-by: Aseda Aboagye <aaboagye@chromium.org> Reviewed-by: Nick Sanders <nsanders@chromium.org> [modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/tigertool/ecusb/stm32usb.py [modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/tigertool/ecusb/stm32uart.py [modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/usb_updater/fw_update.py [modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/usb_serial/console.py [modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/usb_power/powerlog.py
,
Jul 30
Many thanks, good to know you've reproduced the same issue. Howver, we're running servod-tools on Debian Stretch, which uses pyusb 1.0.0. I've tried with 1.0.0b1 which seems to be the version used by the patch mentioned above, but it didn't fix the issue: https://lava.collabora.co.uk/scheduler/job/1212543 I guess what I can do next is look for the revision of each project from the Chrome OS chroot and compare with the Debian packages. Doing this sounds like outside of the scope of this Chrome OS bug report though - which might get resolved now.
,
Aug 3
Is this using usb_serial_simple kernel driver or servod's /dev/pts/XX mechanism?
,
Aug 3
This is running servod-tools with a /dev/pts* device. A servod service is automatically started when the Servo board is detected based on a udev rule, and it takes over the ttyUSB device. [ 3.275334] usb 1-1.1.2.4: new high-speed USB device number 7 using xhci_hcd [ 3.384548] usb 1-1.1.2.4: New USB device found, idVendor=18d1, idProduct=5003 [ 3.384553] usb 1-1.1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 3.384555] usb 1-1.1.2.4: Product: servo:810-10010-02 [ 3.384557] usb 1-1.1.2.4: Manufacturer: Google Inc [ 3.384559] usb 1-1.1.2.4: SerialNumber: 905537-00574 Then servod gets started, and we end up with this: [ 5.268855] usb 1-1.1.1: usbfs: interface 0 claimed by usb_serial_simple while 'python' sets config #1 [ 5.270471] google ttyUSB0: google converter now disconnected from ttyUSB0 [ 5.270700] usb_serial_simple 1-1.1.1:1.0: device disconnected # ls /dev/google-servo/cbg-kevin/ -l total 0 lrwxrwxrwx 1 root root 10 Aug 3 09:00 cpu-uart -> /dev/pts/7 lrwxrwxrwx 1 root root 11 Aug 3 09:00 ec-uart -> /dev/pts/11 |
||||
►
Sign in to add a comment |
||||
Comment 1 by viswa.karala@chromium.org
, Jul 19