New issue
Advanced search Search tips

Issue 865482 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

servod: CPU console stuck in "flushing" state with servo_micro

Reported by guillaum...@gmail.com, Jul 19

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36

Steps to reproduce the problem:
1. connect a gru-kevin with a servo_micro to a Linux PC running servod-tools
2. open the CPU console using `conserver` and `console`
3. after many reboots, the console gets stuck in a "flushing" state until the USB cable gets reconnected

What is the expected behavior?
The CPU console should keep on working, like with other devices using Servo v4.

What went wrong?
It's unclear what the root cause of the issue is, but it may be related to this issue with lost characters on the same set-up:
https://bugs.chromium.org/p/chromium/issues/detail?id=865478

Did this work before? N/A 

Chrome version:   Channel: n/a
OS Version: 
Flash Version: 

Here are some LAVA jobs showing a working console, when it stops responding, and the next job stuck in "flushing" state:

https://lava.collabora.co.uk/scheduler/job/1201015
https://lava.collabora.co.uk/scheduler/job/1201016
https://lava.collabora.co.uk/scheduler/job/1201017
 
Labels: Needs-Triage-M67
Components: OS>Firmware>EC Internals>Core
Labels: Triaged-ET TE-NeedsTriageHelp
guillaume.tucker@ Thanks for the issue.

As this issue is out of scope of triaging at TE end, tentatively adding 'OS>Firmware>EC' and 'Internals>Core' components, 'TE-NeedsTriageHelp' label and requesting the appropriate team to look into the issue and help in further triaging.

Thanks..
Components: -Internals>Core
Labels: -OS-Linux -TE-NeedsTriageHelp -Needs-Triage-M67 OS-Chrome
Owner: nsanders@chromium.org
Status: Assigned (was: Unconfirmed)
Cc: nsanders@chromium.org
Project Member

Comment 5 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734

commit a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734
Author: Mary Ruthven <mruthven@google.com>
Date: Thu Jul 26 11:07:17 2018

Revert "servod: update pyusb to 1.0.2"

This reverts commit 2c74796cd47d3d747baa2326a336c153cb933071.

pyusb 1.0.2 fails under load and locks up the usb endpoint.

BUG=b:111658037, chromium:865482
TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES};
done
CQ-DEPEND=CL:1149509,CL:1149512

Change-Id: Ifa1b5f59f3e519318734ce67bae4c39b0c4c7a47
Signed-off-by: Mary Ruthven <mruthven@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1150863
Commit-Ready: Mary Ruthven <mruthven@chromium.org>
Tested-by: Mary Ruthven <mruthven@chromium.org>
Reviewed-by: Nick Sanders <nsanders@chromium.org>

[modify] https://crrev.com/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734/servo/servo_postinit.py
[modify] https://crrev.com/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734/servo/servo_server.py
[modify] https://crrev.com/a8dcf914ad2f5e4b8d21a67feb5cd98874ab2734/servo/stm32usb.py

Project Member

Comment 6 by bugdroid1@chromium.org, Jul 26

Project Member

Comment 7 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/5ae6338b6a2666d1834998ddf56cfda7c1c53e84

commit 5ae6338b6a2666d1834998ddf56cfda7c1c53e84
Author: Nick Sanders <nsanders@chromium.org>
Date: Thu Jul 26 11:07:15 2018

Revert "pyusb: add correct deps"

This reverts commit 27d8955348f5424594e0e2bd0f1798cb794080af.

pyusb 1.0.2 fails under load and locks up the usb endpoint.

BUG=b:111658037, chromium:865482
TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES}; done

CQ-DEPEND=CL:1149509

Change-Id: Icf5a0aa988ace556e7081e237a0cb1a651f15987
Reviewed-on: https://chromium-review.googlesource.com/1149512
Commit-Ready: Mary Ruthven <mruthven@chromium.org>
Tested-by: Nick Sanders <nsanders@chromium.org>
Reviewed-by: Mary Ruthven <mruthven@chromium.org>

[modify] https://crrev.com/5ae6338b6a2666d1834998ddf56cfda7c1c53e84/dev-util/hdctools/hdctools-9999.ebuild
[modify] https://crrev.com/5ae6338b6a2666d1834998ddf56cfda7c1c53e84/chromeos-base/ec-devutils/ec-devutils-9999.ebuild

Project Member

Comment 8 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/3793a83cbd7349b07453cc5a44674f9e7bc856f3

commit 3793a83cbd7349b07453cc5a44674f9e7bc856f3
Author: Nick Sanders <nsanders@chromium.org>
Date: Thu Jul 26 11:07:16 2018

Revert "ec-devutils: update pyusb to 1.0.2"

This reverts commit 7d26ae49bffcf269df82df7d0fee00e8c3341166.

pyusb 1.0.2 fails under load and locks up the usb endpoint.

BUG=b:111658037, chromium:865482
TEST=while RES=$(dut-control ec_uart_en spi1_buf_en ); do echo ${RES}; done

CQ-DEPEND=CL:1149509

Original change's description:
> ec-devutils: update pyusb to 1.0.2
>
> Pyusb interface has changed yet again. Patch calls to get_string
> and usb.core.find. Modify ordering of detetch kernel driver
> and avoid rereferencing endpoints.
>
> BRANCH=None
> BUG=b:110907544
> TEST=emerge ec-devutils, run powerlog, usb_console, servo_updater
>
> CQ-DEPEND=CL:1119632
>
> Signed-off-by: Nick Sanders <nsanders@chromium.org>
>
> Change-Id: I33df474ce3bb57d2e11c9f384ac013b51d210843
> Reviewed-on: https://chromium-review.googlesource.com/1125354
> Commit-Ready: Nick Sanders <nsanders@chromium.org>
> Tested-by: Nick Sanders <nsanders@chromium.org>
> Reviewed-by: Aseda Aboagye <aaboagye@chromium.org>

Bug: b:110907544
Change-Id: I04f015b4924e7fee76ab0f1734267e8e2d14faaa
Reviewed-on: https://chromium-review.googlesource.com/1150963
Commit-Ready: Nick Sanders <nsanders@chromium.org>
Tested-by: Nick Sanders <nsanders@chromium.org>
Reviewed-by: Aseda Aboagye <aaboagye@chromium.org>
Reviewed-by: Nick Sanders <nsanders@chromium.org>

[modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/tigertool/ecusb/stm32usb.py
[modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/tigertool/ecusb/stm32uart.py
[modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/usb_updater/fw_update.py
[modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/usb_serial/console.py
[modify] https://crrev.com/3793a83cbd7349b07453cc5a44674f9e7bc856f3/extra/usb_power/powerlog.py

Many thanks, good to know you've reproduced the same issue.
Howver, we're running servod-tools on Debian Stretch, which uses
pyusb 1.0.0.  I've tried with 1.0.0b1 which seems to be the
version used by the patch mentioned above, but it didn't fix the
issue:

  https://lava.collabora.co.uk/scheduler/job/1212543

I guess what I can do next is look for the revision of each
project from the Chrome OS chroot and compare with the Debian
packages.  Doing this sounds like outside of the scope of this
Chrome OS bug report though - which might get resolved now.
Is this using usb_serial_simple kernel driver or servod's /dev/pts/XX mechanism?
This is running servod-tools with a /dev/pts* device.  A servod
service is automatically started when the Servo board is detected
based on a udev rule, and it takes over the ttyUSB device.

[    3.275334] usb 1-1.1.2.4: new high-speed USB device number 7 using xhci_hcd
[    3.384548] usb 1-1.1.2.4: New USB device found, idVendor=18d1, idProduct=5003
[    3.384553] usb 1-1.1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.384555] usb 1-1.1.2.4: Product: servo:810-10010-02
[    3.384557] usb 1-1.1.2.4: Manufacturer: Google Inc
[    3.384559] usb 1-1.1.2.4: SerialNumber: 905537-00574

Then servod gets started, and we end up with this:

[    5.268855] usb 1-1.1.1: usbfs: interface 0 claimed by usb_serial_simple while 'python' sets config #1
[    5.270471] google ttyUSB0: google converter now disconnected from ttyUSB0
[    5.270700] usb_serial_simple 1-1.1.1:1.0: device disconnected

# ls /dev/google-servo/cbg-kevin/ -l
total 0
lrwxrwxrwx 1 root root 10 Aug  3 09:00 cpu-uart -> /dev/pts/7
lrwxrwxrwx 1 root root 11 Aug  3 09:00 ec-uart -> /dev/pts/11

Sign in to add a comment