New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 665144 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Nov 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

servo v4: restart servod core dumps

Project Member Reported by dchan@chromium.org, Nov 14 2016

Issue description


Steps:

- see go/servov4testing for setup
- Start sudo servod --board=terra -s 1234-6666 --host 0.0.0.0
- ctrl-c 
- start servod again will failed. See attach for full debug log

You might need to try restart a few times.
The only way to get out of this is to restart v4.


	servod.txt 
54.1 KB   View   Download

#1 nsanders@google.com
It's failing at:
serial = usb.util.get_string(servo_micro, 256, servo_micro.iSerialNumber)


from partner bug https://code.google.com/p/chrome-os-partner/issues/detail?id=56650
 

Comment 1 by dchan@chromium.org, Nov 14 2016

example of carsh log

$ sudo servod --board=orco
2016-11-14 13:14:55,410 - servod - INFO - Start
2016-11-14 13:14:56,518 - servod - INFO - Found servo, vid: 0x18d1 pid: 0x501a sid: 
2016-11-14 13:14:56,521 - servod - INFO - Found servo, vid: 0x18d1 pid: 0x501b sid: ND00033
2016-11-14 13:14:56,521 - servod - INFO - 
2016-11-14 13:14:57,522 - servod - INFO - Press '0' for servo, vid: 0x18d1 pid: 0x501a sid: 
2016-11-14 13:14:57,525 - servod - INFO - Press '1' for servo, vid: 0x18d1 pid: 0x501b sid: ND00033
1
2016-11-14 13:14:58,798 - root - INFO - 
2016-11-14 13:14:58,800 - root - INFO - Chose 1 ... starting servod on servo vid: 0x18d1 pid: 0x501b sid: ND00033
2016-11-14 13:14:58,800 - root - INFO - 
2016-11-14 13:14:58,803 - servod - WARNING - Servo device's iserial was unrecognized.
2016-11-14 13:14:58,803 - servod - INFO - Found XML overlay for board orco
2016-11-14 13:14:58,804 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_v4.xml
2016-11-14 13:14:58,812 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/common.xml
2016-11-14 13:14:58,818 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_v4_inas.xml
2016-11-14 13:14:58,822 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/ina2xx.xml
2016-11-14 13:14:58,831 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_orco_overlay.xml
2016-11-14 13:14:58,832 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_rambi_overlay.xml
2016-11-14 13:14:58,832 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servoflex_v2_r0_p50.xml
2016-11-14 13:14:58,833 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/x86_ec_common.xml
2016-11-14 13:14:58,835 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/ec_common.xml
2016-11-14 13:14:58,844 - Servod - INFO - Initializing interface 21 to stm32_gpio
2016-11-14 13:14:58,844 - Servod - INFO - Sgpio: interface: 1
2016-11-14 13:14:58,847 - Servod - INFO - Initializing interface 22 to stm32_uart
2016-11-14 13:14:58,847 - Servod - INFO - Suart: interface: {'interface': 0, 'index': 21, 'name': 'stm32_uart'}
2016-11-14 13:14:58,851 - Servod - INFO - /dev/pts/17
2016-11-14 13:14:58,851 - Servod - INFO - Initializing interface 23 to stm32_i2c
2016-11-14 13:14:58,851 - Servod - INFO - Si2cBus: interface: {'interface': 2, 'index': 22, 'name': 'stm32_i2c'}
2016-11-14 13:14:58,856 - Servod - INFO - Initializing interface 24 to stm32_uart
2016-11-14 13:14:58,857 - Servod - INFO - Suart: interface: {'interface': 3, 'index': 23, 'name': 'stm32_uart'}
2016-11-14 13:14:58,863 - Servod - INFO - /dev/pts/31
2016-11-14 13:14:58,863 - Servod - INFO - Initializing interface 25 to stm32_uart
2016-11-14 13:14:58,863 - Servod - INFO - Suart: interface: {'interface': 4, 'index': 24, 'name': 'stm32_uart'}
2016-11-14 13:14:58,869 - Servod - INFO - /dev/pts/32
2016-11-14 13:14:58,869 - Servod - INFO - Initializing interface 26 to ec3po_uart
2016-11-14 13:14:58,882 - EC3PO Interface - INFO - /dev/pts/33
2016-11-14 13:14:58,886 - EC3PO.Console - INFO - /dev/pts/33 - EC Console is being served on /dev/pts/33.
Bus 003 Device 089: ID 18d1:501a Google Inc. 
Bus 003 Device 088: ID 18d1:501b Google Inc. 
Bus 003 Device 087: ID 04b4:6572 Cypress Semiconductor Corp. 
Bus 003 Device 003: ID 1050:0211 Yubico.com Gnubby
Bus 002 Device 004: ID 046d:c01e Logitech, Inc. MX518 Optical Mouse
Bus 002 Device 003: ID 0510:0032 Sejin Electron, Inc. 
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 03f0:2514 Hewlett-Packard 4-port hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
2016-11-14 13:14:59,009 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_micro.xml
2016-11-14 13:14:59,013 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/common.xml
2016-11-14 13:14:59,025 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_v4.xml
2016-11-14 13:14:59,028 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/common.xml.
2016-11-14 13:14:59,028 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_v4_inas.xml
2016-11-14 13:14:59,030 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/ina2xx.xml
2016-11-14 13:14:59,040 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/common.xml.
2016-11-14 13:14:59,040 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/servo_v4_inas.xml.
2016-11-14 13:14:59,040 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/ina2xx.xml.
2016-11-14 13:14:59,041 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_orco_overlay.xml
2016-11-14 13:14:59,041 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servo_rambi_overlay.xml
2016-11-14 13:14:59,041 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/servoflex_v2_r0_p50.xml
2016-11-14 13:14:59,042 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/x86_ec_common.xml
2016-11-14 13:14:59,044 - SystemConfig - INFO - Loading XML config /usr/lib64/python2.7/site-packages/servo/data/ec_common.xml
2016-11-14 13:14:59,047 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/servo_rambi_overlay.xml.
2016-11-14 13:14:59,047 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/servoflex_v2_r0_p50.xml.
2016-11-14 13:14:59,047 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/x86_ec_common.xml.
2016-11-14 13:14:59,047 - SystemConfig - WARNING - Already sourced system file /usr/lib64/python2.7/site-packages/servo/data/ec_common.xml.
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/servod", line 9, in <module>
    load_entry_point('servo===0.0.1-4fa5014', 'console_scripts', 'servod')()
  File "/usr/lib64/python2.7/site-packages/servo/servod.py", line 515, in main
    main_function()
  File "/usr/lib64/python2.7/site-packages/servo/servod.py", line 504, in main_function
    usbkm232=options.usbkm232)
  File "/usr/lib64/python2.7/site-packages/servo/servo_server.py", line 184, in __init__
    servo_postinit.post_init(self)
  File "/usr/lib64/python2.7/site-packages/servo/servo_postinit.py", line 255, in post_init
    post_init_class(servod).post_init()
  File "/usr/lib64/python2.7/site-packages/servo/servo_postinit.py", line 239, in post_init
    self.init_servo_micro(servo_micro)
  File "/usr/lib64/python2.7/site-packages/servo/servo_postinit.py", line 209, in init_servo_micro
    serial = usb.util.get_string(servo_micro, 256, servo_micro.iSerialNumber)
  File "/usr/lib64/python2.7/site-packages/usb/util.py", line 248, in get_string
    0
  File "/usr/lib64/python2.7/site-packages/usb/control.py", line 169, in get_descriptor
    data_or_wLength = desc_size
  File "/usr/lib64/python2.7/site-packages/usb/core.py", line 711, in ctrl_transfer
    self.__get_timeout(timeout)
  File "/usr/lib64/python2.7/site-packages/usb/backend/libusb1.py", line 836, in ctrl_transfer
    timeout))
  File "/usr/lib64/python2.7/site-packages/usb/backend/libusb1.py", line 571, in _check
    raise USBError(_str_error[ret], ret, _libusb_errno[ret])
usb.core.USBError: [Errno 110] Operation timed out
Segmentation fault (core dumped)

Can you check if you already have servod running on that device? It seems likely that servo micro is being used by another program.

Can you also try unplugging servo micro and plugging it in again rather than unplugging servo v4?

Is this readily reproducible?

Comment 3 by dchan@chromium.org, Nov 15 2016

Not 100%, i will keep an eye on it.
Normally I disconnect the USB to the servo to reset. I did do a ps and did not see another servo running.  I did ctrl-C to exit, may be that left it in a un-clean state.

Comment 4 by dchan@chromium.org, Nov 15 2016

It just happen again. I disconnect the USB to the micro and reconnect. 
The servod restarted without issue.
Can you run "ps -ef | grep servod" after ctrl-c next time this happens? ctrl-c is a supported way to exit servod.

Comment 6 by dchan@chromium.org, Nov 15 2016

There is no servod running after ctrl-C to exit servod.
Here is the repo steps

1- sudo servod --board=orco
2- screen /dev/pts/41 115200 (the last EC Console from output for #1)
3- Ctrl-A : quit to exit screen
4- ctrl-C to exit servod
5- run #1 again

result is the core dumps.  Somehow even though I close screen, the terminal is still held open ?

The only way to reset is to disconnec the USB connected to the micro. 
I can't repro this on chell. What is an orco, and if it repros reliably can I come by and see it fail? 

Comment 8 by dchan@google.com, Nov 18 2016

orco is just another chromebook

After you exit servod and restart, you might need to repeat a few times.  Once it get into the stat where it failed, only way is to reset micro card.

I can bring over the orco.
Cc: xiaoshi@google.com elerner@chromium.org
Tried with dchan@'s chromebook and servos, but was unable to repro a failure. My best guess is that either newer servod's have a fix and repo sync may fix it, or that the servo_micro is browning out due to a long cable and low-power port. 

Danny will try on his system with a direct cable to his workstation, and also with an updated hdctools. If it's a power problem, unfortunately there's not much to be done 

Comment 10 by dchan@google.com, Nov 19 2016

Status: WontFix (was: Assigned)
looks like this is indeed the USB cable problem. I swap with a  A to micro and seems to work. If I swap back to the A to mini to micro, i can repo the problem.

Sign in to add a comment