Hey Haowei,

Could you list the platforms affected and which ones consistently fail to boot from USB vs those that are flaky?  So far Pyro is the only one I'm aware of that consistently does not boot from USB.

Let's keep this as a top-level bug and create child bugs to focus on debug for each platform affected.

Here is the list

https://docs.google.com/spreadsheets/d/1nf_p1Q0eJX47p0KLh7ajn7FwoU5Tc_UVl9fXxEytvGI/edit#gid=0

Anything marked with "worked with 4 port hub" means this setup.

Labstation -> Servo V4 (usb attached) -> 4 port hub -> DUT. 

4 scenarios: 

Servo Host(workstation or Guado) -> Servo V4 (USB attached) -> DUT  Servod running
Servo Host(workstation or Guado) -> Servo V4 (USB attached) -> DUT  No Servod running
Servo Host(workstation or Guado) -> Servo V4 (USB atached) -> Powered USB2,0 HUB -> DUT  Servod running
Servo Host(workstation or Guado) -> Servo V4 (USB atached) -> Powered USB2,0 HUB -> DUT  No Servod running
Testing above cases, scenario #1 is the only case DUT cannot boot from USB attached on Servo V4. 

Seems like this two controls is not been set or working properly to Servo V4 during usb boot.

Can we force them set at initialization, or servo_v4.xml?

~$ dut-control usb3_mux_sel:dut_sees_usbkey
~$ dut-control usb3_pwr_en:on

usb3_mux_sel is aliased to usb_mux_sel1
usb3_pwr_en is aliased to prtctl4_pwren

so I think we're ok on that front.

Thanks for the summary in #7! I was wondering about:
Servo Host(workstation or Guado) -> Servo V4 (USB attached) -> DUT  No Servod running

Do you kill servod and then set the DUT into recovery mode? Could you dump all the controls from scenario #1?

The controls I listed are from scenario #1. 

2017-05-02 15:39:18,279 - Servod - INFO - Initializing interface 2 to stm32_uart
2017-05-02 15:39:18,279 - Servod - INFO - Suart: interface: {'interface': 0, 'index': 1, 'name': 'stm32_uart'}
2017-05-02 15:39:18,281 - Servod - INFO - /dev/pts/4
2017-05-02 15:39:18,281 - Servod - INFO - Initializing interface 3 to stm32_uart
2017-05-02 15:39:18,281 - Servod - INFO - Suart: interface: {'interface': 3, 'index': 2, 'name': 'stm32_uart'}
2017-05-02 15:39:18,283 - Servod - INFO - /dev/pts/5
2017-05-02 15:39:18,284 - Servod - INFO - Initializing interface 4 to stm32_i2c
2017-05-02 15:39:18,284 - Servod - INFO - Si2cBus: interface: {'interface': 4, 'index': 3, 'name': 'stm32_i2c'}
2017-05-02 15:39:18,286 - Servod - INFO - Initializing interface 5 to stm32_uart
2017-05-02 15:39:18,286 - Servod - INFO - Suart: interface: {'interface': 5, 'index': 4, 'name': 'stm32_uart'}
2017-05-02 15:39:18,289 - Servod - INFO - /dev/pts/6
2017-05-02 15:39:18,289 - Servod - INFO - Initializing interface 6 to stm32_uart
2017-05-02 15:39:18,289 - Servod - INFO - Suart: interface: {'interface': 6, 'index': 5, 'name': 'stm32_uart'}
2017-05-02 15:39:18,292 - Servod - INFO - /dev/pts/7
2017-05-02 15:39:18,292 - Servod - INFO - Initializing interface 7 to ec3po_uart
2017-05-02 15:39:18,299 - EC3PO Interface - INFO - /dev/pts/8
2017-05-02 15:39:18,300 - Servod - INFO - Initializing interface 9 to ec3po_uart
2017-05-02 15:39:18,304 - EC3PO.Console - INFO - /dev/pts/8 - EC Console is being served on /dev/pts/8.
2017-05-02 15:39:18,305 - EC3PO Interface - INFO - /dev/pts/9
2017-05-02 15:39:18,306 - Servod - INFO - Initializing interface 10 to ec3po_uart
2017-05-02 15:39:18,309 - EC3PO.Console - INFO - /dev/pts/9 - EC Console is being served on /dev/pts/9.
2017-05-02 15:39:18,311 - EC3PO Interface - INFO - /dev/pts/10
2017-05-02 15:39:18,314 - EC3PO.Console - INFO - /dev/pts/10 - EC Console is being served on /dev/pts/10.
2017-05-02 15:39:18,423 - Servod - INFO - Initialized uart1_en to on
2017-05-02 15:39:18,579 - Servod - INFO - Initialized servo_jtag_trst to off
2017-05-02 15:39:18,725 - Servod - INFO - Initialized spi1_buf_en to off
2017-05-02 15:39:18,858 - Servod - INFO - Initialized spi2_buf_en to off
2017-05-02 15:39:19,012 - Servod - INFO - Initialized uart2_en to on
2017-05-02 15:39:19,268 - Servod - INFO - Initialized spi1_vref to off
2017-05-02 15:39:19,563 - Servod - INFO - Initialized spi2_vref to off
2017-05-02 15:39:19,734 - Servod - INFO - Initialized servo_jtag_trst_dir to off
2017-05-02 15:39:19,886 - Servod - INFO - Initialized servo_jtag_tdi_dir to off
2017-05-02 15:39:20,025 - Servod - INFO - Initialized gpio_expander_reset to off
2017-05-02 15:39:20,163 - Servod - INFO - Initialized uart3_on_spi1 to off
2017-05-02 15:39:20,332 - Servod - INFO - Initialized SERVO_JTAG_TMS_DIR to off
2017-05-02 15:39:20,469 - Servod - INFO - Initialized SERVO_JTAG_TDO_SEL to off
2017-05-02 15:39:20,629 - Servod - INFO - Initialized SERVO_JTAG_TMS to off
2017-05-02 15:39:20,776 - Servod - INFO - Initialized UART3_TX_SERVO_JTAG_TCK to off
2017-05-02 15:39:20,958 - Servod - INFO - Initialized UART3_RX_JTAG_BUFFER_TO_SERVO_TDO to off
2017-05-02 15:39:21,116 - Servod - INFO - Initialized SERVO_JTAG_TDO_BUFFER_EN to off
2017-05-02 15:39:21,245 - Servod - INFO - Initialized SERVO_JTAG_TDI to off
2017-05-02 15:39:21,398 - Servod - INFO - Initialized SERVO_JTAG_RTCK to off
2017-05-02 15:39:21,562 - Servod - INFO - Initialized JTAG_BUFOUT_EN_L to off
2017-05-02 15:39:21,705 - Servod - INFO - Initialized JTAG_BUFIN_EN_L to off
2017-05-02 15:39:21,810 - Servod - ERROR - Problem initializing fw_wp_en -> off :: [Errno 110] Operation timed out
2017-05-02 15:39:21,913 - Servod - ERROR - Problem initializing tp1 -> off :: [Errno 110] Operation timed out
2017-05-02 15:39:21,915 - Servod - INFO - Initialized cold_reset to off
2017-05-02 15:39:21,918 - Servod - INFO - Initialized pwr_button to release
2017-05-02 15:39:21,921 - Servod - INFO - Initialized warm_reset to off
2017-05-02 15:39:21,926 - Servod - INFO - Initialized rec_mode to off
2017-05-02 15:39:21,930 - Servod - INFO - Initialized fw_up to off
2017-05-02 15:39:22,580 - Servod - INFO - Initialized lid_open to yes
2017-05-02 15:39:22,586 - Servod - INFO - Initialized pch_disable to off
2017-05-02 15:39:22,589 - Servod - INFO - Initialized tp2 to off
2017-05-02 15:39:22,590 - Servod - INFO - Initialized servo_ec3po_interp_connect to on
2017-05-02 15:39:22,590 - Servod - INFO - Initialized ec_ec3po_interp_connect to on
2017-05-02 15:39:22,590 - Servod - INFO - Initialized usbpd_ec3po_interp_connect to on
2017-05-02 15:39:22,595 - Servod - INFO - Initialized dp_hpd to off
2017-05-02 15:39:22,598 - Servod - INFO - Initialized host_hub_reset to off
2017-05-02 15:39:22,602 - Servod - INFO - Initialized fastboot_duthub_mux_sel to host
2017-05-02 15:39:22,604 - Servod - INFO - Initialized dut_usb_mux_enable to on
2017-05-02 15:39:22,607 - Servod - INFO - Initialized dut_hub_usb_reset to off
2017-05-02 15:39:22,609 - Servod - INFO - Initialized at_hwb to off
2017-05-02 15:39:22,611 - Servod - INFO - Initialized cmux_en to on
2017-05-02 15:39:22,613 - Servod - INFO - Initialized sd_en to off
2017-05-02 15:39:22,616 - Servod - INFO - Initialized sd_pwr_en to off
2017-05-02 15:39:22,719 - Servod - ERROR - Problem initializing sbu_uart_sel -> ccd :: [Errno 110] Operation timed out
2017-05-02 15:39:22,821 - Servod - ERROR - Problem initializing atmega_rst -> on :: [Errno 110] Operation timed out
2017-05-02 15:39:22,825 - Servod - INFO - Initialized host_sd_usb_mux_sel to usb
2017-05-02 15:39:22,828 - Servod - INFO - Initialized uservo_pwr_en to on
2017-05-02 15:39:22,830 - Servod - INFO - Initialized uservo_fastboot_mux_sel to uservo
2017-05-02 15:39:22,832 - Servod - INFO - Initialized host_sd_usb_mux_en to on
2017-05-02 15:39:22,835 - Servod - INFO - Initialized sd_mux_sel to servo_sees_usbkey
2017-05-02 15:39:22,835 - Servod - INFO - Initialized servo_v4_ec3po_interp_connect to on

Here is the dut-control content after init servod. 

DUT_CC1_PR3A0:off
DUT_CC1_RA:off
DUT_CC1_RD:off
DUT_CC1_RP1A5:off
DUT_CC1_RPUSB:off
DUT_CC2_PR3A0:on
DUT_CC2_RA:on
DUT_CC2_RD:on
DUT_CC2_RP1A5:on
DUT_CC2_RPUSB:on
JTAG_BUFIN_EN_L:off
JTAG_BUFOUT_EN_L:off
SERVO_JTAG_RTCK:off
SERVO_JTAG_TDI:off
SERVO_JTAG_TDO_BUFFER_EN:off
SERVO_JTAG_TDO_SEL:off
SERVO_JTAG_TMS:off
SERVO_JTAG_TMS_DIR:off
UART3_RX_JTAG_BUFFER_TO_SERVO_TDO:off
UART3_TX_SERVO_JTAG_TCK:off
USB_DET_PP_CHG:off
USB_DET_PP_DUT:off
at_hwb:off
atmega_pty:/dev/pts/24
atmega_rst:on
cmux_en:on
cold_reset:on
cpu_temp:ERR
cpu_uart_capture:False
cpu_uart_cmd:None
cpu_uart_pty:/dev/pts/28
cpu_uart_regexp:None
cpu_uart_stream:''
cpu_uart_timeout:3
dev_mode:off
dongle_detect:off
dp_hpd:off
dut_charge_enable:off
dut_charge_source:typec
dut_hub_usb_reset:off
dut_is_host_or_device:host
dut_pty:/dev/pts/23
dut_usb_mux_enable:on
ec3po_ec_console:/dev/pts/36
ec3po_servo_console:/dev/pts/33
ec3po_servo_v4_console:/dev/pts/25
ec3po_usbpd_console:/dev/pts/34
ec_chip:lm4
ec_ec3po_interp_connect:on
ec_uart_capture:False
ec_uart_cmd:None
ec_uart_multicmd:ERR
ec_uart_pty:/dev/pts/36
ec_uart_regexp:None
ec_uart_stream:''
ec_uart_timeout:3
fan_actual_rpm:ERR
fan_duty:ERR
fan_target_rpm:ERR
fastboot_duthub_mux_sel:host
fw_up:on
fw_wp:on
fw_wp_en:off
fw_wp_state:on
fw_wp_vref:off
gpio_expander_reset:off
host_hub_reset:off
host_sd_usb_mux_en:on
host_sd_usb_mux_sel:usb
hpd:off
jtag_buf_on_flex_en:ERR
kbd_en:off
kbd_m1_a0:on
kbd_m1_a1:on
kbd_m2_a0:on
kbd_m2_a1:on
lid_open:yes
loglevel:info
pch_disable:on
power_state:ERR
ppchg5_alrt_reg:0x0
ppchg5_busv_reg:0x0
ppchg5_cal_reg:0x7fff
ppchg5_cfg_reg:0x4127
ppchg5_cur_reg:0x0
ppchg5_ma:0.0
ppchg5_msken_reg:0x8
ppchg5_mv:0.0
ppchg5_mw:0.0
ppchg5_pwr_reg:0x0
ppchg5_shuntmv:-0.0025
ppchg5_shv_reg:0x1
ppdut5_alrt_reg:0x0
ppdut5_busv_reg:0x1
ppdut5_cal_reg:0x7fff
ppdut5_cfg_reg:0x4127
ppdut5_cur_reg:0x0
ppdut5_ma:0.0
ppdut5_msken_reg:0x8
ppdut5_mv:1.25
ppdut5_mw:0.0
ppdut5_pwr_reg:0x0
ppdut5_shuntmv:-0.005
ppdut5_shv_reg:0x0
ppvar_vbat_ma:ERR
ppvar_vbat_mv:ERR
ppvar_vbat_mw:ERR
proc_hot:on
prtctl4_pwren:off
pwr_button:release
pwr_button_hold:ERR
raw_ec_uart_pty:/dev/pts/32
raw_servo_console_pty:/dev/pts/27
raw_servo_v4_console_pty:/dev/pts/14
raw_usbpd_uart_pty:/dev/pts/26
rec_mode:on
sbu_flip_sel:off
sbu_mux_enable:off
sbu_uart_sel:ccd
sd_en:off
sd_mux_sel:servo_sees_usbkey
sd_pwr_en:off
serialname:C1612060273
servo_console_pty:/dev/pts/33
servo_ec3po_interp_connect:on
servo_jtag_tdi_dir:off
servo_jtag_trst:off
servo_jtag_trst_dir:off
servo_v4_ccd_mode:ERR
servo_v4_console_pty:/dev/pts/25
servo_v4_dts_mode:ERR
servo_v4_ec3po_interp_connect:on
sleep:ERR
spi1_buf_en:off
spi1_buf_on_flex_en:ERR
spi1_vref:off
spi2_buf_en:off
spi2_buf_on_flex_en:ERR
spi2_vref:off
spi_hold:on
t20_rec:on
tp1:off
tp2:off
uart1_enable:on
uart1_pty:/dev/pts/32
uart2_enable:on
uart2_pty:/dev/pts/28
uart3_on_spi1:off
uart3_pty:/dev/pts/26
usb3_mux_en:on
usb3_mux_sel:servo_sees_usbkey
usb3_pwr_en:off
usb3_pwr_fault:off
usb_mux_oe1:on
usb_mux_sel1:servo_sees_usbkey
usbpd_chip:unknown
usbpd_ec3po_interp_connect:on
usbpd_uart_pty:/dev/pts/34
usbpd_uart_routing:ERR
uservo_fastboot_mux_sel:uservo
uservo_pwr_en:on
uservo_pwr_fault:off
warm_reset:on

So just for more details:

Haowei's testing from #7 was on an orco with the servo connected to his workstation.

And when he manually set the controls usb3_mux_sel:dut_sees_usbkey and usb3_pwr_en:on, the dut was able to boot from the usb in setup #1 (no usb hub + servod running).

I think we should focus on the pyro since that seems like it consistently does not boot in situation #1/2 (unsure about #3/4.  I'll grab a pyro from cros distros and repro at my desk.

FYI I was able to boot from USB through servo on the system kevin dropped off. It's possible that this is a USB3 marginality. I can make a usb-2 only fw for you to flash on servo, would you be willing to try it out?

Yes, I would like to try all the possibilities. 

From the other reef issue, it seems that some combinations of usb stick + servo v4 + dut port fail under USB3, but not usb2. 

It's easy to turn on usb3 in software, but it will slow down network access as well.

Another issue is that servod / autotest doesn't currently have the capability to update servo_v4 firmware, so some infra work would be needed to deploy a USB2 only firmware.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/b8de3538addc125e15ee9dd2ae25d51d1331d626

commit b8de3538addc125e15ee9dd2ae25d51d1331d626
Author: Nick Sanders <nsanders@chromium.org>
Date: Fri Jun 23 23:10:07 2017

servo: servo_v4: add dut_usb3_en control

dut_usb3_en enables or disables the usb3 interface
passthrough on the Parade PS8742 mux.

Note that dut_hub_usb_reset should be cycled after
in order for the USB link to be reinitialized.

BUG= chromium:718075 
TEST=Check that USB3 is passed through or not.

Change-Id: I9a669a21263606b69f207f809413eb4a29c9fa3c
Reviewed-on: https://chromium-review.googlesource.com/517352
Commit-Ready: Nick Sanders <nsanders@chromium.org>
Tested-by: Nick Sanders <nsanders@chromium.org>
Reviewed-by: Alexandru M Stan <amstan@chromium.org>

[add] https://crrev.com/b8de3538addc125e15ee9dd2ae25d51d1331d626/servo/drv/ps8742.py
[modify] https://crrev.com/b8de3538addc125e15ee9dd2ae25d51d1331d626/servo/drv/__init__.py
[modify] https://crrev.com/b8de3538addc125e15ee9dd2ae25d51d1331d626/servo/data/servo_v4.xml

ChromeOS Infra P1 Bugscrub.

P1 Bugs in this component should be important enough to get weekly status updates.

Is this already fixed?  -> Fixed
Is this no longer relevant? -> Archived or WontFix
Is this not a P1, based on go/chromeos-infra-bug-slo rubric? -> lower priority.
Is this a Feature Request rather than a bug? Type -> Feature
Is this missing important information or scope needed to decide how to proceed? -> Ask question on bug, possibly reassign.
Does this bug have the wrong owner? -> reassign.

Bugs that remain in this state next week will be downgraded to P2.

Can you check if disabling USB3 fixes the issue? 

I'm likely not the right person to own this considering I don't have or ever used a servo v4...

Aviv does anyone on your team have one at their desk?

kevcheng@ has been working on this.

He just left on paternity leave....

Anyone on your team familiar with autotest + servod?

+jrbarnette you got a servo v4?

Nope, I don't have a servo V4.  Maybe kevcheng@ has one lying about on
his desk?

I have one you can have at my office in b1667.

Chromestop may also stock them, not sure.

I do have a labstation + servo v4 on my workstation up for grabs.

The main debug is happening on crbug.com/716922 (on pyro/reef) and I believe Haowei is best to try it out since this is a bit inconsistent in reproducibility and he has access to multiple duts that can reproduce it.

I have a vague memory though that Haowei tried out disabling usb3 and it did help but Haowei can confirm.

Haowei is OOO until 7/26/17

Back on track of this bug. I haven't tried USB-2 only firmware. Can anyone provide it so I can test to see if the issue can be resolved. I would also like to test how bad the network can go. 

You can try this by starting servod and running: 
dut-control dut_usb3_en:off.

Actually it looks like it should init to off, and USB3 shouldn't be enabled at all.

I tried to turn off dut_usb3_en but the command is not working somehow. 

haoweiw@haoweiw:~/chromiumos$ dut-control dut_usb3_en:off
haoweiw@haoweiw:~/chromiumos$ dut-control dut_usb3_en
dut_usb3_en:on

Any updates on this bug? 

$ dut-control dut_usb3_en
dut_usb3_en:off
$ dut-control dut_usb3_en:on
$ dut-control dut_usb3_en
dut_usb3_en:on
$ dut-control dut_usb3_en:off
$ dut-control dut_usb3_en
dut_usb3_en:off


This works fine for me. Can you try again, and ensure you have the latest servod? If it still fails please run servod with "-d" and open another bug with the servod output after "dut-control dut_usb3_en:off"

the command is working now. The DUT is still not able to boot from USB attached on Servo V4 (type-c). Is this related to crbug/732519?

crbug/732519 is a PD issue. Please don't use a type-c servo for this bug. 

This bug concerns Servo V4 Type-A, are you still able to repro with a type-A servo?

Seems like I'm not able to reproduce this on a type-a Servo V4.

Can you mark fixed if you think it's fixed w/ latest servod? Or is there still more validation to do?

I'm going to mark this as fix now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/80ef3f073e87cfba80666c922111b4c922769279

commit 80ef3f073e87cfba80666c922111b4c922769279
Author: Nick Sanders <nsanders@chromium.org>
Date: Fri Nov 17 05:07:39 2017

servo_v4: disable USB3 interface

This change switches the SS lines off in the main USB mux.
There has been some general flakyness regarding USB3 peripherals
and this change might address it.

BUG= chromium:718075 
BRANCH=None
TEST=lsusb -t indicates 5000M before, 480M after.

Change-Id: Id201fb20dc6489c4a071cb1c9c0624d7aa54652d
Reviewed-on: https://chromium-review.googlesource.com/509130
Commit-Ready: Nick Sanders <nsanders@chromium.org>
Tested-by: Nick Sanders <nsanders@chromium.org>
Reviewed-by: Shawn N <shawnn@chromium.org>

[modify] https://crrev.com/80ef3f073e87cfba80666c922111b4c922769279/board/servo_v4/board.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/ec/+/8c9f2e505d0f6b0a15052052b779dee93ad2936b

commit 8c9f2e505d0f6b0a15052052b779dee93ad2936b
Author: Nick Sanders <nsanders@chromium.org>
Date: Mon Nov 27 19:22:30 2017

servo_v4: disable USB3 interface

This change switches the SS lines off in the main USB mux.
There has been some general flakyness regarding USB3 peripherals
and this change might address it.

BUG= chromium:718075 
BRANCH=None
TEST=lsusb -t indicates 5000M before, 480M after.

Change-Id: Id201fb20dc6489c4a071cb1c9c0624d7aa54652d
Reviewed-on: https://chromium-review.googlesource.com/509130
Commit-Ready: Nick Sanders <nsanders@chromium.org>
Tested-by: Nick Sanders <nsanders@chromium.org>
Reviewed-by: Shawn N <shawnn@chromium.org>
(cherry picked from commit 80ef3f073e87cfba80666c922111b4c922769279)
Reviewed-on: https://chromium-review.googlesource.com/791451
Reviewed-by: Nick Sanders <nsanders@chromium.org>
Commit-Queue: Nick Sanders <nsanders@chromium.org>

[modify] https://crrev.com/8c9f2e505d0f6b0a15052052b779dee93ad2936b/board/servo_v4/board.c