> 2017-05-19T20:22:42.161067+00:00 NOTICE check_ethernet.hook[4463]: Attempting recovery method "reload_usb_ethernet_devices"

This happens because the DUT believed that it had no working network
interface.  When that happens, it tries a series of local steps designed
to restart the network, in an attempt to get back online.  IIRC, this
handling doesn't kick in until at least 10 minutes after boot.

4 failures out of 24 in the last 72, but we have some known infra issues which I think probably caused a few of those.  Full dut-status for those 72 hours included.

Deleted: chromeos2-row8-rack8-host8-dut-status.txt 27.2 KB

chromeos2-row8-rack8-host8-dut-status.txt 27.2 KB View Download

> 4 failures out of 24 in the last 72, but we have
> some known infra issues which I think probably caused
> a few of those.  Full dut-status for those 72 hours included.

I went through all provision failures for kevin in the bvt and cq
pools in the last 24 hours.  Except for the single failure on
chromeos2-row8-rack8-host8 cited here, all of the failures were
attributable to the ongoing RPC problems in bug 723645.

Looking at the failure, I see that provision failed because the
DUT was offline at the outset.  The provision task was able to
gather the crashinfo because the DUT rebooted during provisioning
in time to report the data.  The reboot happened (as noted earlier)
because the DUT knew it was offline, and was trying to get back
online.

I don't know why the DUT went offline in the first place.

> Looking at the failure, I see that provision failed because the
> DUT was offline at the outset.

Actually, no, it seems more complicated than that.  The provisioning
code retried.  The first attempt failed because the DUT never got
online after updating; that triggered the retry.  The second attempt
failed because the DUT was still offline at the start.  Logs were
gathered from the DUT because it successfully got itself back online
before the failure handling went looking for the data.

I went through the logs again, looking at the history.  I see an
anomaly.

On the first boot, you see messages like this:
    2017-05-19T20:10:56.606673+00:00 INFO kernel: [    2.530674] smsc95xx 5-1.2.1:1.0 eth0: register 'smsc95xx' at usb-xhci-hcd.2.auto-1.2.1, smsc95xx USB 2.0 Ethernet, 94:eb:2c:15:88:14

The message occurs multiple times, probably because 'check_ethernet.hook'
was busy trying to restart the network.

Looking at the logs after reboot that succeeded, you see these messages:
    2017-05-19T13:32:01.966059-07:00 INFO kernel: [    2.294875] cdc_ether 6-1.4:2.0 eth0: register 'cdc_ether' at usb-xhci-hcd.2.auto-1.4, CDC Ethernet Device, 14:91:82:3b:ba:b6
    2017-05-19T13:32:02.242153-07:00 INFO kernel: [    2.571642] smsc95xx 5-1.2.1:1.0 eth1: register 'smsc95xx' at usb-xhci-hcd.2.auto-1.2.1, smsc95xx USB 2.0 Ethernet, 94:eb:2c:15:88:14

The 'smsc95xx' device is the ethernet interface from the servo V3.  It's not
connected to anything, thus, the DUT couldn't get online.

The 'cdc_ether' device is the ethernet USB dongle that's actually connected
to the network, but it wasn't detected on the first boot.

We've got a similar complaint about cdc_ether devices plugged into samus,
see bug 724454.

I just talked with Harpreet@ and confirmed:
o smsc9512/14 is builtin to servo v3 (which started rolling out in 2015)
o servo v3 in most cases will be replaced with v4 in the next quarter or two.

I've tried to look for any other cases of smcs95xx driver/device causing link problems but too many hits for "smsc95xx" in our bug tracker.

> o smsc9512/14 is builtin to servo v3 (which started rolling out in 2015)

The problem has nothing to do with the smsc network interface
on the servo V3; that interface isn't plugged into anything,
isn't supposed to be used, and isn't expected to work.

The problem is with the cdc_ether network interface on the USB
dongle.  That dongle wasn't detected on the first boot, which
eventually caused the DUT to reboot.  On reboot, the DUT _did_
detect the network interface, and everything worked fine.

So, the problem is 'cdc_ether' device not detected on USB.
... which sounds a lot like bug 724454.

Richard - understood - I think you are correct and one of the bugs should be closed as a dup. Which ever one stays open, I'd like to add some "color". >:|