New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 597297 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

network not working for chromeos1-row1-rack9-host3 (nyan)

Project Member Reported by shchen@chromium.org, Mar 23 2016

Issue description

checked on DUT in lab and network cable was not connected.  Attached it and DUT still won't go online.
 

Comment 1 by dchan@google.com, Mar 23 2016

Status: Assigned (was: Untriaged)

Comment 2 by dchan@google.com, Mar 23 2016

reflash OS and firmware, for some reason it can't get IP from the lab or if it did, it's not pingable, other device connect is ok.  Even more puzzling the device works find on my desk.  still checking.

Comment 4 by dchan@google.com, Mar 24 2016

as soon as a job/test starts, the unit is no longer ping-able even though the DUT thinks it's still on the ethernet with the same IP address and there are no long during the provisioning stage.

Comment 5 by shchen@chromium.org, Mar 25 2016

Thanks Danny!  I'll take a look at it.

Comment 6 by dchan@google.com, Mar 25 2016

I unit in chromeos1-row1-rack9-host3 is the new unit. (pool:faft-test-tot)
I moved the old unit to chromeos1-row1-rack11-host6 (pool:stress)

looks like both unit is no hosed. 

Is this still a problem?

Comment 8 by dchan@google.com, May 4 2016

yes. the system can be restarted. but as soon as it comes back, it lost the IP.
Last I try update everything but still no go. I just did a restart and looks like the kernel is not happen.

looks like it might related to crytohome, let me pull the unit and see

2016-05-04T11:44:48.499662-07:00 INFO cryptohomed[18377]: TPM error 0x2020 (Key not found in persistent storage): LoadKeyByUuid: failed LoadKeyByUUID
2016-05-04T11:44:49.056870-07:00 ERR cryptohomed[18377]: Couldn't wrap cryptohome key
2016-05-04T11:44:49.056997-07:00 WARNING cryptohomed[18377]: Could not load the device policy file.
2016-05-04T11:44:49.063604-07:00 WARNING chapsd[18368]: SRK does not exist - this is normal when the TPM is not yet owned.
2016-05-04T11:44:49.064428-07:00 WARNING chapsd[18368]: SRK does not exist - this is normal when the TPM is not yet owned.
2016-05-04T11:44:49.096773-07:00 ERR cryptohomed[18377]: stat() of /mnt/stateful_partition/unencrypted/preserve/attestation.epb failed.: No such file or directory

Comment 9 by dchan@google.com, May 4 2016

The device seems to failed to keep the network connection. Here what I've tried
- remove servo connection, connect to network in lab, I was able to ssh to DUT for some 10 seconds.  DUT shows that it's connected but I no longer able to ssh to DUT
- remove device and connect to network on my desk, I was able to ssh to DUT without problem.  This should eliminate Nyan is the problem
- I connect another DUT (cyan) to the network, the DUT is able to maintain the network connection.  This shows the network should be working

At this point the only combo that failed is nyan connects to the lab network.

Comment 10 by dchan@google.com, May 4 2016

The firmware on nyan is Google_Nyan_Big.8188.0.0 not sure where it comes from. Attempt to flash FW resulted in 

localhost log # chromeos-firmwareupdate --mode=factory
Starting Google_Nyan_Big firmware updater v4 (factory_install)...
 - Updater package: [Google_Nyan_Big.5771.63.0 / EC:big_v1.1.1807-2252740]
 - Current system:  [RO:Google_Nyan_Big.8188.0.0 , ACT:Google_Nyan_Big.8188.0.0 / EC:big_v1.1.4511-7ce63a0]
 - Write protection: Hardware: off, Software: Main=off EC=off
 * invoke: flashrom -p host --fast-verify -w bios.bin
 * invoke: flashrom -p ec --fast-verify -w ec.bin
 Execution failed (1): flashrom -p ec --fast-verify -w ec.bin
 Messages:
Erasing and writing flash chip... FAILED
flashrom v0.9.4  : 59543cd : Apr 30 2016 11:27:36 UTC on Linux 3.10.18 (armv7l), built with libpci 3.1.10, GCC 4.9.x 20150123 (prerelease), little endian
__cros_ec_command_dev(): Command 0x08 failed: -1, errno=110
__cros_ec_command_dev(): Command 0x08 failed: -1, errno=110
__cros_ec_command_dev(): Command 0x08 failed: -1, errno=110
Cannot determine write command version
__cros_ec_command_dev(): Command 0x11 failed: -1, errno=110
__cros_ec_command_dev(): Command 0x11 failed: -1, errno=110
__cros_ec_command_dev(): Command 0x11 failed: -1, errno=110
CROS_EC: Flash read error at offset 0x0
Uh oh. Cannot get latest content.
Your flash chip is in an unknown state.
Get help on IRC at irc.freenode.net (channel #flashrom) or
mail flashrom@flashrom.org with FAILED: your board name in the subject line!
-------------------------------------------------------------------------------
DO NOT REBOOT OR POWEROFF!
ERROR: Execution FAILED.
ERROR: Execution failed: ./updater4.sh (error code = 1)

soon after the DUT gone offline and screen started to flash on and off.

Comment 11 by dchan@google.com, May 4 2016

The dut seems to be fine on the network. As soon as cautotest start provisioning or running test, the DUT will go offline (even the OS display it's on the network). I've tried
1- reinstall OS to 50 (from 52)
2- reflash the FW to the official version

All seems to revert to the same problem.  I would declare this slot non-functional. I see we have another nyan_big, I will open that one up for faft testing.


Comment 12 by dchan@google.com, Jun 29 2016

The device is having problem with USB 3.0 dongle (20 reboot cause the device to stuck)
USB 2.0 dongle works  (100 reboot)

Comment 13 by dchan@google.com, Jun 29 2016

samsung usb 3.0 dongle cause DUT to stuck after 20 reboot
cisco usb2.0 dongle pass 100 reboot
iocrest USB 3.0 dongle is current running

Comment 14 by dchan@google.com, Jun 30 2016

I reran on samsung, device non-responsive after 288 reboot. With iocrest, device non-reponsive after 294 reboot.

After 294 reboots with iocrest, EC console shows 
[7032.778208 HC 0x87]
[38114.433268 Failed to get MANUF name]
[38114.434993 charge problem: batt params, 0x402 -> 0x7fc after 36167.534299s]
[38114.435834 try to wake battery]
[38114.443941 charge problem: static update, 0x4 -> 0x1 after 36167.336818s]
[38114.444762 charge_request(12896mV, 256mA)]
[38114.574736 Failed to get MANUF name]
[38114.576039 battery woke up]
[38114.607643 Failed to get MANUF name]
[38114.710807 charge problem: static update, 0x1 -> 0x4 after 0.266866s]
[38114.711533 charge_request(0mV, 0mA)]


Connect to EC console, battery returns
> battery
  Status:    0x00e0 FULL DCHG INIT
  Param flags:00000002
  Temp:      0x0ba7 = 298.3 K (25.2 C)
  V:         0x2f93 = 12179 mV
  V-desired: 0x0000 = 0 mV
  I:         0x0000 = 0 mA
  I-desired: 0x0000 = 0 mA
  Charging:  Not Allowed
  Charge:    97 %
  Manuf:     (error)
  Device:    (error)
  Chem:      (error)
  Serial:    0x154f
  V-design:  0x2c88 = 11400 mV
  Mode:      0x6001
  Abs charge:93 %
  Remaining: 2981 mAh
  Cap-full:  3097 mAh
    Design:  3220 mAh
  Time-full: 0h:0
    Empty:   0h:0


I think the (error) are expected. I EC reset, battery command return the same output


and chgstate returns
> chgstate
state = charge
ac = 1
batt_is_charging = 1
chg.*:
        voltage = 0mV
        current = 0mA
        input_current = 2048mA
        status = 0x11
        option = 0xf913
        flags = 0x0
batt.*:
        temperature = 25C
        state_of_charge = 97%
        voltage = 12180mV
        current = 0mA
        desired_voltage = 0mV
        desired_current = 0mA
        flags = 0x2
        remaining_capacity = 2981mAh
        full_capacity = 3097mAh
        is_present = YES
requested_voltage = 0mV
requested_current = 0mA
force_idle = 0
manual_mode = 0
user_current_limit = -1mA
battery_seems_to_be <<<< seems to stuck here


retest with USB 2.0 dongle

Comment 15 by dchan@google.com, Jul 1 2016

dongle 2.0 also died.
I servo reflash with the offical FW (to get rid of flash error in c#10) and rerun with 2.0 dongle.

Comment 16 by dchan@chromium.org, Oct 26 2016

Labels: servov4

Comment 17 by dchan@chromium.org, Oct 26 2016

Labels: -servov4

Comment 18 by dchan@google.com, May 19 2017

Cc: twreid@chromium.org
Labels: -Pri-1 Pri-3
Owner: modia@chromium.org
reassign to modia@ 

Cc: -twreid@chromium.org dgoyette@chromium.org

Sign in to add a comment