New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 653979 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocked on:
issue 684620



Sign in to add a comment

Internal modem intermittently disappears

Reported by james.j...@co.polk.or.us, Oct 7 2016

Issue description

This happens on 50+ machines starting in the last month or two in a variety of locations.  Many of these have been working for over a year up until recently.

Chrome Version: Google Chrome 52.0.2743.116 and up (unable to test prior)
Chrome OS Version: Platform 8350.68.0 (Official Build) stable-channel kip
Chrome OS Platform: HP Chromebook 11 G3, 14 G3, 11 G4 EE (same chip)
Network info: AT&T
HP lists the modem as: HP lt4110 LTE/HSPA+ 4G Module.
Crosh shows:  Manufacturer: Huawei  / Model: MBIM [12D1:15BB]  / Revision: 11.350.20.01.00


Steps To Reproduce:
(1) Disable WiFi/use just the AT&T for internet for a while.  Streaming music is an easy test.

(2) Wait for 10 min - a few hours, service is lost.  GUI shows  "Initializing Cellular Modem" and crosh no longer shows anything for modem status


How frequently does this problem reproduce? Always - sometimes very quickly, sometimes takes hours.

What is the impact to the user, and is there a workaround? If so, what is
it?   Any work may be lost - machine has to be power cycled for the modem to reappear.



We are seeing things like this in the logs:
-----------------------------------------------
2016-09-21T10:20:09.653352-07:00 INFO shill[934]: [INFO:cellular.cc(888)] wwan0 is down.
2016-09-21T10:20:09.663248-07:00 DEBUG ModemManager[3116]: [/dev/cdc-wdm0] unexpected port hangup!
2016-09-21T10:20:09.685826-07:00 INFO ModemManager[3116]: <info>  (net/wwan0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1
2016-09-21T10:20:09.701363-07:00 INFO ModemManager[3116]: <info>  (usbmisc/cdc-wdm0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1
2016-09-21T10:20:09.749367-07:00 INFO shill[934]: [INFO:modem_manager.cc(83)] Remove modem: /org/freedesktop/ModemManager1/Modem/0
2016-09-21T10:20:09.749381-07:00 INFO shill[934]: [INFO:modem.cc(66)] Modem destructed: /org/freedesktop/ModemManager1/Modem/0
2016-09-21T10:20:09.749392-07:00 INFO shill[934]: [INFO:cellular.cc(675)] Deregistering cellular service 14 for device wwan0
2016-09-21T10:20:09.750033-07:00 INFO shill[934]: [INFO:service.cc(309)] Service 14 destroyed.

2016-09-21T10:20:11.432682-07:00 DEBUG ModemManager[3116]: opening device...
2016-09-21T10:20:12.430045-07:00 INFO ModemManager[3116]: <info>  Couldn't find support for device at '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1': not supported by any plugin
2016-09-21T10:20:41.468845-07:00 DEBUG ModemManager[3116]: open operation timed out: closed
2016-09-21T10:20:41.469260-07:00 INFO ModemManager[3116]: <info>  Creating modem with plugin 'Huawei' and '2' ports
2016-09-21T10:20:41.470201-07:00 WARNING ModemManager[3116]: <warn>  Could not grab port (usbmisc/cdc-wdm0): 'Cannot add port 'usbmisc/cdc-wdm0', unsupported'
2016-09-21T10:20:41.470226-07:00 WARNING ModemManager[3116]: <warn>  Couldn't create modem for device at '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1': Failed to find primary AT port
--------------------------------------------------



When working, this appears under crosh modem status

-------------------------------------------------
modem /org/freedesktop/ModemManager1/Modem/0:
  GetStatus:
    state: 11
    signal-quality: 0, false
    current-bands: 0
    access-technologies: 16384
    m3gpp-registration-state: 1
    m3gpp-operator-code: 310410
    m3gpp-operator-name: AT&T
  Properties:
    Sim: /org/freedesktop/ModemManager1/SIM/0
    Bearers: /org/freedesktop/ModemManager1/Bearer/0
    SupportedCapabilities: 4
    CurrentCapabilities: 4
    MaxBearers: 1
    MaxActiveBearers: 1
    Manufacturer: Huawei
    Model: MBIM [12D1:15BB]
    Revision: 11.350.20.01.00
    DeviceIdentifier: d534288f6d80ccb7ed93e018f158164b9b8fe855
    Device: /sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1
    Drivers: cdc_mbim
    Plugin: Huawei
    PrimaryPort: cdc-wdm0
    Ports: cdc-wdm0, 7, wwan0, 2
    EquipmentIdentifier: xxxxxx
    UnlockRequired: 1
    UnlockRetries: 3
    State: 11
    StateFailedReason: 0
    AccessTechnologies: 16384
    SignalQuality: 0, false
    OwnNumbers: xxxxxxxxxx
    PowerState: 3
    SupportedModes: 6, 0
    CurrentModes: 6, 0
    SupportedBands: 0
    CurrentBands: 0
    SupportedIpFamilies: 7
  3GPP:
    Imei: xxxxxxxxxxxxxxxxxxx
    RegistrationState: 1
    OperatorCode: 310410
    OperatorName: AT&T
    EnabledFacilityLocks: 0
    SubscriptionState: 0
  CDMA:
  SIM /org/freedesktop/ModemManager1/SIM/0:
    SimIdentifier: xxxxxxxxxxxxxxxxxxxx
    Imsi: xxxxxxxxxxxxxxxxx
    OperatorIdentifier: 310410
    OperatorName: AT&T
----------------------------------------------------------
 
5CD50938X7.png
1.7 MB View Download
debug-logs_20160921-162912.tgz
2.0 MB Download

Comment 1 by mef@chromium.org, Oct 10 2016

Components: OS>Systems>Mobile

Comment 2 by kotah@chromium.org, Oct 17 2016

Cc: kotah@chromium.org
Labels: -Pri-3 Hotlist-Enterprise Pri-2
I also get the same error every time I try to run any MBIM Compliance unittest.

Here is the log I get:
================================================================
2016-10-28T11:59:50.958426+05:30 INFO ModemManager[1207]: <debug> Signal quality value not updated in 60s, marking as not being recent
2016-10-28T11:59:53.860626+05:30 INFO shill[779]: [INFO:cellular.cc(888)] wwan0 is down.
2016-10-28T11:59:53.860836+05:30 INFO shill[779]: [INFO:service.cc(404)] Service 2: state Online -> Idle
2016-10-28T11:59:53.860846+05:30 INFO shill[779]: [INFO:manager.cc(1455)] Service 2 updated; state: Idle failure Unknown
2016-10-28T11:59:53.861517+05:30 INFO shill[779]: [INFO:service.cc(309)] Auto-connecting to service 2
2016-10-28T11:59:53.861539+05:30 INFO shill[779]: [INFO:service.cc(325)] Connect to service 2: AutoConnect
2016-10-28T11:59:53.861565+05:30 ERR shill[779]: [ERROR:error.cc(146)] [cellular.cc(685)]: Already connected; connection request ignored.
2016-10-28T11:59:53.862082+05:30 INFO shill[779]: [INFO:manager.cc(1480)] Device wwan0 updated: enabled
2016-10-28T11:59:53.862801+05:30 INFO ModemManager[1207]: <info>  (net/wwan0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5
2016-10-28T11:59:53.868543+05:30 DEBUG ModemManager[1207]: [/dev/cdc-wdm0] unexpected port hangup!
2016-10-28T11:59:53.868761+05:30 INFO ModemManager[1207]: <info>  (usbmisc/cdc-wdm0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5
2016-10-28T11:59:53.868786+05:30 INFO ModemManager[1207]: <debug> Removing empty device '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-5'
2016-10-28T11:59:53.868991+05:30 INFO ModemManager[1207]: <debug> Removing from DBus bearer at '/org/freedesktop/ModemManager1/Bearer/0'
2016-10-28T11:59:53.869075+05:30 INFO ModemManager[1207]: <debug> Unexported modem '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-5' from path '/org/freedesktop/ModemManager1/Modem/0'
2016-10-28T11:59:53.869329+05:30 INFO ModemManager[1207]: <debug> Modem (Generic) '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-5' completely disposed
2016-10-28T12:00:00.808832+05:30 INFO shill[779]: [INFO:modem_manager_1.cc(122)] MM1:  Removing interfaces from /org/freedesktop/ModemManager1/Modem/0
2016-10-28T12:00:00.808853+05:30 INFO shill[779]: [INFO:modem_manager.cc(83)] Remove modem: /org/freedesktop/ModemManager1/Modem/0
2016-10-28T12:00:00.808860+05:30 INFO shill[779]: [INFO:modem.cc(66)] Modem destructed: /org/freedesktop/ModemManager1/Modem/0
2016-10-28T12:00:00.809035+05:30 INFO shill[779]: [INFO:cellular.cc(675)] Deregistering cellular service 2 for device wwan0
2016-10-28T12:00:00.809046+05:30 INFO shill[779]: [INFO:service.cc(302)] Service 2 destroyed.
2016-10-28T12:00:00.814017+05:30 INFO shill[779]: [INFO:device.cc(245)] Device destructed: wwan0 index 3
2016-10-28T12:00:00.814798+05:30 ERR shill[779]: [ERROR:utils.cc(88)] AddDBusError(...): Domain=dbus, Code=org.freedesktop.DBus.Error.UnknownMethod, Message=No such interface `org.freedesktop.ModemManager1.Modem' on object at path /org/freedesktop/ModemManager1/Modem/0
2016-10-28T12:00:00.814993+05:30 ERR shill[779]: [ERROR:utils.cc(88)] AddDBusError(...): Domain=dbus, Code=org.freedesktop.DBus.Error.UnknownMethod, Message=No such interface `org.freedesktop.ModemManager1.Modem.Simple' on object at path /org/freedesktop/ModemManager1/Modem/0
=====================================================================

Comment 4 by roy...@google.com, Nov 9 2016

Cc: josa...@chromium.org dskaram@chromium.org
Owner: dskaram@chromium.org
David/Josafat: What is the best way to route this issue ?
Cc: vpalatin@chromium.org yoshiat@google.com cernekee@chromium.org
Adding Vincent and Kevin who can perhaps help us understand what's happening.

Yoshi can perhaps help route to the proper subteam.
Cc: benchan@chromium.org
+Ben who is helping (again) on modem support
When correlating the kernel log, something is strange:
before the first shill trace for the event, something is resetting the USB device and triggering the disconnect (or the other way round, the modem disconnects from the USB bus triggering a reset)

2016-09-21T10:06:57.655006-07:00 INFO shill[934]: [INFO:service.cc(325)] Suppressed autoconnect to service 3 (no endpoints)
2016-09-21T10:20:09.653352-07:00 INFO shill[934]: [INFO:cellular.cc(888)] wwan0 is down.
2016-09-21T10:20:09.657064-07:00 INFO shill[934]: [INFO:service.cc(411)] Service 14: state Online -> Idle
2016-09-21T10:20:09.657122-07:00 INFO shill[934]: [INFO:manager.cc(1455)] Service 14 updated; state: Idle failure Unknown


2016-09-21T10:20:09.428021-07:00 INFO kernel: [  853.615249] usb 1-2: reset high-speed USB device number 2 using xhci_hcd
2016-09-21T10:20:09.443010-07:00 WARNING kernel: [  853.629907] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880076e16f40
2016-09-21T10:20:09.652354-07:00 INFO kernel: [  853.839262] usb 1-2.1: USB disconnect, device number 5
2016-09-21T10:20:09.652392-07:00 INFO kernel: [  853.839428] cdc_mbim 1-2.1:3.0 wwan0: unregister 'cdc_mbim' usb-0000:00:14.0-2.1, CDC MBIM


By the way, that's a 'Kip' device (Bay-Trail platform) with the Huawei LTE modem, do we have a provisioned one somewhere (e.g in the lab) ?
I'm happy to test on our end if there's something useful for me to do and some quick instructions.

If needed, we should be able to ship one out temporarily with active AT&T service if that can be coordinated via Google Case #10711712.

We've had similar reports of issues on the HP 14" G3. Unlike the rest, they have an NVIDIA Tegra K1 - fairly certain the modem is the same.  Working on getting one back to the office to test.


Thanks all

Comment 9 by thomas.j...@hp.com, Dec 2 2016

Hey Ben,
Has this issue been resolved? Let me know if you need to obtain captures for this issue and I will request these from the customer. I could also repro the issue and provide logs if that would be easier. Thank you!!
As Vincent suggested in comment #7, the modem reset or got reset over USB. When the modem was re-enumerated over USB, its MBIM interface didn't respond to the MBIM_OPEN_MSG command from ModemManager before the 1s timeout. The actual cause of the USB reset isn't very clear from the log, which may require assistance from the modem vendor to collect internal modem logs to further debug it.

Regardless of the actual trigger of the USB reset, there is a chance that the modem may take a bit longer than expected to respond to MBIM_OPEN_MSG after it gets reconnected to USB. But first we need to determine whether the modem is totally unresponsive or just slow to respond. I'll try to find a Kip device and reproduce the issue. 


######


# Signal quality: 35%

2016-09-21T15:40:48.552356-07:00 DEBUG shill[921]: [VERBOSE1:object_proxy.cc(500)] Signal received: message_type: MESSAGE_SIGNAL#012path: /org/freedesktop/ModemManager1/Modem/0#012interface: org.freedesktop.DBus.Properties#012member: PropertiesChanged#012sender: :1.24#012signature: sa{sv}as#012serial: 31#012#012string "org.freedesktop.ModemManager1.Modem"#012array [#012  dict entry {#012    string "SignalQuality"#012    variant       struct {#012        uint32_t 35#012        bool false#012      }#012  }#012]#012array [#012]#012
2016-09-21T15:40:48.552468-07:00 DEBUG shill[921]: [VERBOSE2:cellular.cc(612)] /device/wwan0 Signal strength: 35

# USB reset and modem disconnected

2016-09-21T15:41:17.623716-07:00 INFO kernel: [ 9831.119166] usb 1-2: reset high-speed USB device number 2 using xhci_hcd
2016-09-21T15:41:17.635277-07:00 WARNING kernel: [ 9831.130881] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88007764af40
2016-09-21T15:41:17.843459-07:00 INFO kernel: [ 9831.338849] usb 1-2.1: USB disconnect, device number 5
2016-09-21T15:41:17.843470-07:00 INFO kernel: [ 9831.339033] cdc_mbim 1-2.1:3.0 wwan0: unregister 'cdc_mbim' usb-0000:00:14.0-2.1, CDC MBIM
2016-09-21T15:41:17.887942-07:00 DEBUG ModemManager[3158]: [/dev/cdc-wdm0] unexpected port hangup!

# modem re-enumerated on USB, ModemManager cleaned up the old modem object and add a new one

2016-09-21T15:41:17.927247-07:00 INFO kernel: [ 9831.423228] usb 1-2.1: new high-speed USB device number 6 using xhci_hcd
2016-09-21T15:41:17.940253-07:00 INFO kernel: [ 9831.436477] usb 1-2.1: New USB device found, idVendor=12d1, idProduct=15bb
2016-09-21T15:41:17.940271-07:00 INFO kernel: [ 9831.436493] usb 1-2.1: New USB device strings: Mfr=6, Product=5, SerialNumber=0
2016-09-21T15:41:17.940305-07:00 INFO kernel: [ 9831.436504] usb 1-2.1: Product: HUAWEI Mobile Broadband Module
2016-09-21T15:41:17.940311-07:00 INFO kernel: [ 9831.436513] usb 1-2.1: Manufacturer: Huawei Technologies Co., Ltd.
2016-09-21T15:41:17.944407-07:00 INFO kernel: [ 9831.440279] usb 1-2.1: MAC-Address: 02:2c:80:13:92:63
2016-09-21T15:41:17.944430-07:00 INFO kernel: [ 9831.440597] cdc_ncm 1-2.1:2.0 usb0: register 'cdc_ncm' at usb-0000:00:14.0-2.1, CDC NCM, 02:2c:80:13:92:63

2016-09-21T15:41:18.033947-07:00 INFO ModemManager[3158]: <info>  (usbmisc/cdc-wdm0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1
2016-09-21T15:41:18.033981-07:00 INFO ModemManager[3158]: <info>  (net/wwan0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1
2016-09-21T15:41:18.140251-07:00 INFO kernel: [ 9831.636736] cdc_ncm 1-2.1:2.0 usb0: unregister 'cdc_ncm' usb-0000:00:14.0-2.1, CDC NCM

2016-09-21T15:41:18.150392-07:00 INFO kernel: [ 9831.646675] cdc_mbim 1-2.1:3.0: cdc-wdm0: USB WDM device
2016-09-21T15:41:18.150413-07:00 INFO kernel: [ 9831.646929] cdc_mbim 1-2.1:3.0 wwan0: register 'cdc_mbim' at usb-0000:00:14.0-2.1, CDC MBIM, 56:4a:0c:68:b3:b8

2016-09-21T15:41:18.527958-07:00 INFO ModemManager[3158]: <info>  (net/usb0): released by modem /sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1

# ModemManager failed to open the cdc-wdm0 port and ended probing for the modem

2016-09-21T15:41:20.030434-07:00 WARNING ModemManager[3158]: <warn>  (Plugin Manager) (Huawei) [usb0] error when checking support: '(Huawei) Missing port probe for port (net/usb0)'
2016-09-21T15:41:20.030470-07:00 WARNING ModemManager[3158]: <warn>  (Plugin Manager) (Generic) [usb0] error when checking support: '(Generic) Missing port probe for port (net/usb0)'
2016-09-21T15:41:20.067799-07:00 DEBUG ModemManager[3158]: opening device...
2016-09-21T15:41:20.067829-07:00 DEBUG ModemManager[3158]: [/dev/cdc-wdm0] Queried max control message size: 1024

2016-09-21T15:41:50.115418-07:00 DEBUG ModemManager[3158]: open operation timed out: closed
2016-09-21T15:41:50.134201-07:00 INFO ModemManager[3158]: <info>  Creating modem with plugin 'Huawei' and '2' ports
2016-09-21T15:41:50.158388-07:00 WARNING ModemManager[3158]: <warn>  Could not grab port (usbmisc/cdc-wdm0): 'Cannot add port 'usbmisc/cdc-wdm0', unsupported'
2016-09-21T15:41:50.158578-07:00 WARNING ModemManager[3158]: <warn>  Couldn't create modem for device at '/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2.1': Failed to find primary AT port
(END)

@ Ben 

I have requested a unit from the customer that is able to reproduce the issue. Let me know if its easier for me to repro and send logs or to send it to you. 

Thanks Thomas, that would be very helpful.

If you can reproduce the issue, could you test one thing for me? That would give me a data point about whether the modem is completely unresponsive to MBIM_OPEN_MSG from ModemManager, or it's just temporarily unresponsive.

- Turn the Chromebook into developer mode, so that you can access the root shell later
- Open a root shell and restart ModemManager with debug logging:

  restart modemmanager MM_LOGLEVEL=DEBUG

- Reproduce the issue. You can monitor /var/log/net.log to look for this message:

     DEBUG ModemManager[3158]: open operation timed out: closed

- Wait for 10 seconds. Open the root shell again, and restart ModemManager with debug logging:

  restart modemmanager MM_LOGLEVEL=DEBUG

- Check /var/log/net.log again to see if ModemManager can now open the modem.

Comment 13 by thomas.j...@hp.com, Dec 13 2016

Hey Ben, 
I reproduced the issue. I seen the open operation timed out but It was not 3158. I have attached the log file. After the issue reproduced I performed the modem manager restart again and the issue still persisted. 
net.log
361 KB View Download
Thanks Thomas. Did you perform any specific activity to reproduce the issue?

Comment 15 Deleted

Comment 16 by tjrey...@gmail.com, Dec 15 2016

Hey Ben, 
Nothing Special, played music on youtube and let it go through playlists until the WWAN dropped and continued to initialize. This is with Wifi disabled. Please let me know what I can do to assist on this. Thanks Ben
Thomas, based on your comment #13, it seems like the modem was in a state that didn't response to MBIM_OPEN_MSG (even after ModemManager started from scratch). We will focus on looking into from two different aspects: (1) what triggered the modem into such a state, and (2) how to get modem out of such a state. If we can't address (1), resolving (2) would at least get the modem back to normal (despite a temporal drop of connectivity). Debugging (1) would require getting USB logs and internal modem logs, which requires some special setup. I need to first reproduce the issue.

Unfortunately, I haven't been able to reproduce the issue locally yet. But then when I cross examine your log with mine, I noticed that we have different revisions of modem firmware.

  Your modem: 11.350.20.01.00
    My modem: 11.350.16.12.00

It seems like your modem have a newer firmware. I'll redo my test with that modem firmware revision. 

Comment 18 by tjrey...@gmail.com, Dec 15 2016

Please let me know if you would like to me mail this unit to you for helping expedite this investigation. Send me your address and I will get this out today. Thank you
Hi all.  Appreciate you working on this.

I suspect that modem firmware upgrade may be what created this issue for us, do we know when it was released?  All the units were running fine for a fair bit of time then broke all around the same time.  All of them I checked show that newer firmware version.

We would be happy to ship another active unit out if it would be helpful.  We have existing tickets with Google and HP to coordinate that.


Thanks!
Josh

Comment 20 by thomas.j...@hp.com, Dec 16 2016

Hey Ben, 
Unit will deliver tomorrow @ 10:30 am. FEDEX # 7779 6522 7195
Cc: -dskaram@chromium.org
Owner: benchan@chromium.org

Comment 22 by thomas.j...@hp.com, Dec 19 2016

Hey Ben did you receive the unit?
Thomas, thanks for sending us the unit. We received the unit and are trying to reproduce the issue. I also noticed that your unit has an AT&T SIM, while I was testing with a T-Mobile SIM. The customer also seems to use AT&T as well. Given that the customer is using the same carrier, same modem with the same firmware, and we haven't rolled any changes to this SKU, there is also a possibility that the modem doesn't respond correctly to a change rolled out from the carrier side. And that may explain why the customer suddenly encounters the issue after using the SKU for a year.  We will continue to look into the root cause and update this bug with any finding.

Comment 24 by thomas.j...@hp.com, Dec 27 2016

Hey Ben were you able to reproduce the issue with the debugger attached and get this out to Huawei for investigation?
Here's one USB trace that captured the moment when the modem NAKing  SEND_ENCAPSULATED_COMMAND from hosts.
kip-me936-failed.tdc
446 KB Download
Project Member

Comment 26 by bugdroid1@chromium.org, Jan 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/627944047dd910c016ccf1af5b543eac39cc4727

commit 627944047dd910c016ccf1af5b543eac39cc4727
Author: Ben Chan <benchan@chromium.org>
Date: Fri Jan 20 22:26:05 2017

chromeos-base/modem-watchdog: add package

This CL adds a chromeos-base/modem-watchdog package, which implements a
watchdog to monitor if the built-in modem (Huawei ME936) on Kip is
successfully picked up ModemManager within a specified amount of time
after the modem appears on USB under USB configuration 3 (i.e. MBIM
mode). The watchdog will power cycle the modem upon timeout. This is a
temporary workaround of an out-of-sync issue between the modem and the
host (chromium:653979).

BUG= chromium:653979 
TEST=Tested the following on a Kip with a built-in Huawei ME936 modem:
- Verified that the watchdog performs nothing and exits gracefully
  when the modem is picked up by ModemManger within the specified
  timeout.
- After reproducing the issue described in chromium:653979. verified
  that the modem is power-cycled by the watchdog and then picked up
  by ModemManger afterwards.

Change-Id: Ie5efebcc04b8aa92f8c294efc4707f3131f4e4a0
Reviewed-on: https://chromium-review.googlesource.com/431200
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r1.ebuild
[add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1.ebuild
[add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh
[add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/files/99-chromeos-kip-modem-watchdog.rules

Project Member

Comment 27 by bugdroid1@chromium.org, Jan 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/c006c96c7f605dec93740cf30d325b843c7592dc

commit c006c96c7f605dec93740cf30d325b843c7592dc
Author: Ben Chan <benchan@chromium.org>
Date: Fri Jan 20 22:32:44 2017

chromeos-bsp-kip: include chromeos-base/modem-watchdog package

This CL modifies the Kip BSP to include the chromeos-base/modem-watchdog
package, which implements a workaround for the out-of-sync issue
(chromium:653979) between the host and the built-in modem on Kip.

BUG= chromium:653979 
TEST=Verified that the watchdog files are installed on kip.

Change-Id: I9f6a98c8520965b868ef15e3080726e7f0057027
Reviewed-on: https://chromium-review.googlesource.com/431201
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[rename] https://crrev.com/c006c96c7f605dec93740cf30d325b843c7592dc/overlay-kip/chromeos-base/chromeos-bsp-kip/chromeos-bsp-kip-0.0.1-r15.ebuild
[modify] https://crrev.com/c006c96c7f605dec93740cf30d325b843c7592dc/overlay-kip/chromeos-base/chromeos-bsp-kip/chromeos-bsp-kip-0.0.1.ebuild

Blockedon: 684620
For the purpose of merging the watchdog changes to M57, I filed a separate  issue 684620  to track the watchdog changes while leaving this bug open until we identify the root cause.
Blockedon: 684620
Project Member

Comment 31 by bugdroid1@chromium.org, Jan 26 2017

Labels: merge-merged-release-R57-9202.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/604bc18852feedd9bde9d1398c202cc3a6ea611d

commit 604bc18852feedd9bde9d1398c202cc3a6ea611d
Author: Ben Chan <benchan@chromium.org>
Date: Fri Jan 20 22:26:05 2017

chromeos-base/modem-watchdog: add package

This CL adds a chromeos-base/modem-watchdog package, which implements a
watchdog to monitor if the built-in modem (Huawei ME936) on Kip is
successfully picked up ModemManager within a specified amount of time
after the modem appears on USB under USB configuration 3 (i.e. MBIM
mode). The watchdog will power cycle the modem upon timeout. This is a
temporary workaround of an out-of-sync issue between the modem and the
host (chromium:653979).

BUG= chromium:653979 
TEST=Tested the following on a Kip with a built-in Huawei ME936 modem:
- Verified that the watchdog performs nothing and exits gracefully
  when the modem is picked up by ModemManger within the specified
  timeout.
- After reproducing the issue described in chromium:653979. verified
  that the modem is power-cycled by the watchdog and then picked up
  by ModemManger afterwards.

Change-Id: Ie5efebcc04b8aa92f8c294efc4707f3131f4e4a0
Reviewed-on: https://chromium-review.googlesource.com/431200
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit 627944047dd910c016ccf1af5b543eac39cc4727)
Reviewed-on: https://chromium-review.googlesource.com/433380
Commit-Queue: Ben Chan <benchan@chromium.org>

[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r1.ebuild
[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1.ebuild
[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh
[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/files/99-chromeos-kip-modem-watchdog.rules

Project Member

Comment 32 by bugdroid1@chromium.org, Jan 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/604bc18852feedd9bde9d1398c202cc3a6ea611d

commit 604bc18852feedd9bde9d1398c202cc3a6ea611d
Author: Ben Chan <benchan@chromium.org>
Date: Fri Jan 20 22:26:05 2017

chromeos-base/modem-watchdog: add package

This CL adds a chromeos-base/modem-watchdog package, which implements a
watchdog to monitor if the built-in modem (Huawei ME936) on Kip is
successfully picked up ModemManager within a specified amount of time
after the modem appears on USB under USB configuration 3 (i.e. MBIM
mode). The watchdog will power cycle the modem upon timeout. This is a
temporary workaround of an out-of-sync issue between the modem and the
host (chromium:653979).

BUG= chromium:653979 
TEST=Tested the following on a Kip with a built-in Huawei ME936 modem:
- Verified that the watchdog performs nothing and exits gracefully
  when the modem is picked up by ModemManger within the specified
  timeout.
- After reproducing the issue described in chromium:653979. verified
  that the modem is power-cycled by the watchdog and then picked up
  by ModemManger afterwards.

Change-Id: Ie5efebcc04b8aa92f8c294efc4707f3131f4e4a0
Reviewed-on: https://chromium-review.googlesource.com/431200
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit 627944047dd910c016ccf1af5b543eac39cc4727)
Reviewed-on: https://chromium-review.googlesource.com/433380
Commit-Queue: Ben Chan <benchan@chromium.org>

[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r1.ebuild
[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1.ebuild
[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh
[add] https://crrev.com/604bc18852feedd9bde9d1398c202cc3a6ea611d/overlay-kip/chromeos-base/modem-watchdog/files/99-chromeos-kip-modem-watchdog.rules

Project Member

Comment 33 by bugdroid1@chromium.org, Jan 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/b8ac0838851ea69924c04698cfc162e7fe5a61b2

commit b8ac0838851ea69924c04698cfc162e7fe5a61b2
Author: Ben Chan <benchan@chromium.org>
Date: Tue Jan 24 18:49:57 2017

chromeos-base/modem-watchdog: log watchdog PID instead of logger PID

This CL fixes the log functions in chromeos-kip-modem-watchdog.sh to use
the watchdog PID instead of the logger PID when outputting messages to
the system log. This helps correlate messages produced by the same
watchdog process.

BUG= chromium:653979 
BUG= chromium:684620 
TEST=Tested the following on a Kip with a built-in Huawei ME936 modem:
- Observed /var/log/messages for modem-watchdog messages. The messages
  produced by the same watchdog process have the same process ID.
- Manually power cycle the modem and observd a new watchdog process is
  triggered. The messages produced by the new watchdog process have a
  different process ID.

Change-Id: Ia741d273db08afcd26808622ac661be522f3c649
Reviewed-on: https://chromium-review.googlesource.com/431837
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit b2418aacf04534edd91b529d27658dc3188e9a06)
Reviewed-on: https://chromium-review.googlesource.com/433382
Commit-Queue: Ben Chan <benchan@chromium.org>

[modify] https://crrev.com/b8ac0838851ea69924c04698cfc162e7fe5a61b2/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh
[rename] https://crrev.com/b8ac0838851ea69924c04698cfc162e7fe5a61b2/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r2.ebuild

Cc: -yoshiat@google.com
Status: Fixed (was: Unconfirmed)

Sign in to add a comment