New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 684620 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 653979



Sign in to add a comment

Add modem watchdog on kip to detect and power cycle out-of-sync modem

Project Member Reported by benchan@chromium.org, Jan 24 2017

Issue description

Forking from  issue 653979  where we observe that the built-in modem on the Kip device sometimes goes out of sync with the host. When the problem occurs, the modem is first dropped off from the USB bus and then gets re-enumerated on USB. When the modem is detected again on the host (by ModemManager), the MBIM communication between the modem and the host breaks down. The exact trigger to the out-of-sync problem is being investigation in  issue 653979 . Simply power-cycling the modem seems to resolve the issue.

As an interim mitigation to the issue, we add a modem watchdog on Kip. The watchdog is activated when the modem (under the MBIM configuration) appears on USB. The watchdog waits for ModemManager to go through initial communications with the modem and expose a modem object. If the watchdog doesn't see a modem object exposed by ModemManager within a specified amount of time (30s), it powers cycle the modem.
 
Blocking: 653979
The following CLs have been committed (noted  issue 653979 #c26 and 653979#c27):

  The following revision refers to this bug:
    https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/627944047dd910c016ccf1af5b543eac39cc4727

  commit 627944047dd910c016ccf1af5b543eac39cc4727
  Author: Ben Chan <benchan@chromium.org>
  Date: Fri Jan 20 22:26:05 2017

  chromeos-base/modem-watchdog: add package

  This CL adds a chromeos-base/modem-watchdog package, which implements a
  watchdog to monitor if the built-in modem (Huawei ME936) on Kip is
  successfully picked up ModemManager within a specified amount of time
  after the modem appears on USB under USB configuration 3 (i.e. MBIM
  mode). The watchdog will power cycle the modem upon timeout. This is a
  temporary workaround of an out-of-sync issue between the modem and the
  host (chromium:653979).

  BUG= chromium:653979 
  TEST=Tested the following on a Kip with a built-in Huawei ME936 modem:
  - Verified that the watchdog performs nothing and exits gracefully
    when the modem is picked up by ModemManger within the specified
    timeout.
  - After reproducing the issue described in chromium:653979. verified
    that the modem is power-cycled by the watchdog and then picked up
    by ModemManger afterwards.

  Change-Id: Ie5efebcc04b8aa92f8c294efc4707f3131f4e4a0
  Reviewed-on: https://chromium-review.googlesource.com/431200
  Commit-Ready: Ben Chan <benchan@chromium.org>
  Tested-by: Ben Chan <benchan@chromium.org>
  Reviewed-by: Mike Frysinger <vapier@chromium.org>

  [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r1.ebuild
  [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1.ebuild
  [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh
  [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/files/99-chromeos-kip-modem-watchdog.rules


  The following revision refers to this bug:
    https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/c006c96c7f605dec93740cf30d325b843c7592dc

  commit c006c96c7f605dec93740cf30d325b843c7592dc
  Author: Ben Chan <benchan@chromium.org>
  Date: Fri Jan 20 22:32:44 2017

  chromeos-bsp-kip: include chromeos-base/modem-watchdog package

  This CL modifies the Kip BSP to include the chromeos-base/modem-watchdog
  package, which implements a workaround for the out-of-sync issue
  (chromium:653979) between the host and the built-in modem on Kip.

  BUG= chromium:653979 
  TEST=Verified that the watchdog files are installed on kip.

  Change-Id: I9f6a98c8520965b868ef15e3080726e7f0057027
  Reviewed-on: https://chromium-review.googlesource.com/431201
  Commit-Ready: Ben Chan <benchan@chromium.org>
  Tested-by: Ben Chan <benchan@chromium.org>
  Reviewed-by: Mike Frysinger <vapier@chromium.org>

  [rename] https://crrev.com/c006c96c7f605dec93740cf30d325b843c7592dc/overlay-kip/chromeos-base/chromeos-bsp-kip/chromeos-bsp-kip-0.0.1-r15.ebuild
  [modify] https://crrev.com/c006c96c7f605dec93740cf30d325b843c7592dc/overlay-kip/chromeos-base/chromeos-bsp-kip/chromeos-bsp-kip-0.0.1.ebuild

Labels: Merge-Request-57
Status: Fixed (was: Assigned)
Blocking: 653979
Project Member

Comment 5 by sheriffbot@chromium.org, Jan 25 2017

Labels: -Merge-Request-57 Hotlist-Merge-Approved Merge-Approved-57
Your change meets the bar and is auto-approved for M57. Please go ahead and merge the CL to branch 2987 manually. Please contact milestone owner if you have questions.
Owners: amineer@(clank), cmasso@(bling), ketakid@(cros), govind@(desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Cc: keta...@chromium.org
+ketakid to confirm comment 5
govind@ do you have any concerns with this? Looks like a Chrome CL.
It's a Chrome OS CL, which only affects the Kip device with a built-in modem
benchan@ please go ahead and land on M57.
Project Member

Comment 10 by bugdroid1@chromium.org, Jan 26 2017

Labels: merge-merged-release-R57-9202.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/b8ac0838851ea69924c04698cfc162e7fe5a61b2

commit b8ac0838851ea69924c04698cfc162e7fe5a61b2
Author: Ben Chan <benchan@chromium.org>
Date: Tue Jan 24 18:49:57 2017

chromeos-base/modem-watchdog: log watchdog PID instead of logger PID

This CL fixes the log functions in chromeos-kip-modem-watchdog.sh to use
the watchdog PID instead of the logger PID when outputting messages to
the system log. This helps correlate messages produced by the same
watchdog process.

BUG= chromium:653979 
BUG= chromium:684620 
TEST=Tested the following on a Kip with a built-in Huawei ME936 modem:
- Observed /var/log/messages for modem-watchdog messages. The messages
  produced by the same watchdog process have the same process ID.
- Manually power cycle the modem and observd a new watchdog process is
  triggered. The messages produced by the new watchdog process have a
  different process ID.

Change-Id: Ia741d273db08afcd26808622ac661be522f3c649
Reviewed-on: https://chromium-review.googlesource.com/431837
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit b2418aacf04534edd91b529d27658dc3188e9a06)
Reviewed-on: https://chromium-review.googlesource.com/433382
Commit-Queue: Ben Chan <benchan@chromium.org>

[modify] https://crrev.com/b8ac0838851ea69924c04698cfc162e7fe5a61b2/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh
[rename] https://crrev.com/b8ac0838851ea69924c04698cfc162e7fe5a61b2/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r2.ebuild

Labels: Merge-Merged
Project Member

Comment 12 by sheriffbot@chromium.org, Jan 30 2017

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -Hotlist-Merge-Approved -Merge-Approved-57 -merge-merged-release-R57-9202.B
Labels: VerifyIn-61

Comment 15 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment