Add modem watchdog on kip to detect and power cycle out-of-sync modem |
||||||||||
Issue descriptionForking from issue 653979 where we observe that the built-in modem on the Kip device sometimes goes out of sync with the host. When the problem occurs, the modem is first dropped off from the USB bus and then gets re-enumerated on USB. When the modem is detected again on the host (by ModemManager), the MBIM communication between the modem and the host breaks down. The exact trigger to the out-of-sync problem is being investigation in issue 653979 . Simply power-cycling the modem seems to resolve the issue. As an interim mitigation to the issue, we add a modem watchdog on Kip. The watchdog is activated when the modem (under the MBIM configuration) appears on USB. The watchdog waits for ModemManager to go through initial communications with the modem and expose a modem object. If the watchdog doesn't see a modem object exposed by ModemManager within a specified amount of time (30s), it powers cycle the modem.
,
Jan 24 2017
The following CLs have been committed (noted issue 653979 #c26 and 653979#c27): The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/627944047dd910c016ccf1af5b543eac39cc4727 commit 627944047dd910c016ccf1af5b543eac39cc4727 Author: Ben Chan <benchan@chromium.org> Date: Fri Jan 20 22:26:05 2017 chromeos-base/modem-watchdog: add package This CL adds a chromeos-base/modem-watchdog package, which implements a watchdog to monitor if the built-in modem (Huawei ME936) on Kip is successfully picked up ModemManager within a specified amount of time after the modem appears on USB under USB configuration 3 (i.e. MBIM mode). The watchdog will power cycle the modem upon timeout. This is a temporary workaround of an out-of-sync issue between the modem and the host (chromium:653979). BUG= chromium:653979 TEST=Tested the following on a Kip with a built-in Huawei ME936 modem: - Verified that the watchdog performs nothing and exits gracefully when the modem is picked up by ModemManger within the specified timeout. - After reproducing the issue described in chromium:653979. verified that the modem is power-cycled by the watchdog and then picked up by ModemManger afterwards. Change-Id: Ie5efebcc04b8aa92f8c294efc4707f3131f4e4a0 Reviewed-on: https://chromium-review.googlesource.com/431200 Commit-Ready: Ben Chan <benchan@chromium.org> Tested-by: Ben Chan <benchan@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r1.ebuild [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1.ebuild [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh [add] https://crrev.com/627944047dd910c016ccf1af5b543eac39cc4727/overlay-kip/chromeos-base/modem-watchdog/files/99-chromeos-kip-modem-watchdog.rules The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/c006c96c7f605dec93740cf30d325b843c7592dc commit c006c96c7f605dec93740cf30d325b843c7592dc Author: Ben Chan <benchan@chromium.org> Date: Fri Jan 20 22:32:44 2017 chromeos-bsp-kip: include chromeos-base/modem-watchdog package This CL modifies the Kip BSP to include the chromeos-base/modem-watchdog package, which implements a workaround for the out-of-sync issue (chromium:653979) between the host and the built-in modem on Kip. BUG= chromium:653979 TEST=Verified that the watchdog files are installed on kip. Change-Id: I9f6a98c8520965b868ef15e3080726e7f0057027 Reviewed-on: https://chromium-review.googlesource.com/431201 Commit-Ready: Ben Chan <benchan@chromium.org> Tested-by: Ben Chan <benchan@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [rename] https://crrev.com/c006c96c7f605dec93740cf30d325b843c7592dc/overlay-kip/chromeos-base/chromeos-bsp-kip/chromeos-bsp-kip-0.0.1-r15.ebuild [modify] https://crrev.com/c006c96c7f605dec93740cf30d325b843c7592dc/overlay-kip/chromeos-base/chromeos-bsp-kip/chromeos-bsp-kip-0.0.1.ebuild
,
Jan 24 2017
,
Jan 24 2017
,
Jan 25 2017
Your change meets the bar and is auto-approved for M57. Please go ahead and merge the CL to branch 2987 manually. Please contact milestone owner if you have questions. Owners: amineer@(clank), cmasso@(bling), ketakid@(cros), govind@(desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Jan 25 2017
+ketakid to confirm comment 5
,
Jan 25 2017
govind@ do you have any concerns with this? Looks like a Chrome CL.
,
Jan 25 2017
It's a Chrome OS CL, which only affects the Kip device with a built-in modem
,
Jan 26 2017
benchan@ please go ahead and land on M57.
,
Jan 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/b8ac0838851ea69924c04698cfc162e7fe5a61b2 commit b8ac0838851ea69924c04698cfc162e7fe5a61b2 Author: Ben Chan <benchan@chromium.org> Date: Tue Jan 24 18:49:57 2017 chromeos-base/modem-watchdog: log watchdog PID instead of logger PID This CL fixes the log functions in chromeos-kip-modem-watchdog.sh to use the watchdog PID instead of the logger PID when outputting messages to the system log. This helps correlate messages produced by the same watchdog process. BUG= chromium:653979 BUG= chromium:684620 TEST=Tested the following on a Kip with a built-in Huawei ME936 modem: - Observed /var/log/messages for modem-watchdog messages. The messages produced by the same watchdog process have the same process ID. - Manually power cycle the modem and observd a new watchdog process is triggered. The messages produced by the new watchdog process have a different process ID. Change-Id: Ia741d273db08afcd26808622ac661be522f3c649 Reviewed-on: https://chromium-review.googlesource.com/431837 Commit-Ready: Ben Chan <benchan@chromium.org> Tested-by: Ben Chan <benchan@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> (cherry picked from commit b2418aacf04534edd91b529d27658dc3188e9a06) Reviewed-on: https://chromium-review.googlesource.com/433382 Commit-Queue: Ben Chan <benchan@chromium.org> [modify] https://crrev.com/b8ac0838851ea69924c04698cfc162e7fe5a61b2/overlay-kip/chromeos-base/modem-watchdog/files/chromeos-kip-modem-watchdog.sh [rename] https://crrev.com/b8ac0838851ea69924c04698cfc162e7fe5a61b2/overlay-kip/chromeos-base/modem-watchdog/modem-watchdog-0.0.1-r2.ebuild
,
Jan 26 2017
,
Jan 30 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Feb 2 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by benchan@chromium.org
, Jan 24 2017