New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 655332 link

Starred by 1 user
Status: Verified
Owner:
groeck@chromium.org
Closed: Oct 2016
Cc:
bleung@chromium.org
ka...@chromium.org
sontis@chromium.org
█ yoshiat@chromium.org
█ kirtika@chromium.org
█ cernekee@chromium.org
█ tienchang@chromium.org
█ jongkyuj@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

asix.ko: hung task when probing adapter

Project Member Reported by aashuto...@chromium.org, Oct 12 2016 
Here is sample failure,

<0>[62278.200057] Kernel panic - not syncing: hung_task: blocked tasks
<4>[62278.200069] CPU: 1 PID: 31 Comm: khungtaskd Tainted: G        W      3.18.0-13298-g94ace9e #1
<4>[62278.200081] Hardware name: GOOGLE Cyan, BIOS Google_Cyan.7287.57.64 05/20/2016
<4>[62278.200092]  0000000000000000 00000000a14ef563 ffff88017a847da8 ffffffff8e093077
<4>[62278.200108]  0000000000000000 ffffffff8e3a5969 ffff88017a847e28 ffffffff8e091d26
<4>[62278.200124]  ffff880100000008 ffff88017a847e38 ffff88017a847dd8 00000000a14ef563
<4>[62278.200139] Call Trace:
<4>[62278.200149]  [<ffffffff8e093077>] dump_stack+0x4e/0x71
<4>[62278.200160]  [<ffffffff8e091d26>] panic+0xcd/0x1e3
<4>[62278.200170]  [<ffffffff8dadde5d>] watchdog+0x282/0x300
<4>[62278.200181]  [<ffffffff8daddbdb>] ? reset_hung_task_detector+0x15/0x15
<4>[62278.200193]  [<ffffffff8da7c3d2>] kthread+0x10e/0x116
<4>[62278.200204]  [<ffffffff8da7c2c4>] ? __kthread_parkme+0x67/0x67
<4>[62278.200215]  [<ffffffff8e0981ec>] ret_from_fork+0x7c/0xb0
<4>[62278.200226]  [<ffffffff8da7c2c4>] ? __kthread_parkme+0x67/0x67
<0>[62278.200327] Kernel Offset: 0xca00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
<0>[62278.203887] gsmi: Log Shutdown Reason 0x02

logs attached.
log-101216-144208.tar.gz
2.4 MB Download
kernel.20161012.103806.0.kcrash
182 KB Download
kernel.20161011.171932.0.meta
166 bytes Download
kernel.20161011.171932.0.kcrash
259 KB Download
kernel.20161012.103806.0.meta
166 bytes Download

Comment 1 by aashuto...@chromium.org, Oct 12 2016 

Cyan 8872.6.0 R-55

Comment 2 by ka...@chromium.org, Oct 12 2016

Labels: Stability-Crash
Can you add more details about how this crash occurred?

Comment 3 by aashuto...@chromium.org, Oct 12 2016 

I don't have any more details. If I see any pattern, will update.

Comment 4 by dtor@chromium.org, Oct 12 2016

Summary: asix.ko: hung task when probing adapter (was: Cyan is seeing kernel panic . )
Looks like USB ethernet dongle is unhappy:

<6>[62108.732707] usb 1-3: new high-speed USB device number 5 using xhci_hcd
<6>[62108.914421] usb 1-3: New USB device found, idVendor=0b95, idProduct=7720
<6>[62108.914463] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
<6>[62108.914476] usb 1-3: Product: AX88x72A
<6>[62108.914486] usb 1-3: Manufacturer: ASIX Elec. Corp.
<6>[62108.914495] usb 1-3: SerialNumber: 000001
<4>[62114.109109] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to write reg index 0x0000: -110
<3>[62114.109139] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to send software reset: ffffff92
<4>[62119.109048] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to write reg index 0x0000: -110
<3>[62119.109085] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to enable software MII access
<4>[62124.109968] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to read reg index 0x0000: -110
<4>[62129.109533] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to write reg index 0x0000: -110
<3>[62129.109583] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to enable software MII access
<4>[62134.110823] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to read reg index 0x0000: -110
<27>[62138.948230] udevd[140]: worker [27037] /devices/pci0000:00/0000:00:14.0/usb1/1-3 timeout; kill it
<27>[62138.948341] udevd[140]: seq 1626 '/devices/pci0000:00/0000:00:14.0/usb1/1-3' killed
<4>[62139.110829] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to write reg index 0x0000: -110
<3>[62139.110851] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to enable software MII access
<4>[62144.111936] asix 1-3:1.0 (unnamed net_device) (uninitialized): Failed to read reg index 0x0000: -110
...

<6>[62278.180957] kworker/2:0     D 0000000000000000     0  5744      2 0x00000000
<6>[62278.180978] Workqueue: usb_hub_wq hub_event
<4>[62278.181029]  ffff880177f833b8 0000000000000046 ffff88017fd00000 ffff88017b126d80
<4>[62278.181048]  ffff880177f83fd8 ffff880065a71b60 0000000000013340 ffff880065a71b60
<4>[62278.181065]  0000000000000286 0000000103b1c199 0000000000001388 0000000000000002
<4>[62278.181081] Call Trace:
<4>[62278.181092]  [<ffffffff8e0971fd>] ? console_conditional_schedule+0x2c/0x2c
<4>[62278.181105]  [<ffffffff8e094f7b>] schedule+0x69/0x6b
<4>[62278.181117]  [<ffffffff8e0972e0>] schedule_timeout+0xe3/0x11d
<4>[62278.181133]  [<ffffffff8daadb1b>] ? trace_timer_start+0x51/0x51
<4>[62278.181146]  [<ffffffff8e095a05>] do_wait_for_common+0x12f/0x16c
<4>[62278.181162]  [<ffffffff8da856a7>] ? wake_up_process+0x39/0x39
<4>[62278.181174]  [<ffffffff8e095aee>] wait_for_common+0x52/0x6d
<4>[62278.181187]  [<ffffffff8e095b3b>] wait_for_completion_timeout+0x13/0x15
<4>[62278.181201]  [<ffffffff8de676ce>] usb_start_wait_urb+0x93/0xf1
<4>[62278.181214]  [<ffffffff8de6780d>] usb_control_msg+0xe1/0x11d
<4>[62278.181230]  [<ffffffffc037d629>] usbnet_write_cmd+0x9c/0xc6 [usbnet]
<4>[62278.181286]  [<ffffffffc03af793>] asix_write_cmd+0x4e/0x7e [asix]
<4>[62278.181300]  [<ffffffffc03afb41>] asix_set_sw_mii+0x25/0x4e [asix]
<4>[62278.181314]  [<ffffffffc03b001d>] asix_mdio_read+0x51/0x109 [asix]
<4>[62278.181329]  [<ffffffffc03ae9ea>] 0xffffffffc03ae9ea
<4>[62278.181340]  [<ffffffffc03af50f>] 0xffffffffc03af50f
<4>[62278.181354]  [<ffffffffc037e0d1>] usbnet_probe+0x2c1/0x7f6 [usbnet]
<4>[62278.181373]  [<ffffffff8de6a8fb>] usb_probe_interface+0x17a/0x21b
<4>[62278.181389]  [<ffffffff8ddff760>] driver_probe_device+0x109/0x2b1
<4>[62278.181403]  [<ffffffff8ddff968>] __device_attach_driver+0x60/0x68
<4>[62278.181415]  [<ffffffff8ddff908>] ? driver_probe_device+0x2b1/0x2b1
<4>[62278.181430]  [<ffffffff8ddfdb76>] bus_for_each_drv+0x87/0xaa
<4>[62278.181444]  [<ffffffff8ddffaa1>] __device_attach+0xb0/0x126
<4>[62278.181456]  [<ffffffff8ddffb3c>] device_initial_probe+0x13/0x15
<4>[62278.181468]  [<ffffffff8ddfebf3>] bus_probe_device+0x34/0x98
<4>[62278.181482]  [<ffffffff8ddfce3e>] device_add+0x32e/0x545
<4>[62278.181495]  [<ffffffff8de69059>] usb_set_configuration+0x625/0x675
<4>[62278.181508]  [<ffffffff8de71d9d>] generic_probe+0x45/0x77
<4>[62278.181520]  [<ffffffff8de6a76e>] usb_probe_device+0x2d/0x40
<4>[62278.181532]  [<ffffffff8ddff760>] driver_probe_device+0x109/0x2b1
<4>[62278.181544]  [<ffffffff8ddff968>] __device_attach_driver+0x60/0x68
<4>[62278.181556]  [<ffffffff8ddff908>] ? driver_probe_device+0x2b1/0x2b1
<4>[62278.181568]  [<ffffffff8ddfdb76>] bus_for_each_drv+0x87/0xaa
<4>[62278.181580]  [<ffffffff8ddffaa1>] __device_attach+0xb0/0x126
<4>[62278.181591]  [<ffffffff8ddffb3c>] device_initial_probe+0x13/0x15
<4>[62278.181604]  [<ffffffff8ddfebf3>] bus_probe_device+0x34/0x98
<4>[62278.181619]  [<ffffffff8ddfce3e>] device_add+0x32e/0x545
<4>[62278.181633]  [<ffffffff8de6122a>] usb_new_device+0x2d5/0x419
<4>[62278.181648]  [<ffffffff8de62875>] hub_event+0xbb9/0xf7a
<4>[62278.181665]  [<ffffffff8da77983>] process_one_work+0x176/0x2d4
<4>[62278.181681]  [<ffffffff8da78308>] worker_thread+0x1ec/0x2bf
<4>[62278.181697]  [<ffffffff8da7811c>] ? rescuer_thread+0x2db/0x2db
<4>[62278.181718]  [<ffffffff8da7c3d2>] kthread+0x10e/0x116
<4>[62278.181733]  [<ffffffff8da7c2c4>] ? __kthread_parkme+0x67/0x67
<4>[62278.181748]  [<ffffffff8e0981ec>] ret_from_fork+0x7c/0xb0
<4>[62278.181763]  [<ffffffff8da7c2c4>] ? __kthread_parkme+0x67/0x67

Comment 5 by aashuto...@chromium.org, Oct 12 2016

Components: OS>Systems>Network
Owner: snanda@chromium.org
based on comment#4.

Comment 6 by tienchang@chromium.org, Oct 13 2016

Cc: yoshiat@chromium.org
Labels: -Pri-3 Pri-2

Comment 7 by snanda@chromium.org, Oct 13 2016

Cc: kirtika@chromium.org cernekee@chromium.org
Was this a one-off or a more repeatable failure?

Comment 8 by aashuto...@chromium.org, Oct 13 2016 

I saw this twice in  two days. I have attached both the crash logs. If I see any more similar crashes on same or different devices, will update this issue.

Comment 9 by kirtika@chromium.org, Oct 13 2016

Cc: groeck@chromium.org bleung@chromium.org
CC-ing Benson and Guenter to check if they know the appropriate USB owner.

Comment 10 by kirtika@chromium.org, Oct 13 2016

Cc: -groeck@chromium.org

Comment 11 by groeck@chromium.org, Oct 13 2016 

asix_mdio_read() should probably abort if it experiences -ETIMEDOUT (just like -ENODEV). That won't make the adapter functional, but at least it won't crash.

Also, asix_mdio_read() ignores errors other than -ENODEV, which doesn't look correct and may result in secondary errors.

Comment 12 by groeck@chromium.org, Oct 13 2016

Owner: groeck@chromium.org
Status: Started (was: Untriaged)
Project Member

Comment 13 by bugdroid1@chromium.org, Oct 17 2016

Labels: merge-merged-chromeos-4.4
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/17aa5e3a7c9ce73b0d1db009017877ddd31cc811

commit 17aa5e3a7c9ce73b0d1db009017877ddd31cc811
Author: Guenter Roeck <linux@roeck-us.net>
Date: Thu Oct 13 23:43:16 2016

UPSTREAM: net: asix: Avoid looping when the device does not respond

Check answers from USB stack and avoid re-sending the request
multiple times if the device does not respond.

This fixes the following problem, observed with a probably flaky adapter.

[62108.732707] usb 1-3: new high-speed USB device number 5 using xhci_hcd
[62108.914421] usb 1-3: New USB device found, idVendor=0b95, idProduct=7720
[62108.914463] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[62108.914476] usb 1-3: Product: AX88x72A
[62108.914486] usb 1-3: Manufacturer: ASIX Elec. Corp.
[62108.914495] usb 1-3: SerialNumber: 000001
[62114.109109] asix 1-3:1.0 (unnamed net_device) (uninitialized):
	Failed to write reg index 0x0000: -110
[62114.109139] asix 1-3:1.0 (unnamed net_device) (uninitialized):
	Failed to send software reset: ffffff92
[62119.109048] asix 1-3:1.0 (unnamed net_device) (uninitialized):
	Failed to write reg index 0x0000: -110
...

Since the USB timeout is 5 seconds, and the operation is retried 30 times,
this results in

[62278.180353] INFO: task mtpd:1725 blocked for more than 120 seconds.
[62278.180373]       Tainted: G        W      3.18.0-13298-g94ace9e #1
[62278.180383] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
...
[62278.180957] kworker/2:0     D 0000000000000000     0  5744      2 0x00000000
[62278.180978] Workqueue: usb_hub_wq hub_event
[62278.181029]  ffff880177f833b8 0000000000000046 ffff88017fd00000 ffff88017b126d80
[62278.181048]  ffff880177f83fd8 ffff880065a71b60 0000000000013340 ffff880065a71b60
[62278.181065]  0000000000000286 0000000103b1c199 0000000000001388 0000000000000002
[62278.181081] Call Trace:
[62278.181092]  [<ffffffff8e0971fd>] ? console_conditional_schedule+0x2c/0x2c
[62278.181105]  [<ffffffff8e094f7b>] schedule+0x69/0x6b
[62278.181117]  [<ffffffff8e0972e0>] schedule_timeout+0xe3/0x11d
[62278.181133]  [<ffffffff8daadb1b>] ? trace_timer_start+0x51/0x51
[62278.181146]  [<ffffffff8e095a05>] do_wait_for_common+0x12f/0x16c
[62278.181162]  [<ffffffff8da856a7>] ? wake_up_process+0x39/0x39
[62278.181174]  [<ffffffff8e095aee>] wait_for_common+0x52/0x6d
[62278.181187]  [<ffffffff8e095b3b>] wait_for_completion_timeout+0x13/0x15
[62278.181201]  [<ffffffff8de676ce>] usb_start_wait_urb+0x93/0xf1
[62278.181214]  [<ffffffff8de6780d>] usb_control_msg+0xe1/0x11d
[62278.181230]  [<ffffffffc037d629>] usbnet_write_cmd+0x9c/0xc6 [usbnet]
[62278.181286]  [<ffffffffc03af793>] asix_write_cmd+0x4e/0x7e [asix]
[62278.181300]  [<ffffffffc03afb41>] asix_set_sw_mii+0x25/0x4e [asix]
[62278.181314]  [<ffffffffc03b001d>] asix_mdio_read+0x51/0x109 [asix]
...

Signed-off-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: David S. Miller <davem@davemloft.net>

BUG= chromium:655332 
TEST=Build and run with affected adapter

(cherry picked from commit 610df1d2342d639ec87a8fb5290ca4799d91d59b)
Signed-off-by: Guenter Roeck <groeck@chromium.org>

Change-Id: I24ac0c13828db9e16ff7393d31ff82faa579e129
Reviewed-on: https://chromium-review.googlesource.com/399584
Commit-Ready: Guenter Roeck <groeck@chromium.org>
Tested-by: Guenter Roeck <groeck@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/17aa5e3a7c9ce73b0d1db009017877ddd31cc811/drivers/net/usb/asix_common.c

Comment 14 by groeck@chromium.org, Oct 17 2016

Status: Fixed (was: Started)

Comment 15 by ka...@chromium.org, Dec 1 2016

Status: Verified (was: Fixed)
Bulk-Verify
►

Sign in to add a comment