New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 786514 link

Starred by 8 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

ChromeOS issue: ARC++ bridge device triggers Cisco Client Exclusion rules

Project Member Reported by vkhabarov@chromium.org, Nov 17 2017

Issue description

ChromeOS version: 60.0.3112.114+
ChromeOS device model: Setzer
Case#: 13956004

Description: When Chromebook connects to WiFi, it seems to be sending packets with source IP 100.115.92.1, which triggers Cisco Client Exclusion security rules and prevents from connecting to their network for a long time.

Steps to reproduce: 
1. Start Chromebook
2. Try to connect to network

Current Behavior / Reproduction: 
Chromebook sends packets from 100.115.92.1 before it gets DHCP address and is blacklisted

Expected Behavior:
Chromebook not sending packets from 100.115.92.1 to external network

Drive link to logs: 
https://drive.google.com/a/google.com/file/d/1ppP_yzuhQObGT0uN0TB25oRa8JSG4Klm/view?usp=sharing
Cisco logs - 
https://drive.google.com/a/google.com/file/d/1tkvdnYomT4LGKdPx64vhmIaQeO2gFN1H/view?usp=sharing

Info from customer network specialist - 
"Some more comments from our network specialist (translated):

That IP seems to be in some IANA-RESERVED IP-scope that are used for Carrier grade NAT at ISP:s… 

*apfOrphanSocketTask: Nov 08 11:18:31.959: [SA] 88:78:73:fc:e2:d9 Orphan Packet from STA - IP 100.115.92.1
*apfOrphanSocketTask: Nov 08 11:18:31.959: [SA] 88:78:73:fc:e2:d9 apfBlacklistMobileStationEntry2 (apf_ms.c:6582) Changing state for mobile 88:78:73:fc:e2:d9 on AP d0:57:4c:56:4d:10 from Associated to Exclusion-list (1)

WLC expects the client to be on the interface below...

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Mobility query, PEM State: L2AUTHCOMPLETE

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Building Mobile Announce : 

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Building Client Payload:

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Client Ip: 0.0.0.0

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Client Vlan Ip: 10.110.16.14, Vlan mask : 255.255.240.0 

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Client Vap Security: 16384

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Virtual Ip: 1.1.1.2

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 ssid: test-vxo

*Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Building VlanIpPayload.

But it don’t get a chance to go through the DHCP process before it starts communicating on 10.115.92.1 IP above..."

 

Comment 1 by defo...@kmsd.edu, Nov 17 2017

We have been seeing this on our network as well and it's only on systems with the Android subsystem present.  We pulled a list of systems pulling this error and compared it against an export from our Domain and they were all systems capable of running Android apps.

I investigated it a little bit and found that I could put the device in developer mode and change the IP on the bridge and it didn't seem to break anything. I also played with removing the IP completely and it didn't seem to break anything either.  After reading up on Linux bridges, it sounds like you don't need an IP unless you are doing Multicast, which very well might be what's happening in this case and is why they chose to assign an IP.
We haven't been able to verify if this behavior is causing a delay in attaching to the network, but we're assuming that when we sometimes have systems that take a long time to connect, that this is probably what is happening in the background.

This issue seems to be causing problems with doing host scans on the subnet from the Android subsystem as well:
https://github.com/aaronjwood/PortAuthority/issues/78

There is an active Cisco forum discussing this problem as well:
https://supportforums.cisco.com/t5/security-and-network-management/prime-3-1-reports-100-115-xxx-xxx-addresses-which-are-not-part/td-p/3205899
Components: -Internals>Network -Internals>Services>Network Platform>ARC
Internals>Network and Internals>Services>Network are relating to the browser net stack, while this sounds like an issue with ARC++. Removing those labels and adding ARC label.

Comment 3 by jayhlee@google.com, Nov 29 2017

Owner: elijahtaylor@chromium.org
Elijah, is there someone on your team that can take a look? Why would these internal IPs be exposed on the external network?
Cc: abhishekbh@chromium.org
Owner: cernekee@chromium.org
Kevin, can you take a look at this?
Cc: alberto@chromium.org smbar...@chromium.org
> Why would these internal IPs be exposed on the external network?

I would not expect this to be the case.  Suggest attaching .pcap files containing the offending packets.

(I'll try it on my own later but there's no guarantee that I can reproduce it.)

> That IP seems to be in some IANA-RESERVED IP-scope that are used for Carrier grade NAT at ISP:s

This is unfortunate if true.  We specifically requested a range of *public* Google-owned IPv4 addresses, so that we could guarantee that there would never be a conflict with any IANA-RESERVED addresses that customers used for their LANs/WANs.

+Alberto, who might be able to help us allocate a new range.

Comment 6 by defo...@kmsd.edu, Nov 30 2017

If you read what we believe is happening - the Android subsystem seems to on initial connection to the network - present the IP of 100.115.92.1 to the wireless controller.  Because it does this, the controller blacklists the device because it has seen other Chromebooks present that same IP and it believes an IP conflict is occurring. The default period of blacklist is 60 seconds.
The issue seems to be more that the Android subsystem is presenting the bridge IP of 100.115.92.1 on the wireless interface prior to ChromeOS trying to make a connection than the fact that the IP is the same on all Chromeooks.  If there is a way to make sure that ChromeOS fully connects to wireless prior to initializing the Android subsystem's network - that will likely fix the issue.  That, or, possibly looking at removing the IP address from the bridge interface (which from reading up on Linux bridges, might not be required).  One would not expect to be able to see the IP of a bridge interface ever on the outside of a device.  It should be completely transparent.
Sitting at the login screen after a reboot (no Android code running at all), connected to an open wifi AP, I saw the attached IGMP join/leave requests.  Some of these have arcbr0's 100.115.92.1 IP.  Are these the packets that are causing an issue?  And if so, do you ever see bad join requests, or only bad leave requests?

(I'm testing on a near-ToT samus, 3.14 kernel.)

I was able to suppress them with:

    iptables -I OUTPUT -o wlan0 -s 100.115.92.0/24 -j DROP

These appear to be related to multicast subscriptions for the mDNS service.  `restart avahi` generates a handful of IGMP membership reports, although they mostly seem to have the correct IP.  I can trigger it to send the "bad" membership reports by running this while connected to the open AP:

    tcpdump -n -i wlan0 igmp &
    ifconfig wlan0 0.0.0.0

This coincides with:

    2017-12-02T21:49:23.493898-08:00 INFO avahi-daemon[2025]: Withdrawing address record for 100.110.200.207 on wlan0.
    2017-12-02T21:49:23.493976-08:00 INFO avahi-daemon[2025]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 100.110.200.207.
    2017-12-02T21:49:23.494032-08:00 INFO avahi-daemon[2025]: Interface wlan0.IPv4 no longer relevant for mDNS.

If I run `killall -STOP avahi-daemon`, the IGMP activity in question is deferred until I run `killall -CONT avahi-daemon`.

I did not see any 100.115.92.x packets on wlan0 when the user was logged in and Android was running.  So if these IGMP packets are the source of the problem, I would expect that the problem would persist even if the Play Store was disabled via admin.google.com. :-(


FWIW we are running avahi 0.6.32, which is one release behind.  The changelog for v0.7 does not indicate that it has a fix for this:

https://github.com/lathiat/avahi/releases/tag/v0.7
igmp.pcap
444 bytes Download

Comment 8 by alberto@google.com, Dec 4 2017

Kevin, let me know if we still need a new range. thx

Comment 9 by defo...@kmsd.edu, Dec 4 2017

We have only seen the issue on our wireless network on machines that are running the Android subsystem.  I checked all of the MAC addresses vs our inventory list and all were machines capable of running Android - none were old models.

If I were to guess, the issue is likely occurring prior to any DHCP requests on network association.  This could occur on initial bootup or likely on any user login as well (as we use different 802.1x credentials for pre-login and post-login).
The 100.115.92.1 IP address will always be configured on Chromebooks that support Android apps, even if Android support has been disabled.  The repro case in c#7 happens at the login screen when the Android OS is not running.

Do we have any way to figure out whether the IGMP packet in c#7 is what is triggering the block on the Cisco hardware?  (If it isn't, then any steps taken to stop the IGMP packet from being sent will probably not address the real issue.)
Cc: marcore@chromium.org
Labels: -Pri-2 Pri-1
Project Member

Comment 13 by bugdroid1@chromium.org, Dec 14 2017

Labels: merge-merged-chromeos-4.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2dc6db8f0927923da42e67ba1736c137908f61fa

commit 2dc6db8f0927923da42e67ba1736c137908f61fa
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Thu Dec 14 06:33:07 2017

FROMGIT: net: igmp: Use correct source address on IGMPv3 reports

Closing a multicast socket after the final IPv4 address is deleted
from an interface can generate a membership report that uses the
source IP from a different interface.  The following test script, run
from an isolated netns, reproduces the issue:

    #!/bin/bash

    ip link add dummy0 type dummy
    ip link add dummy1 type dummy
    ip link set dummy0 up
    ip link set dummy1 up
    ip addr add 10.1.1.1/24 dev dummy0
    ip addr add 192.168.99.99/24 dev dummy1

    tcpdump -U -i dummy0 &
    socat EXEC:"sleep 2" \
        UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 &

    sleep 1
    ip addr del 10.1.1.1/24 dev dummy0
    sleep 5
    kill %tcpdump

RFC 3376 specifies that the report must be sent with a valid IP source
address from the destination subnet, or from address 0.0.0.0.  Add an
extra check to make sure this is the case.

Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
(cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0)

BUG= chromium:786514 
TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network

Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389
Reviewed-on: https://chromium-review.googlesource.com/825922
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Guenter Roeck <groeck@chromium.org>

[modify] https://crrev.com/2dc6db8f0927923da42e67ba1736c137908f61fa/net/ipv4/igmp.c

Project Member

Comment 14 by bugdroid1@chromium.org, Dec 14 2017

Labels: merge-merged-chromeos-3.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/05dcc3c7bbe79e2b0e4fc09dee96cffbac592c5e

commit 05dcc3c7bbe79e2b0e4fc09dee96cffbac592c5e
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Thu Dec 14 06:33:15 2017

FROMGIT: net: igmp: Use correct source address on IGMPv3 reports

Closing a multicast socket after the final IPv4 address is deleted
from an interface can generate a membership report that uses the
source IP from a different interface.  The following test script, run
from an isolated netns, reproduces the issue:

    #!/bin/bash

    ip link add dummy0 type dummy
    ip link add dummy1 type dummy
    ip link set dummy0 up
    ip link set dummy1 up
    ip addr add 10.1.1.1/24 dev dummy0
    ip addr add 192.168.99.99/24 dev dummy1

    tcpdump -U -i dummy0 &
    socat EXEC:"sleep 2" \
        UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 &

    sleep 1
    ip addr del 10.1.1.1/24 dev dummy0
    sleep 5
    kill %tcpdump

RFC 3376 specifies that the report must be sent with a valid IP source
address from the destination subnet, or from address 0.0.0.0.  Add an
extra check to make sure this is the case.

Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
(cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0)

BUG= chromium:786514 
TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network

Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389
Reviewed-on: https://chromium-review.googlesource.com/825773
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Guenter Roeck <groeck@chromium.org>

[modify] https://crrev.com/05dcc3c7bbe79e2b0e4fc09dee96cffbac592c5e/net/ipv4/igmp.c

Project Member

Comment 15 by bugdroid1@chromium.org, Dec 14 2017

Labels: merge-merged-chromeos-4.4
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2606df717ce796c98dc2c6a03adfd353a38e3fc9

commit 2606df717ce796c98dc2c6a03adfd353a38e3fc9
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Thu Dec 14 06:33:19 2017

FROMGIT: net: igmp: Use correct source address on IGMPv3 reports

Closing a multicast socket after the final IPv4 address is deleted
from an interface can generate a membership report that uses the
source IP from a different interface.  The following test script, run
from an isolated netns, reproduces the issue:

    #!/bin/bash

    ip link add dummy0 type dummy
    ip link add dummy1 type dummy
    ip link set dummy0 up
    ip link set dummy1 up
    ip addr add 10.1.1.1/24 dev dummy0
    ip addr add 192.168.99.99/24 dev dummy1

    tcpdump -U -i dummy0 &
    socat EXEC:"sleep 2" \
        UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 &

    sleep 1
    ip addr del 10.1.1.1/24 dev dummy0
    sleep 5
    kill %tcpdump

RFC 3376 specifies that the report must be sent with a valid IP source
address from the destination subnet, or from address 0.0.0.0.  Add an
extra check to make sure this is the case.

Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
(cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0)

BUG= chromium:786514 
TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network

Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389
Reviewed-on: https://chromium-review.googlesource.com/825778
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Guenter Roeck <groeck@chromium.org>

[modify] https://crrev.com/2606df717ce796c98dc2c6a03adfd353a38e3fc9/net/ipv4/igmp.c

Project Member

Comment 16 by bugdroid1@chromium.org, Dec 14 2017

Labels: merge-merged-chromeos-3.18
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/75f317e5cefd60a3cb6e306063530a0cf283893f

commit 75f317e5cefd60a3cb6e306063530a0cf283893f
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Thu Dec 14 06:33:11 2017

FROMGIT: net: igmp: Use correct source address on IGMPv3 reports

Closing a multicast socket after the final IPv4 address is deleted
from an interface can generate a membership report that uses the
source IP from a different interface.  The following test script, run
from an isolated netns, reproduces the issue:

    #!/bin/bash

    ip link add dummy0 type dummy
    ip link add dummy1 type dummy
    ip link set dummy0 up
    ip link set dummy1 up
    ip addr add 10.1.1.1/24 dev dummy0
    ip addr add 192.168.99.99/24 dev dummy1

    tcpdump -U -i dummy0 &
    socat EXEC:"sleep 2" \
        UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 &

    sleep 1
    ip addr del 10.1.1.1/24 dev dummy0
    sleep 5
    kill %tcpdump

RFC 3376 specifies that the report must be sent with a valid IP source
address from the destination subnet, or from address 0.0.0.0.  Add an
extra check to make sure this is the case.

Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
(cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0)

BUG= chromium:786514 
TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network

Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389
Reviewed-on: https://chromium-review.googlesource.com/825654
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Guenter Roeck <groeck@chromium.org>

[modify] https://crrev.com/75f317e5cefd60a3cb6e306063530a0cf283893f/net/ipv4/igmp.c

Status: Fixed (was: Untriaged)
The IGMPv3 source address change is verified on samus canary 10215.0.0.  The kernel patch has also been accepted upstream.

This fix will not affect IGMPv1 or IGMPv2.  Based on a cursory glance at the code, I don't think IGMPv1/IGMPv2 will normally be used unless the kernel detects IGMPv1/IGMPv2 multicasts from another host on the LAN.

If you're still seeing the problem on >=10215.0.0, please attach a .pcap file showing the packet(s) at fault, so their source can be tracked down.
Project Member

Comment 18 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/268af7ab4188b9c99b8d28f67d2d41698c27ac93

commit 268af7ab4188b9c99b8d28f67d2d41698c27ac93
Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
Date: Fri Feb 02 08:23:29 2018

UPSTREAM: net: igmp: fix source address check for IGMPv3 reports

[ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ]

Commit "net: igmp: Use correct source address on IGMPv3 reports"
introduced a check to validate the source address of locally generated
IGMPv3 packets.
Instead of checking the local interface address directly, it uses
inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the
local subnet (or equal to the point-to-point address if used).

This breaks for point-to-point interfaces, so check against
ifa->ifa_local directly.

Cc: Kevin Cernekee <cernekee@chromium.org>
Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Tested-by: Florian Wolters <florian@florian-wolters.de>
(cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1
Reviewed-on: https://chromium-review.googlesource.com/898213
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/268af7ab4188b9c99b8d28f67d2d41698c27ac93/net/ipv4/igmp.c

Project Member

Comment 19 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b2ffb7f761f2a0421aea41ec6c5b6c6762cc9522

commit b2ffb7f761f2a0421aea41ec6c5b6c6762cc9522
Author: Eric Dumazet <edumazet@google.com>
Date: Fri Feb 02 08:23:32 2018

FROMGIT: net: igmp: add a missing rcu locking section

Newly added igmpv3_get_srcaddr() needs to be called under rcu lock.

Timer callbacks do not ensure this locking.

=============================
WARNING: suspicious RCU usage
4.15.0+ #200 Not tainted
-----------------------------
./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

rcu_scheduler_active = 2, debug_locks = 1
3 locks held by syzkaller616973/4074:
 #0:  (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline]
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline]
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600

stack backtrace:
CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592
 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline]
 igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline]
 igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389
 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432
 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565
 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605
 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722
 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831
 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326
 expire_timers kernel/time/timer.c:1363 [inline]
 __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666
 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692
 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
 invoke_softirq kernel/softirq.c:365 [inline]
 irq_exit+0x1cc/0x200 kernel/softirq.c:405
 exiting_irq arch/x86/include/asm/apic.h:541 [inline]
 smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052
 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938

Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: syzbot <syzkaller@googlegroups.com>
(cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e
Reviewed-on: https://chromium-review.googlesource.com/898214
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/b2ffb7f761f2a0421aea41ec6c5b6c6762cc9522/net/ipv4/igmp.c

Project Member

Comment 20 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e444947eb5150954580cc1bb3f11122ff7dc7416

commit e444947eb5150954580cc1bb3f11122ff7dc7416
Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
Date: Fri Feb 02 18:17:11 2018

UPSTREAM: net: igmp: fix source address check for IGMPv3 reports

[ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ]

Commit "net: igmp: Use correct source address on IGMPv3 reports"
introduced a check to validate the source address of locally generated
IGMPv3 packets.
Instead of checking the local interface address directly, it uses
inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the
local subnet (or equal to the point-to-point address if used).

This breaks for point-to-point interfaces, so check against
ifa->ifa_local directly.

Cc: Kevin Cernekee <cernekee@chromium.org>
Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Tested-by: Florian Wolters <florian@florian-wolters.de>
(cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1
Reviewed-on: https://chromium-review.googlesource.com/898266
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/e444947eb5150954580cc1bb3f11122ff7dc7416/net/ipv4/igmp.c

Project Member

Comment 21 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/1d6f8fa079fc495a13b28769cfa8bf75cfd40226

commit 1d6f8fa079fc495a13b28769cfa8bf75cfd40226
Author: Eric Dumazet <edumazet@google.com>
Date: Fri Feb 02 18:17:14 2018

FROMGIT: net: igmp: add a missing rcu locking section

Newly added igmpv3_get_srcaddr() needs to be called under rcu lock.

Timer callbacks do not ensure this locking.

=============================
WARNING: suspicious RCU usage
4.15.0+ #200 Not tainted
-----------------------------
./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

rcu_scheduler_active = 2, debug_locks = 1
3 locks held by syzkaller616973/4074:
 #0:  (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline]
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline]
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600

stack backtrace:
CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592
 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline]
 igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline]
 igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389
 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432
 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565
 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605
 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722
 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831
 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326
 expire_timers kernel/time/timer.c:1363 [inline]
 __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666
 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692
 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
 invoke_softirq kernel/softirq.c:365 [inline]
 irq_exit+0x1cc/0x200 kernel/softirq.c:405
 exiting_irq arch/x86/include/asm/apic.h:541 [inline]
 smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052
 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938

Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: syzbot <syzkaller@googlegroups.com>
(cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e
Reviewed-on: https://chromium-review.googlesource.com/898267
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/1d6f8fa079fc495a13b28769cfa8bf75cfd40226/net/ipv4/igmp.c

Project Member

Comment 22 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/96307b9462b2bc8240ec2ee5899bc9309789bcb2

commit 96307b9462b2bc8240ec2ee5899bc9309789bcb2
Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
Date: Fri Feb 02 18:17:00 2018

UPSTREAM: net: igmp: fix source address check for IGMPv3 reports

[ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ]

Commit "net: igmp: Use correct source address on IGMPv3 reports"
introduced a check to validate the source address of locally generated
IGMPv3 packets.
Instead of checking the local interface address directly, it uses
inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the
local subnet (or equal to the point-to-point address if used).

This breaks for point-to-point interfaces, so check against
ifa->ifa_local directly.

Cc: Kevin Cernekee <cernekee@chromium.org>
Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Tested-by: Florian Wolters <florian@florian-wolters.de>
(cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1
Reviewed-on: https://chromium-review.googlesource.com/897408
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/96307b9462b2bc8240ec2ee5899bc9309789bcb2/net/ipv4/igmp.c

Project Member

Comment 23 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6528f26d01d6fd2869107bd84d3c8ee160015b56

commit 6528f26d01d6fd2869107bd84d3c8ee160015b56
Author: Eric Dumazet <edumazet@google.com>
Date: Fri Feb 02 18:17:03 2018

FROMGIT: net: igmp: add a missing rcu locking section

Newly added igmpv3_get_srcaddr() needs to be called under rcu lock.

Timer callbacks do not ensure this locking.

=============================
WARNING: suspicious RCU usage
4.15.0+ #200 Not tainted
-----------------------------
./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

rcu_scheduler_active = 2, debug_locks = 1
3 locks held by syzkaller616973/4074:
 #0:  (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline]
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline]
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600

stack backtrace:
CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592
 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline]
 igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline]
 igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389
 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432
 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565
 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605
 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722
 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831
 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326
 expire_timers kernel/time/timer.c:1363 [inline]
 __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666
 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692
 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
 invoke_softirq kernel/softirq.c:365 [inline]
 irq_exit+0x1cc/0x200 kernel/softirq.c:405
 exiting_irq arch/x86/include/asm/apic.h:541 [inline]
 smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052
 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938

Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: syzbot <syzkaller@googlegroups.com>
(cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e
Reviewed-on: https://chromium-review.googlesource.com/897409
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/6528f26d01d6fd2869107bd84d3c8ee160015b56/net/ipv4/igmp.c

Project Member

Comment 24 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/04aa472ba1b6973ce55954ff8866433e1593fbd8

commit 04aa472ba1b6973ce55954ff8866433e1593fbd8
Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
Date: Fri Feb 02 18:17:07 2018

UPSTREAM: net: igmp: fix source address check for IGMPv3 reports

[ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ]

Commit "net: igmp: Use correct source address on IGMPv3 reports"
introduced a check to validate the source address of locally generated
IGMPv3 packets.
Instead of checking the local interface address directly, it uses
inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the
local subnet (or equal to the point-to-point address if used).

This breaks for point-to-point interfaces, so check against
ifa->ifa_local directly.

Cc: Kevin Cernekee <cernekee@chromium.org>
Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Tested-by: Florian Wolters <florian@florian-wolters.de>
(cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1
Reviewed-on: https://chromium-review.googlesource.com/897406
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/04aa472ba1b6973ce55954ff8866433e1593fbd8/net/ipv4/igmp.c

Project Member

Comment 25 by bugdroid1@chromium.org, Feb 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d25b501bf1ca79c678613b73327e7a2317a73fd1

commit d25b501bf1ca79c678613b73327e7a2317a73fd1
Author: Eric Dumazet <edumazet@google.com>
Date: Fri Feb 02 18:17:09 2018

FROMGIT: net: igmp: add a missing rcu locking section

Newly added igmpv3_get_srcaddr() needs to be called under rcu lock.

Timer callbacks do not ensure this locking.

=============================
WARNING: suspicious RCU usage
4.15.0+ #200 Not tainted
-----------------------------
./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

rcu_scheduler_active = 2, debug_locks = 1
3 locks held by syzkaller616973/4074:
 #0:  (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline]
 #1:  ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline]
 #2:  (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600

stack backtrace:
CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592
 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline]
 igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline]
 igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389
 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432
 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565
 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605
 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722
 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831
 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326
 expire_timers kernel/time/timer.c:1363 [inline]
 __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666
 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692
 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
 invoke_softirq kernel/softirq.c:365 [inline]
 irq_exit+0x1cc/0x200 kernel/softirq.c:405
 exiting_irq arch/x86/include/asm/apic.h:541 [inline]
 smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052
 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938

Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: syzbot <syzkaller@googlegroups.com>
(cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5)

BUG= chromium:786514 
TEST=buildbots

Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e
Reviewed-on: https://chromium-review.googlesource.com/897407
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/d25b501bf1ca79c678613b73327e7a2317a73fd1/net/ipv4/igmp.c

Comment 26 by defo...@kmsd.edu, Mar 16 2018

It sounds like we shouldn't be seeing this problem anymore, but it's still happening.  I have 100s of alerts for client exclusion again with an IP of 100.115.92.1.  This seemed to be better for awhile and now it's back.

Please let us know what information you would need in order to keep digging into why this is still happening.  I'm seeing the alerts on a Cisco Prime Infrastructure server coming from Chrome devices.
M65 should have the fix.  If you're still seeing alerts, could you please attach a packet capture so we can see whether the culprit is IGMPv3 or something else?

Comment 28 by defo...@kmsd.edu, Mar 16 2018

Is it a new fix for M65?  I see that M65 shows March 13th as the release date for the stable build - so there are likely computers that haven't updated yet.

I'll have to think about how I could grab a packet capture.  I'd have to dig into MAC addresses to determine if these are coming from our devices or from devices that we don't own.

In all honesty - I'm sick of being a network tester for Google on all this stuff. I'm participating in at least 3 different bug forums right now for various network issues that we're experiencing with Chromebooks right now and I have never had to do anything like this for any other platform that hits our network.  Everything from 802.1x, to this and DHCP issues.

If I can track down a device with the issue - what's your recommendation for doing a packet capture on it?
> Is it a new fix for M65?

Yes.  The CL went into ToT in mid-December.  M64 branched Nov 30.  M65 branched Jan 18.  So it's in M65 but not M64.

> I'll have to think about how I could grab a packet capture.

Is it possible to connect a Linux PC to a switch port that can see all traffic (e.g. via SPAN) and run something like `tcpdump -n -i eth1 -w /tmp/badip.pcap net 100.115.92.0/24` to capture any traffic that uses the reserved IP range?

Sign in to add a comment