ChromeOS issue: ARC++ bridge device triggers Cisco Client Exclusion rules |
||||||||||||
Issue descriptionChromeOS version: 60.0.3112.114+ ChromeOS device model: Setzer Case#: 13956004 Description: When Chromebook connects to WiFi, it seems to be sending packets with source IP 100.115.92.1, which triggers Cisco Client Exclusion security rules and prevents from connecting to their network for a long time. Steps to reproduce: 1. Start Chromebook 2. Try to connect to network Current Behavior / Reproduction: Chromebook sends packets from 100.115.92.1 before it gets DHCP address and is blacklisted Expected Behavior: Chromebook not sending packets from 100.115.92.1 to external network Drive link to logs: https://drive.google.com/a/google.com/file/d/1ppP_yzuhQObGT0uN0TB25oRa8JSG4Klm/view?usp=sharing Cisco logs - https://drive.google.com/a/google.com/file/d/1tkvdnYomT4LGKdPx64vhmIaQeO2gFN1H/view?usp=sharing Info from customer network specialist - "Some more comments from our network specialist (translated): That IP seems to be in some IANA-RESERVED IP-scope that are used for Carrier grade NAT at ISP:s… *apfOrphanSocketTask: Nov 08 11:18:31.959: [SA] 88:78:73:fc:e2:d9 Orphan Packet from STA - IP 100.115.92.1 *apfOrphanSocketTask: Nov 08 11:18:31.959: [SA] 88:78:73:fc:e2:d9 apfBlacklistMobileStationEntry2 (apf_ms.c:6582) Changing state for mobile 88:78:73:fc:e2:d9 on AP d0:57:4c:56:4d:10 from Associated to Exclusion-list (1) WLC expects the client to be on the interface below... *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Mobility query, PEM State: L2AUTHCOMPLETE *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Building Mobile Announce : *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Building Client Payload: *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Client Ip: 0.0.0.0 *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Client Vlan Ip: 10.110.16.14, Vlan mask : 255.255.240.0 *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Client Vap Security: 16384 *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Virtual Ip: 1.1.1.2 *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 ssid: test-vxo *Dot1x_NW_MsgTask_1: Nov 08 11:18:31.070: [SA] 88:78:73:fc:e2:d9 Building VlanIpPayload. But it don’t get a chance to go through the DHCP process before it starts communicating on 10.115.92.1 IP above..."
,
Nov 20 2017
Internals>Network and Internals>Services>Network are relating to the browser net stack, while this sounds like an issue with ARC++. Removing those labels and adding ARC label.
,
Nov 29 2017
Elijah, is there someone on your team that can take a look? Why would these internal IPs be exposed on the external network?
,
Nov 30 2017
Kevin, can you take a look at this?
,
Nov 30 2017
> Why would these internal IPs be exposed on the external network? I would not expect this to be the case. Suggest attaching .pcap files containing the offending packets. (I'll try it on my own later but there's no guarantee that I can reproduce it.) > That IP seems to be in some IANA-RESERVED IP-scope that are used for Carrier grade NAT at ISP:s This is unfortunate if true. We specifically requested a range of *public* Google-owned IPv4 addresses, so that we could guarantee that there would never be a conflict with any IANA-RESERVED addresses that customers used for their LANs/WANs. +Alberto, who might be able to help us allocate a new range.
,
Nov 30 2017
If you read what we believe is happening - the Android subsystem seems to on initial connection to the network - present the IP of 100.115.92.1 to the wireless controller. Because it does this, the controller blacklists the device because it has seen other Chromebooks present that same IP and it believes an IP conflict is occurring. The default period of blacklist is 60 seconds. The issue seems to be more that the Android subsystem is presenting the bridge IP of 100.115.92.1 on the wireless interface prior to ChromeOS trying to make a connection than the fact that the IP is the same on all Chromeooks. If there is a way to make sure that ChromeOS fully connects to wireless prior to initializing the Android subsystem's network - that will likely fix the issue. That, or, possibly looking at removing the IP address from the bridge interface (which from reading up on Linux bridges, might not be required). One would not expect to be able to see the IP of a bridge interface ever on the outside of a device. It should be completely transparent.
,
Dec 3 2017
Sitting at the login screen after a reboot (no Android code running at all), connected to an open wifi AP, I saw the attached IGMP join/leave requests. Some of these have arcbr0's 100.115.92.1 IP. Are these the packets that are causing an issue? And if so, do you ever see bad join requests, or only bad leave requests?
(I'm testing on a near-ToT samus, 3.14 kernel.)
I was able to suppress them with:
iptables -I OUTPUT -o wlan0 -s 100.115.92.0/24 -j DROP
These appear to be related to multicast subscriptions for the mDNS service. `restart avahi` generates a handful of IGMP membership reports, although they mostly seem to have the correct IP. I can trigger it to send the "bad" membership reports by running this while connected to the open AP:
tcpdump -n -i wlan0 igmp &
ifconfig wlan0 0.0.0.0
This coincides with:
2017-12-02T21:49:23.493898-08:00 INFO avahi-daemon[2025]: Withdrawing address record for 100.110.200.207 on wlan0.
2017-12-02T21:49:23.493976-08:00 INFO avahi-daemon[2025]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 100.110.200.207.
2017-12-02T21:49:23.494032-08:00 INFO avahi-daemon[2025]: Interface wlan0.IPv4 no longer relevant for mDNS.
If I run `killall -STOP avahi-daemon`, the IGMP activity in question is deferred until I run `killall -CONT avahi-daemon`.
I did not see any 100.115.92.x packets on wlan0 when the user was logged in and Android was running. So if these IGMP packets are the source of the problem, I would expect that the problem would persist even if the Play Store was disabled via admin.google.com. :-(
FWIW we are running avahi 0.6.32, which is one release behind. The changelog for v0.7 does not indicate that it has a fix for this:
https://github.com/lathiat/avahi/releases/tag/v0.7
,
Dec 4 2017
Kevin, let me know if we still need a new range. thx
,
Dec 4 2017
We have only seen the issue on our wireless network on machines that are running the Android subsystem. I checked all of the MAC addresses vs our inventory list and all were machines capable of running Android - none were old models. If I were to guess, the issue is likely occurring prior to any DHCP requests on network association. This could occur on initial bootup or likely on any user login as well (as we use different 802.1x credentials for pre-login and post-login).
,
Dec 4 2017
The 100.115.92.1 IP address will always be configured on Chromebooks that support Android apps, even if Android support has been disabled. The repro case in c#7 happens at the login screen when the Android OS is not running. Do we have any way to figure out whether the IGMP packet in c#7 is what is triggering the block on the Cisco hardware? (If it isn't, then any steps taken to stop the IGMP packet from being sent will probably not address the real issue.)
,
Dec 4 2017
,
Dec 4 2017
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2dc6db8f0927923da42e67ba1736c137908f61fa commit 2dc6db8f0927923da42e67ba1736c137908f61fa Author: Kevin Cernekee <cernekee@chromium.org> Date: Thu Dec 14 06:33:07 2017 FROMGIT: net: igmp: Use correct source address on IGMPv3 reports Closing a multicast socket after the final IPv4 address is deleted from an interface can generate a membership report that uses the source IP from a different interface. The following test script, run from an isolated netns, reproduces the issue: #!/bin/bash ip link add dummy0 type dummy ip link add dummy1 type dummy ip link set dummy0 up ip link set dummy1 up ip addr add 10.1.1.1/24 dev dummy0 ip addr add 192.168.99.99/24 dev dummy1 tcpdump -U -i dummy0 & socat EXEC:"sleep 2" \ UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 & sleep 1 ip addr del 10.1.1.1/24 dev dummy0 sleep 5 kill %tcpdump RFC 3376 specifies that the report must be sent with a valid IP source address from the destination subnet, or from address 0.0.0.0. Add an extra check to make sure this is the case. Signed-off-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Andrew Lunn <andrew@lunn.ch> Signed-off-by: David S. Miller <davem@davemloft.net> (cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0) BUG= chromium:786514 TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389 Reviewed-on: https://chromium-review.googlesource.com/825922 Commit-Ready: Kevin Cernekee <cernekee@chromium.org> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/2dc6db8f0927923da42e67ba1736c137908f61fa/net/ipv4/igmp.c
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/05dcc3c7bbe79e2b0e4fc09dee96cffbac592c5e commit 05dcc3c7bbe79e2b0e4fc09dee96cffbac592c5e Author: Kevin Cernekee <cernekee@chromium.org> Date: Thu Dec 14 06:33:15 2017 FROMGIT: net: igmp: Use correct source address on IGMPv3 reports Closing a multicast socket after the final IPv4 address is deleted from an interface can generate a membership report that uses the source IP from a different interface. The following test script, run from an isolated netns, reproduces the issue: #!/bin/bash ip link add dummy0 type dummy ip link add dummy1 type dummy ip link set dummy0 up ip link set dummy1 up ip addr add 10.1.1.1/24 dev dummy0 ip addr add 192.168.99.99/24 dev dummy1 tcpdump -U -i dummy0 & socat EXEC:"sleep 2" \ UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 & sleep 1 ip addr del 10.1.1.1/24 dev dummy0 sleep 5 kill %tcpdump RFC 3376 specifies that the report must be sent with a valid IP source address from the destination subnet, or from address 0.0.0.0. Add an extra check to make sure this is the case. Signed-off-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Andrew Lunn <andrew@lunn.ch> Signed-off-by: David S. Miller <davem@davemloft.net> (cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0) BUG= chromium:786514 TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389 Reviewed-on: https://chromium-review.googlesource.com/825773 Commit-Ready: Kevin Cernekee <cernekee@chromium.org> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/05dcc3c7bbe79e2b0e4fc09dee96cffbac592c5e/net/ipv4/igmp.c
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2606df717ce796c98dc2c6a03adfd353a38e3fc9 commit 2606df717ce796c98dc2c6a03adfd353a38e3fc9 Author: Kevin Cernekee <cernekee@chromium.org> Date: Thu Dec 14 06:33:19 2017 FROMGIT: net: igmp: Use correct source address on IGMPv3 reports Closing a multicast socket after the final IPv4 address is deleted from an interface can generate a membership report that uses the source IP from a different interface. The following test script, run from an isolated netns, reproduces the issue: #!/bin/bash ip link add dummy0 type dummy ip link add dummy1 type dummy ip link set dummy0 up ip link set dummy1 up ip addr add 10.1.1.1/24 dev dummy0 ip addr add 192.168.99.99/24 dev dummy1 tcpdump -U -i dummy0 & socat EXEC:"sleep 2" \ UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 & sleep 1 ip addr del 10.1.1.1/24 dev dummy0 sleep 5 kill %tcpdump RFC 3376 specifies that the report must be sent with a valid IP source address from the destination subnet, or from address 0.0.0.0. Add an extra check to make sure this is the case. Signed-off-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Andrew Lunn <andrew@lunn.ch> Signed-off-by: David S. Miller <davem@davemloft.net> (cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0) BUG= chromium:786514 TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389 Reviewed-on: https://chromium-review.googlesource.com/825778 Commit-Ready: Kevin Cernekee <cernekee@chromium.org> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/2606df717ce796c98dc2c6a03adfd353a38e3fc9/net/ipv4/igmp.c
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/75f317e5cefd60a3cb6e306063530a0cf283893f commit 75f317e5cefd60a3cb6e306063530a0cf283893f Author: Kevin Cernekee <cernekee@chromium.org> Date: Thu Dec 14 06:33:11 2017 FROMGIT: net: igmp: Use correct source address on IGMPv3 reports Closing a multicast socket after the final IPv4 address is deleted from an interface can generate a membership report that uses the source IP from a different interface. The following test script, run from an isolated netns, reproduces the issue: #!/bin/bash ip link add dummy0 type dummy ip link add dummy1 type dummy ip link set dummy0 up ip link set dummy1 up ip addr add 10.1.1.1/24 dev dummy0 ip addr add 192.168.99.99/24 dev dummy1 tcpdump -U -i dummy0 & socat EXEC:"sleep 2" \ UDP4-DATAGRAM:239.101.1.68:8889,ip-add-membership=239.0.1.68:10.1.1.1 & sleep 1 ip addr del 10.1.1.1/24 dev dummy0 sleep 5 kill %tcpdump RFC 3376 specifies that the report must be sent with a valid IP source address from the destination subnet, or from address 0.0.0.0. Add an extra check to make sure this is the case. Signed-off-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Andrew Lunn <andrew@lunn.ch> Signed-off-by: David S. Miller <davem@davemloft.net> (cherry picked from davem/net commit a46182b00290839fa3fa159d54fd3237bd8669f0) BUG= chromium:786514 TEST=run `ifconfig wlan0 0.0.0.0` at login screen and sniff network Change-Id: I9d5df63cd70055ce9c19934cbce36ca79f89e389 Reviewed-on: https://chromium-review.googlesource.com/825654 Commit-Ready: Kevin Cernekee <cernekee@chromium.org> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/75f317e5cefd60a3cb6e306063530a0cf283893f/net/ipv4/igmp.c
,
Dec 15 2017
The IGMPv3 source address change is verified on samus canary 10215.0.0. The kernel patch has also been accepted upstream. This fix will not affect IGMPv1 or IGMPv2. Based on a cursory glance at the code, I don't think IGMPv1/IGMPv2 will normally be used unless the kernel detects IGMPv1/IGMPv2 multicasts from another host on the LAN. If you're still seeing the problem on >=10215.0.0, please attach a .pcap file showing the packet(s) at fault, so their source can be tracked down.
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/268af7ab4188b9c99b8d28f67d2d41698c27ac93 commit 268af7ab4188b9c99b8d28f67d2d41698c27ac93 Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org> Date: Fri Feb 02 08:23:29 2018 UPSTREAM: net: igmp: fix source address check for IGMPv3 reports [ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ] Commit "net: igmp: Use correct source address on IGMPv3 reports" introduced a check to validate the source address of locally generated IGMPv3 packets. Instead of checking the local interface address directly, it uses inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the local subnet (or equal to the point-to-point address if used). This breaks for point-to-point interfaces, so check against ifa->ifa_local directly. Cc: Kevin Cernekee <cernekee@chromium.org> Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Felix Fietkau <nbd@nbd.name> Signed-off-by: David S. Miller <davem@davemloft.net> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Tested-by: Florian Wolters <florian@florian-wolters.de> (cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa) BUG= chromium:786514 TEST=buildbots Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1 Reviewed-on: https://chromium-review.googlesource.com/898213 Commit-Ready: Kevin Cernekee <cernekee@chromium.org> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/268af7ab4188b9c99b8d28f67d2d41698c27ac93/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/b2ffb7f761f2a0421aea41ec6c5b6c6762cc9522 commit b2ffb7f761f2a0421aea41ec6c5b6c6762cc9522 Author: Eric Dumazet <edumazet@google.com> Date: Fri Feb 02 08:23:32 2018 FROMGIT: net: igmp: add a missing rcu locking section Newly added igmpv3_get_srcaddr() needs to be called under rcu lock. Timer callbacks do not ensure this locking. ============================= WARNING: suspicious RCU usage 4.15.0+ #200 Not tainted ----------------------------- ./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 3 locks held by syzkaller616973/4074: #0: (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355 #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline] #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316 #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline] #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600 stack backtrace: CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x257 lib/dump_stack.c:53 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline] igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline] igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285 invoke_softirq kernel/softirq.c:365 [inline] irq_exit+0x1cc/0x200 kernel/softirq.c:405 exiting_irq arch/x86/include/asm/apic.h:541 [inline] smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938 Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: syzbot <syzkaller@googlegroups.com> (cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5) BUG= chromium:786514 TEST=buildbots Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e Reviewed-on: https://chromium-review.googlesource.com/898214 Commit-Ready: Kevin Cernekee <cernekee@chromium.org> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/b2ffb7f761f2a0421aea41ec6c5b6c6762cc9522/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e444947eb5150954580cc1bb3f11122ff7dc7416 commit e444947eb5150954580cc1bb3f11122ff7dc7416 Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org> Date: Fri Feb 02 18:17:11 2018 UPSTREAM: net: igmp: fix source address check for IGMPv3 reports [ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ] Commit "net: igmp: Use correct source address on IGMPv3 reports" introduced a check to validate the source address of locally generated IGMPv3 packets. Instead of checking the local interface address directly, it uses inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the local subnet (or equal to the point-to-point address if used). This breaks for point-to-point interfaces, so check against ifa->ifa_local directly. Cc: Kevin Cernekee <cernekee@chromium.org> Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Felix Fietkau <nbd@nbd.name> Signed-off-by: David S. Miller <davem@davemloft.net> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Tested-by: Florian Wolters <florian@florian-wolters.de> (cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa) BUG= chromium:786514 TEST=buildbots Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1 Reviewed-on: https://chromium-review.googlesource.com/898266 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/e444947eb5150954580cc1bb3f11122ff7dc7416/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/1d6f8fa079fc495a13b28769cfa8bf75cfd40226 commit 1d6f8fa079fc495a13b28769cfa8bf75cfd40226 Author: Eric Dumazet <edumazet@google.com> Date: Fri Feb 02 18:17:14 2018 FROMGIT: net: igmp: add a missing rcu locking section Newly added igmpv3_get_srcaddr() needs to be called under rcu lock. Timer callbacks do not ensure this locking. ============================= WARNING: suspicious RCU usage 4.15.0+ #200 Not tainted ----------------------------- ./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 3 locks held by syzkaller616973/4074: #0: (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355 #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline] #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316 #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline] #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600 stack backtrace: CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x257 lib/dump_stack.c:53 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline] igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline] igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285 invoke_softirq kernel/softirq.c:365 [inline] irq_exit+0x1cc/0x200 kernel/softirq.c:405 exiting_irq arch/x86/include/asm/apic.h:541 [inline] smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938 Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: syzbot <syzkaller@googlegroups.com> (cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5) BUG= chromium:786514 TEST=buildbots Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e Reviewed-on: https://chromium-review.googlesource.com/898267 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/1d6f8fa079fc495a13b28769cfa8bf75cfd40226/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/96307b9462b2bc8240ec2ee5899bc9309789bcb2 commit 96307b9462b2bc8240ec2ee5899bc9309789bcb2 Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org> Date: Fri Feb 02 18:17:00 2018 UPSTREAM: net: igmp: fix source address check for IGMPv3 reports [ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ] Commit "net: igmp: Use correct source address on IGMPv3 reports" introduced a check to validate the source address of locally generated IGMPv3 packets. Instead of checking the local interface address directly, it uses inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the local subnet (or equal to the point-to-point address if used). This breaks for point-to-point interfaces, so check against ifa->ifa_local directly. Cc: Kevin Cernekee <cernekee@chromium.org> Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Felix Fietkau <nbd@nbd.name> Signed-off-by: David S. Miller <davem@davemloft.net> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Tested-by: Florian Wolters <florian@florian-wolters.de> (cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa) BUG= chromium:786514 TEST=buildbots Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1 Reviewed-on: https://chromium-review.googlesource.com/897408 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/96307b9462b2bc8240ec2ee5899bc9309789bcb2/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6528f26d01d6fd2869107bd84d3c8ee160015b56 commit 6528f26d01d6fd2869107bd84d3c8ee160015b56 Author: Eric Dumazet <edumazet@google.com> Date: Fri Feb 02 18:17:03 2018 FROMGIT: net: igmp: add a missing rcu locking section Newly added igmpv3_get_srcaddr() needs to be called under rcu lock. Timer callbacks do not ensure this locking. ============================= WARNING: suspicious RCU usage 4.15.0+ #200 Not tainted ----------------------------- ./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 3 locks held by syzkaller616973/4074: #0: (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355 #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline] #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316 #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline] #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600 stack backtrace: CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x257 lib/dump_stack.c:53 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline] igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline] igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285 invoke_softirq kernel/softirq.c:365 [inline] irq_exit+0x1cc/0x200 kernel/softirq.c:405 exiting_irq arch/x86/include/asm/apic.h:541 [inline] smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938 Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: syzbot <syzkaller@googlegroups.com> (cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5) BUG= chromium:786514 TEST=buildbots Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e Reviewed-on: https://chromium-review.googlesource.com/897409 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/6528f26d01d6fd2869107bd84d3c8ee160015b56/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/04aa472ba1b6973ce55954ff8866433e1593fbd8 commit 04aa472ba1b6973ce55954ff8866433e1593fbd8 Author: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org> Date: Fri Feb 02 18:17:07 2018 UPSTREAM: net: igmp: fix source address check for IGMPv3 reports [ Upstream commit ad23b750933ea7bf962678972a286c78a8fa36aa ] Commit "net: igmp: Use correct source address on IGMPv3 reports" introduced a check to validate the source address of locally generated IGMPv3 packets. Instead of checking the local interface address directly, it uses inet_ifa_match(fl4->saddr, ifa), which checks if the address is on the local subnet (or equal to the point-to-point address if used). This breaks for point-to-point interfaces, so check against ifa->ifa_local directly. Cc: Kevin Cernekee <cernekee@chromium.org> Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Reported-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Felix Fietkau <nbd@nbd.name> Signed-off-by: David S. Miller <davem@davemloft.net> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Tested-by: Florian Wolters <florian@florian-wolters.de> (cherry picked from commit ad23b750933ea7bf962678972a286c78a8fa36aa) BUG= chromium:786514 TEST=buildbots Change-Id: I2c5f3a516f732be4d15d955058fe53ac098cf9f1 Reviewed-on: https://chromium-review.googlesource.com/897406 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/04aa472ba1b6973ce55954ff8866433e1593fbd8/net/ipv4/igmp.c
,
Feb 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d25b501bf1ca79c678613b73327e7a2317a73fd1 commit d25b501bf1ca79c678613b73327e7a2317a73fd1 Author: Eric Dumazet <edumazet@google.com> Date: Fri Feb 02 18:17:09 2018 FROMGIT: net: igmp: add a missing rcu locking section Newly added igmpv3_get_srcaddr() needs to be called under rcu lock. Timer callbacks do not ensure this locking. ============================= WARNING: suspicious RCU usage 4.15.0+ #200 Not tainted ----------------------------- ./include/linux/inetdevice.h:216 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 3 locks held by syzkaller616973/4074: #0: (&mm->mmap_sem){++++}, at: [<00000000bfce669e>] __do_page_fault+0x32d/0xc90 arch/x86/mm/fault.c:1355 #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] lockdep_copy_map include/linux/lockdep.h:178 [inline] #1: ((&im->timer)){+.-.}, at: [<00000000619d2f71>] call_timer_fn+0x1c6/0x820 kernel/time/timer.c:1316 #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] spin_lock_bh include/linux/spinlock.h:315 [inline] #2: (&(&im->lock)->rlock){+.-.}, at: [<000000005f833c5c>] igmpv3_send_report+0x98/0x5b0 net/ipv4/igmp.c:600 stack backtrace: CPU: 0 PID: 4074 Comm: syzkaller616973 Not tainted 4.15.0+ #200 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x257 lib/dump_stack.c:53 lockdep_rcu_suspicious+0x123/0x170 kernel/locking/lockdep.c:4592 __in_dev_get_rcu include/linux/inetdevice.h:216 [inline] igmpv3_get_srcaddr net/ipv4/igmp.c:329 [inline] igmpv3_newpack+0xeef/0x12e0 net/ipv4/igmp.c:389 add_grhead.isra.27+0x235/0x300 net/ipv4/igmp.c:432 add_grec+0xbd3/0x1170 net/ipv4/igmp.c:565 igmpv3_send_report+0xd5/0x5b0 net/ipv4/igmp.c:605 igmp_send_report+0xc43/0x1050 net/ipv4/igmp.c:722 igmp_timer_expire+0x322/0x5c0 net/ipv4/igmp.c:831 call_timer_fn+0x228/0x820 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7ee/0xb70 kernel/time/timer.c:1666 run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285 invoke_softirq kernel/softirq.c:365 [inline] irq_exit+0x1cc/0x200 kernel/softirq.c:405 exiting_irq arch/x86/include/asm/apic.h:541 [inline] smp_apic_timer_interrupt+0x16b/0x700 arch/x86/kernel/apic/apic.c:1052 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:938 Fixes: a46182b00290 ("net: igmp: Use correct source address on IGMPv3 reports") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: syzbot <syzkaller@googlegroups.com> (cherry picked from davem/net.git e7aadb27a5415e8125834b84a74477bfbee4eff5) BUG= chromium:786514 TEST=buildbots Change-Id: I8c48c99188d27fd02707ea1e808a4f84ceefbf7e Reviewed-on: https://chromium-review.googlesource.com/897407 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Kevin Cernekee <cernekee@chromium.org> Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org> [modify] https://crrev.com/d25b501bf1ca79c678613b73327e7a2317a73fd1/net/ipv4/igmp.c
,
Mar 16 2018
It sounds like we shouldn't be seeing this problem anymore, but it's still happening. I have 100s of alerts for client exclusion again with an IP of 100.115.92.1. This seemed to be better for awhile and now it's back. Please let us know what information you would need in order to keep digging into why this is still happening. I'm seeing the alerts on a Cisco Prime Infrastructure server coming from Chrome devices.
,
Mar 16 2018
M65 should have the fix. If you're still seeing alerts, could you please attach a packet capture so we can see whether the culprit is IGMPv3 or something else?
,
Mar 16 2018
Is it a new fix for M65? I see that M65 shows March 13th as the release date for the stable build - so there are likely computers that haven't updated yet. I'll have to think about how I could grab a packet capture. I'd have to dig into MAC addresses to determine if these are coming from our devices or from devices that we don't own. In all honesty - I'm sick of being a network tester for Google on all this stuff. I'm participating in at least 3 different bug forums right now for various network issues that we're experiencing with Chromebooks right now and I have never had to do anything like this for any other platform that hits our network. Everything from 802.1x, to this and DHCP issues. If I can track down a device with the issue - what's your recommendation for doing a packet capture on it?
,
Mar 16 2018
> Is it a new fix for M65? Yes. The CL went into ToT in mid-December. M64 branched Nov 30. M65 branched Jan 18. So it's in M65 but not M64. > I'll have to think about how I could grab a packet capture. Is it possible to connect a Linux PC to a switch port that can see all traffic (e.g. via SPAN) and run something like `tcpdump -n -i eth1 -w /tmp/badip.pcap net 100.115.92.0/24` to capture any traffic that uses the reserved IP range? |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by defo...@kmsd.edu
, Nov 17 2017