New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 609610 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: May 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

MobLab ToT not showing network bridge

Project Member Reported by sbasi@chromium.org, May 5 2016

Issue description

The last canary run and recent paladin runs keep consistently hitting an error similar to crbug.com/607196 though I suspect it may be unrelated.

The moblab reports 0 duts. It gets repaired to a stable image, the duts reappear.

Suspect a bug in ToT. Investigating.

 

Comment 1 by sbasi@chromium.org, May 5 2016

Cc: olofj@chromium.org pstew@chromium.org
ifconfig is not showing the network bridge:

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.186.227  netmask 255.255.254.0  broadcast 172.18.187.255
        inet6 fe80::2e60:cff:fea9:6aa9  prefixlen 64  scopeid 0x20<link>
        ether 2c:60:0c:a9:6a:a9  txqueuelen 1000  (Ethernet)
        RX packets 1035  bytes 305845 (298.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1612  bytes 1780053 (1.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 80:3f:5d:9f:73:5d  txqueuelen 1000  (Ethernet)
        RX packets 96  bytes 9066 (8.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 82  bytes 6512 (6.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 82  bytes 6512 (6.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


There should be an interface labelled lxcbr0. Suspecting something broke this.

Comment 2 by sbasi@chromium.org, May 5 2016

Summary: MobLab ToT not showing network bridge (was: Suspect breakage in MobLab ToT)
So manually trying to add the network bridge fails:

$ sudo brctl addbr lxcbr0
device lxcbr0 already exists; can't create bridge with the same name

But it is still not listed under ifconfig.

pstew, wiley any thoughts?

Comment 3 by sbasi@chromium.org, May 5 2016

Cc: semenzato@chromium.org
I suspect it is this cl which reveved Shill

https://chromium-review.googlesource.com/#/c/341463/
Project Member

Comment 4 by bugdroid1@chromium.org, May 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/f371649ba9cfef6aca9670914befa620b4884c59

commit f371649ba9cfef6aca9670914befa620b4884c59
Author: Luigi Semenzato <semenzato@chromium.org>
Date: Thu May 05 23:10:10 2016

Revert "Marking 9999 ebuild for chromeos-base/shill as stable."

This reverts commit 97f2a4cb020f6ef8356c9fd79ea59d0f713197e0.

This is causing issues in moblab.

https://bugs.chromium.org/p/chromium/issues/detail?id=609610

From the bug:

So manually trying to add the network bridge fails:

$ sudo brctl addbr lxcbr0
device lxcbr0 already exists; can't create bridge with the same name

But it is still not listed under ifconfig.

BUG= chromium:609610 
TEST=none

Change-Id: I0bd65ca9999836040cd35cd405b1f62ee35217ea
Reviewed-on: https://chromium-review.googlesource.com/342777
Commit-Queue: Luigi Semenzato <semenzato@chromium.org>
Tested-by: Luigi Semenzato <semenzato@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>

[rename] https://crrev.com/f371649ba9cfef6aca9670914befa620b4884c59/chromeos-base/shill/shill-0.0.3-r11.ebuild

Cc: -pstew@chromium.org rbhagavatula@chromium.org
Alejandro pointed out that the failure of "sudo brctl addbr lxcbr0" is unlikely to be related to shill changes.  I was too much in a panic to notice, sorry.
Cc: -kcwu@chromium.org pstew@chromium.org
(sorry, didn't mean to remove Paul)
Cc: kcwu@chromium.org
ARGH!  Bad UI.

Comment 9 by sbasi@chromium.org, May 5 2016

Luigi's CL included:

202ed0d shill: improve handling of bridge interfaces

That seems suspect to me.
List of possible CLs:

https://crosland.corp.google.com/log/8282.0.0..8283.0.0


Note the fact this hit the canary build means it landed.
Cc: gdk@chromium.org
+gdk

Here is the CL I suspect https://android-review.googlesource.com/#/c/214451/

To summarize the issue is:
* An init script creates a network bridge and restarts shill (twice).
* After everything is initialized the network bridge is not listed under ifconfig

* Trying to create a bridge with the same name after boot complains that it exists.
* Interestingly if I create a bridge with a different name it works.

On the same device 8282.0.0 is fine, if I flash to 8283.0.0 the problem occurs.

The kernel is 3.14.0
Cc: de...@chromium.org

Comment 13 by gdk@chromium.org, May 6 2016

https://android-review.googlesource.com/#/c/214451/ shouldn't be able to cause what you're seeing, I believe.  My change's most meaningful impact is that bridge devices get picked up by shill faster.

In your third bullet, are you trying to create the bridge interface before shill is started and it fails?
Maybe the code might help:

https://chromium.googlesource.com/chromiumos/overlays/board-overlays.git/+/master/project-moblab/chromeos-base/chromeos-bsp-moblab/files/moblab-network-bridge-init.conf


We do know that the problem is in one of the commits in the mass shill uprev Luigi committed as reverting that solved the problem.
I've opened  issue 609852  to resolve the "invisible bridge" problem.  The behavior is strange enough that it deserves some attention, even if it doesn't cause this bug.

I think we can close this since the revert fixed the problem.
Status: Fixed (was: Assigned)
Tree is back green.
Cc: caiz@chromium.org
Status: Verified (was: Fixed)
Closing. please reopen if its not fixed.

Sign in to add a comment