Summary search for this crash:

https://crash.corp.google.com/browse?q=product.name%3D%27ChromeOS%27%20OMIT%20RECORD%20IF%20SUM(CrashedStackTrace.StackFrame.FunctionName%3D%27arc_networkd%3A%3AManager%3A%3AInitialSetup%27)%20%3D%200

Kevin, any update on this one? 

I'm experiencing this crash almost every 5 min on my Samus (M54)

When you see this crash the Android container is not starting, correct?

I can check next time it crashes .. I'm not seeing this that frequently any more

Crash numbers are substantially reduced in last M54 builds but crash is still there 
https://crash.corp.google.com/browse?q=stable_signature%3D%27raise-105fbe2d%27%20AND%20product.version%20like%20%278743.%25%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion,magicsignature,stablesignature

Crash does not seem to be new in M54 though (e.g. it is in M53 too)

Kevin, are you still the right owner to evaluate next steps/fix?

M55 still has the issue: go/crash/08d7c10500000000

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/9f649d56e242c5606ec6ade4605ab5a81d079fb0

commit 9f649d56e242c5606ec6ade4605ab5a81d079fb0
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Sun Dec 04 04:37:07 2016

arc-networkd: Don't crash if rt_tables isn't populated right away

If the user logs in with no active network connection, arc-networkd
will check every few seconds to see if: 1) arc0 inside the container is
up, and 2) rt_tables has an entry for arc0.  (1) should happen in the
absence of a LAN connection, but (2) will not, causing arc-networkd
to time out and abort.

Move the check for (2) into the Set() function, which runs a couple
of seconds after the external LAN interface is up.

BUG= chromium:649247 
TEST=`tail -f /var/log/net.log` with no connection, wifi connection

Change-Id: Ie2680c68810af9ff082a589e136ae98fbf7bd00c
Reviewed-on: https://chromium-review.googlesource.com/416362
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Mattias Nissler <mnissler@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/9f649d56e242c5606ec6ade4605ab5a81d079fb0/arc-networkd/arc_ip_config.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/9e2e4de211e492c4462e7e6cc15fbda4f77932d6

commit 9e2e4de211e492c4462e7e6cc15fbda4f77932d6
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Sun Dec 04 04:49:42 2016

arc-networkd: Log a sane exit code if Minijail::RunSyncAndDestroy fails

|status| is a status code from waitpid(), so it needs to be run through
WEXITSTATUS in order to make it human-readable.

BUG= chromium:649247 
TEST=watch net.log when a command fails

Change-Id: I545b04062b1b5177c10c7e7d810ad3dface80e7c
Reviewed-on: https://chromium-review.googlesource.com/416363
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Mattias Nissler <mnissler@chromium.org>
Reviewed-by: Abhishek Bhardwaj <abhishekbh@google.com>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/9e2e4de211e492c4462e7e6cc15fbda4f77932d6/arc-networkd/arc_ip_config.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/60007a42b43484f5d2090556389eaff75324b24e

commit 60007a42b43484f5d2090556389eaff75324b24e
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Sun Dec 04 05:04:59 2016

arc-networkd: Don't log expected `ip` command failures

There are a couple of cases in which arc-networkd may race with the
kernel or ARC in tearing down IPv6 configuration parameters.  If these
fail, do not send a warning to net.log as it isn't necessarily a sign
that anything went wrong.

BUG= chromium:649247 
TEST=trigger the `ip` failures and watch net.log for results

Change-Id: I6a89972a6c47956014a36b5c903939b35fc6ef3b
Reviewed-on: https://chromium-review.googlesource.com/416364
Commit-Ready: Kevin Cernekee <cernekee@chromium.org>
Tested-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-by: Mattias Nissler <mnissler@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/60007a42b43484f5d2090556389eaff75324b24e/arc-networkd/arc_ip_config.cc
[modify] https://crrev.com/60007a42b43484f5d2090556389eaff75324b24e/arc-networkd/arc_ip_config.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/14fff422cf8fc0f5e75faffedb7b888d65d6b5eb

commit 14fff422cf8fc0f5e75faffedb7b888d65d6b5eb
Author: Kevin Cernekee <cernekee@chromium.org>
Date: Tue Dec 06 19:52:50 2016

arc-networkd: Log termination signals

Currently the error handler only prints the exit code, but this will
not provide any useful information if the process died due to a fatal
signal.

BUG= chromium:649247 
TEST=compile-tested

Change-Id: If7794d782ed726e2d7d1b1170fcf18645da69ba6
Signed-off-by: Kevin Cernekee <cernekee@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/417136

[modify] https://crrev.com/14fff422cf8fc0f5e75faffedb7b888d65d6b5eb/arc-networkd/arc_ip_config.cc

Are we merging this back to 56? 55?

crosreview.com/416362 might be worth merging.  I'd try it on M56 first and see if people are still hitting the LOG(FATAL).

Any idea what version has CL:416362?
The arc_networkd SIGABRTs are still pretty common in R56 (9000.29.0).

Your change meets the bar and is auto-approved for M56 (branch: 2924)

go/crosland claims that it landed in 9056.0.0.

When I search for crash reports for minnie on >= 9056.0.0 I see:

https://goto.google.com/peuhx

A bunch of these crashes show OnSubprocessExited in the backtrace.  This is the parent process crashing because the subprocess hit a fatal error.  Those dumps don't tell us anything useful.

But for each OnSubprocessExited crash there is usually a corresponding report from the subprocess that does have a useful backtrace.  The ones that I see show a crash in ArcIpConfig::Set(), hitting the LOG(FATAL):

  // At this point, arc0 is up and the LAN interface has been up for several
  // seconds.  If the routing table name has not yet been populated,
  // something really bad probably happened on the Android side.
  if (routing_table_id_ == kInvalidTableId) {
    routing_table_id_ = ReadTableId(con_ifname_);
    if (routing_table_id_ == kInvalidTableId) {
      LOG(FATAL) << "Could not look up routing table ID in "
                 << kRoutingTableNames;
    }
  }

Perhaps this means that there is a real problem in the Android container, or perhaps it means that arc-networkd is making a bad assumption about when the rt_tables file should be getting populated.

It would be helpful to see the system logs in order to figure out whether this is only happening on the initial Android boot, if it is correlated with some other issue, or if it's an intermittent error that randomly affects otherwise-functioning systems.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Reassigning to Abhishek per our offline conversation.

Typical crash looks like: https://goto.google.com/uynic

This issue hasn't been updated in the last 6 weeks, so removing its merge approval label. Please re-request a merge if needed.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Seeing these crashes daily from ChromeOS  65.0.3325.107 - looking at the stack and code, it seems that |bus_| must be null when InitialSetup() is executed. That is PostTask()'d from OnInit with a comment that it must execute only after |bus_| has been initialized, but no explanation as to why posting the task achieves that.

See crashes 6fac68db42ade8a0, 8661a2365a8b475f, 16f100b1dcc835b1.

Could you please file feedback next time you see it, and add @cernekee to the note?  Logs would be very helpful.