From dmesg:

INFO kernel: [  306.987654] r8169 0000:01:00.0 eth0: link down

correction, seems to go away pretty quickly after it boots on the order of O(mins) 

Any chance it could be a hardware issue?

no, I booted up with a guado_moblab image and interface is still up so probably something I'm missing in the image for stability (or added that causes instability).

Check /var/log/messages and /var/log/net.*.log for anything that might indicate what happened.

from /var/log/net.log before losing connection:

016-03-02T08:33:41.662092+00:00 INFO shill[1871]: [INFO:manager.cc(1454)] Device eth0 updated: disabled
2016-03-02T08:33:41.675565+00:00 INFO dhcpcd[9344]: eth0: releasing lease of 100.96.50.148
2016-03-02T08:33:41.675584+00:00 INFO dhcpcd[9344]: eth0: sending RELEASE with xid 0x219d883a
2016-03-02T08:33:41.682458+00:00 INFO dhcpcd[9344]: status changed to Release
2016-03-02T08:33:41.682657+00:00 INFO shill[1871]: [INFO:dhcp_config.cc(223)] Stopping 9344 (ReleaseIP)
2016-03-02T08:33:41.682760+00:00 INFO dhcpcd[9344]: received SIGTERM, stopping
2016-03-02T08:33:41.682768+00:00 INFO dhcpcd[9344]: eth0: removing interface
2016-03-02T08:33:41.690884+00:00 INFO dhcpcd[9344]: status changed to Release
2016-03-02T08:33:41.690903+00:00 INFO dhcpcd[9344]: dhcpcd exited

That's what you'd expect to see if someone requests shill to disable the eth0 interface.  How fast does this happen after boot?  Is Chrome running?

Chrome was running but seems to have died.  This always seems to be the case where the system boots up, chrome starts up and eth0 is up.  Then when eth0 is down chrome seems to have crashed. 

Here's a fuller snippet (with the full log attached):
2016-03-07T11:27:26.136310-08:00 INFO shill[1399]: [INFO:dhcpv4_config.cc(126)] Event reason: BOUND
2016-03-07T11:27:26.138029-08:00 INFO shill[1399]: [INFO:connection.cc(240)] UpdateFromIPConfig: Installing with parameters: local=100.96.50.57 broadcast=100.96.51.255 peer=<unknown> gateway=100.96.51.254
2016-03-07T11:27:26.138072-08:00 INFO shill[1399]: [INFO:service.cc(411)] Service 1: state Configuring -> Connected
2016-03-07T11:27:26.144544-08:00 INFO shill[1399]: [INFO:manager.cc(1439)] Service 1 updated; state: Connected failure Unknown
2016-03-07T11:27:26.153234-08:00 INFO shill[1399]: [INFO:service.cc(411)] Service 1: state Connected -> Online
2016-03-07T11:27:26.153343-08:00 INFO shill[1399]: [INFO:manager.cc(1439)] Service 1 updated; state: Online failure Unknown
2016-03-07T11:32:44.395345-08:00 INFO shill[1399]: [INFO:manager.cc(1454)] Device eth0 updated: disabled
2016-03-07T11:32:44.406672-08:00 INFO dhcpcd[4482]: eth0: releasing lease of 100.96.50.57
2016-03-07T11:32:44.406712-08:00 INFO dhcpcd[4482]: eth0: sending RELEASE with xid 0x3409c68e
2016-03-07T11:32:44.414666-08:00 INFO dhcpcd[4482]: status changed to Release
2016-03-07T11:32:44.414939-08:00 INFO shill[1399]: [INFO:dhcp_config.cc(223)] Stopping 4482 (ReleaseIP)
2016-03-07T11:32:44.414991-08:00 INFO dhcpcd[4482]: received SIGTERM, stopping
2016-03-07T11:32:44.414999-08:00 INFO dhcpcd[4482]: eth0: removing interface
2016-03-07T11:32:44.423631-08:00 INFO dhcpcd[4482]: status changed to Release
2016-03-07T11:32:44.423926-08:00 INFO dhcpcd[4482]: dhcpcd exited

Deleted: net.1.log 14.0 KB

net.1.log 14.0 KB View Download

Forgot to answer the other question:

> How fast does this happen after boot?
Seems random, within a couple mins to at least a day after bootup.

need to look into how to truly disable chrome, Wiley mentioned that a possibility is that shill disable the interface due to inactivity on that interface which might make sense given the randomness of when the interface gets turned off.

Hmm... might be some policy thing... (this is from /var/log/messages)

2016-04-08T06:56:57.013081+08:00 WARNING cryptohomed[1696]: Could not load the device policy file.
2016-04-08T06:56:57.044332+08:00 INFO kernel: [  305.740556] tpm_tis tpm_tis: command 0x15 (size 14) returned code 0x0
2016-04-08T06:56:57.086370+08:00 INFO kernel: [  305.782597] tpm_tis tpm_tis: command 0x14 (size 34) returned code 0x0
2016-04-08T06:56:57.089679+08:00 INFO cryptohomed[1696]: RecursiveCopy: /home/.shadow/skeleton/.pki
2016-04-08T06:56:57.090093+08:00 INFO cryptohomed[1696]: RecursiveCopy: /home/.shadow/skeleton/.pki/nssdb
2016-04-08T06:56:57.091661+08:00 INFO cryptohomed[1696]: RecursiveCopy: /home/.shadow/skeleton/log
2016-04-08T06:56:57.092000+08:00 INFO cryptohomed[1696]: RecursiveCopy: /home/.shadow/skeleton/.ssh
2016-04-08T06:56:57.101005+08:00 INFO session_manager[879]: [INFO:policy_key.cc(54)] No policy key on disk at /home/root/e5336716e661f546f078ce469976695d30b57a8b/session_manager/policy/key
2016-04-08T06:56:57.105347+08:00 ERR session_manager[879]: [ERROR:nss_util.cc(133)] Error opening persistent database (configDir='sql:/home/user/e5336716e661f546f078ce469976695d30b57a8b/.pki/nssdb' tokenDescription='/home/user/e5336716e661f546f078ce469976695d30b57a8b'): -8018
2016-04-08T06:56:57.105395+08:00 ERR session_manager[879]: [ERROR:session_manager_impl.cc(731)] Could not open the current user's NSS database.
Timeout, server brillo-teststation not responding.

Hmmm...  Is it your intent to install (and run) Chrome on these
configurations?

definitely not, I'd love to disable chrome on this (but am having a hard time finding out how).

Take a look at the beaglebone overlay.  I think it starts with
USE="+embedded".  There may be some other relevant USE flags.

disabling cryptohomed seems to have done the trick, system has been up accessible night long (previously for only a couple mins).  I just added a file (/etc/init/cryptohomed.conf with 'manual' in it) to keep cryptohomed from starting up.  I think I'll just have a hack to disable cryptohomed for now, setup the builders, then work on getting the labstation to be chromeless (taking a look at beaglebone and panther-embedded for pointers) and remove the hack then.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/287cc2a136c33fd4208111d0329f0a25573a8fa6

commit 287cc2a136c33fd4208111d0329f0a25573a8fa6
Author: Kevin Cheng <kevcheng@chromium.org>
Date: Fri Apr 08 16:53:47 2016

labstation: Add in cryptohomed override to not start on startup.

Cryptohomed was disabling the eth0 iface because it couldn't find the
policy files.  This was troublesome since eth0 was our only way to
access the labstation.  This is a hack while I work on getting the
labstation to become chromeless.

BUG= chromium:591091 
TEST=built and flashed and cryptohomed did not start up.

Change-Id: I0a741e16ec7b74bd24d92de0173fc7676a0bdd60
Reviewed-on: https://chromium-review.googlesource.com/337990
Commit-Ready: Kevin Cheng <kevcheng@chromium.org>
Tested-by: Kevin Cheng <kevcheng@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/287cc2a136c33fd4208111d0329f0a25573a8fa6/project-labstation/chromeos-base/chromeos-bsp-labstation/chromeos-bsp-labstation-0.0.1.ebuild
[add] https://crrev.com/287cc2a136c33fd4208111d0329f0a25573a8fa6/project-labstation/chromeos-base/chromeos-bsp-labstation/files/init/cryptohomed.override
[rename] https://crrev.com/287cc2a136c33fd4208111d0329f0a25573a8fa6/project-labstation/chromeos-base/chromeos-bsp-labstation/chromeos-bsp-labstation-0.0.1-r3.ebuild

Bulk verified