haddowk: I'm deputy so it's hard for me to focus on one thing this week.
Can you help find the culprit?

I think it should be a CL in the last few consistently green builds:

https://luci-milo.appspot.com/buildbot/chromiumos/moblab-generic-vm-paladin/

Looking at one of the failures: https://luci-milo.appspot.com/buildbot/chromiumos/moblab-generic-vm-paladin/988

gs_offloader didn't drop any logs whatsoever: https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/moblab-generic-vm-paladin/R67-10465.0.0-rc3/moblab_vm_test_results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite/sysinfo/var/log/autotest

Compare this with a good run: 
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/moblab-generic-vm-paladin/R67-10466.0.0-rc1/moblab_vm_test_results/results-1-moblab_DummyServerNoSspSuite/sysinfo/var/log/autotest


gs_offloader's upstart job doesn't leave any bootup logs unfortunately.

Last CQ run failed just because of this. Gotta mark this experimental :(

Updated tree status.

There is no single CL in all the recent failures: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildCompare?lastN=20&masterBuilder=master-paladin&botConfig=moblab-generic-vm-paladin&buildBotMaster=chromiumos#highlightKeyValues=status%E2%86%92fail&operator=and

So this is indeed a bug in ToT.

One thing I see in /var/log/messages for the bad runs but not the good run I am comparing against is:

2018-03-07T05:39:46.231493+00:00 INFO kernel: [   31.769299] crossystem[3301]: segfault at 22 ip 000055c0b9496e2e sp 00007fffbf5b1d50 error 4 in crossystem[55c0b9494000+9000]

From the crash dump looks like crossystem is crashing

exec_name=crossystem
ver=10468.0.0-rc3
payload=/var/spool/crash/crossystem.20180307.094340.4024.dmp
payload_size=46120
done=1

Figured out how to do stack dump with symbols

Operating system: Linux
                  0.0.0 Linux 4.4.120-13171-g478a13d28edc #1 SMP PREEMPT Wed Mar 7 09:04:25 PST 2018 x86_64
CPU: amd64
     family 6 model 42 stepping 1
     8 CPUs

GPU: UNKNOWN

Crash reason:  SIGSEGV
Crash address: 0x0
Process uptime: not available

Thread 0 (crashed)
 0  crossystem!vb2_get_nv_storage + 0x2c
    rax = 0x0000000000000000   rdx = 0x00005a61898590c0
    rcx = 0x0000000000000000   rbx = 0x0000000000000013
    rsi = 0x0000000000000000   rdi = 0x00005a6189859140
    rbp = 0x00007ffe755d2660   rsp = 0x00007ffe755d2650
     r8 = 0x0000000000000000    r9 = 0x00000000000000ef
    r10 = 0x00007ba172d71530   r11 = 0x0000000000000246
    r12 = 0x00007ffe755d66a8   r13 = 0x0000000000000000
    r14 = 0x0000000000000000   r15 = 0x0000000000000000
    rip = 0x00005a6189851e2e
    Found by: given as instruction pointer in context
 1  crossystem!VbGetSystemPropertyInt.part.2 + 0x50
    rbx = 0x00005a6189854f48   rbp = 0x00007ffe755d6690
    rsp = 0x00007ffe755d2670   r12 = 0x00007ffe755d66a8
    r13 = 0x0000000000000000   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00005a618985263d
    Found by: call frame info
 2  crossystem!PrintAllParams + 0x60
    rbx = 0x00005a6189858360   rbp = 0x00007ffe755d86d0
    rsp = 0x00007ffe755d66a0   r12 = 0x00007ffe755d66a8
    r13 = 0x0000000000000000   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00005a6189851a43
    Found by: call frame info
 3  libc-2.23.so!__libc_start_main [libc-start.c : 289 + 0x1a]
    rbx = 0x0000000000000000   rbp = 0x00007ffe755d87a0
    rsp = 0x00007ffe755d86e0   r12 = 0x00005a6189854d60
    r13 = 0x00007ffe755d87c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007ba172c5d736
    Found by: call frame info
 4  crossystem!_start + 0x29
    rbx = 0x0000000000000000   rbp = 0x0000000000000000
    rsp = 0x00007ffe755d87b0   r12 = 0x00005a61898515d0
    r13 = 0x00007ffe755d87c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00005a61898515f9
    Found by: call frame info
 5  0x7ffe755d87b8
    rbx = 0x0000000000000000   rbp = 0x0000000000000000
    rsp = 0x00007ffe755d87b8   r12 = 0x00005a61898515d0
    r13 = 0x00007ffe755d87c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007ffe755d87b8
    Found by: call frame info
 6  crossystem!main + 0x290
    rsp = 0x00007ffe755d8900   rip = 0x00005a61898515d0
    Found by: stack scanning

Loaded modules:
0x5a618984f000 - 0x5a6189857fff  crossystem  ???  (main)
0x7ba172c3d000 - 0x7ba172dddfff  libc-2.23.so  ???
0x7ba172fe8000 - 0x7ba17300bfff  ld-2.23.so  ???
0x7ffe755e8000 - 0x7ffe755e9fff  linux-gate.so  ???
2018-03-07 13:40:08: minidump.cc:4980: INFO: Minidump closing minidump

Adding the last two people who made changes in sigsegv - not that these changes are likely to have caused the issue but in case they might be able to help me figure out why crossystem is crashing.

s/who made changes in sigsegv/who made changes in crossystem/

The good news it is reproducible on the VM - not on actual guados

moblab@localhost /var/log/bootup $ crossystem
arch                   = x86                            # Platform architecture
Segmentation fault (core dumped)

 Issue 819576  has been merged into this issue.

It is unclear why the crossystem makes the gs_offloader fail - on my VM gs_offloader started, however after a segv there are crash reporters running etc so I am not sure how reliable the system is at that point.

The autotest code that calls crossystem should be hardened to deal more gracefully with crossystem failures https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/utils.py?l=1105

I am trying to build a debug crossystem to try to get a good traceback to the problem and try to find a culprit CL - or just add in some failure handling to the code to prevent the segv

i guess it only crashes when run inside the VM ?  odd that it only crashes on moblab as we do quite a bit of VM tests elsewhere.

running `crossystem` on my VM from vboot_reference-1.0-r1465 runs fine.

This CL is the cause of the failure

https://chromium-review.googlesource.com/c/chromiumos/platform/vboot_reference/+/942031

Reverting it gets crossystem to work again on the moblab vm

Double-checking my change...

I think the problem is this:

VbSharedDataHeader* sh = VbSharedDataRead();
if (sh->flags & VBSD_NVDATA_V2)

It works on a real system, but I'm not checking the return value before I dereference it, so if it returns NULL on the VM it'll die.  The other places where that's called all check if (!sh).

Fix coming imminently.

Fix here:

https://chromium-review.googlesource.com/954712

Thanks for testing the fix.

Not sure if you want to keep the bug open to cover hardening autotest so it'll report a more graceful error the next time I break crossystem.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/bff4a078938035f865b320fcecb9f456f866c7da

commit bff4a078938035f865b320fcecb9f456f866c7da
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Mar 08 08:37:51 2018

chromeos_config: Mark more flaky paladins experimental.

These are currently marked experimental via tree status. They have
failed more than 2 builds in the last 24 hours. Bugs are being actively
worked on. Tree status is an unreliable place for this, so mark them
here.

BUG=chromium:819287
BUG=chromium:819017
BUG=chromium:794242
BUG= chromium:819695 
TEST=unittests

Change-Id: I8c2d3a9339dfd32e9123bf66a5a0adb0107d5032
Reviewed-on: https://chromium-review.googlesource.com/953336
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>

[modify] https://crrev.com/bff4a078938035f865b320fcecb9f456f866c7da/cbuildbot/config_dump.json
[modify] https://crrev.com/bff4a078938035f865b320fcecb9f456f866c7da/cbuildbot/chromeos_config.py

Downgrading the bug - the fix is stuck in the CQ - but coming.

Leaving bug open to try to do better job of handling errors when crossystem crashes.

Thanks much rspangler@ and haddowk@ for digging into this one.
It was a pesky flaky failure, so thanks for going after it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/0bdb8713be40abfe963d9ef625dbb67961068840

commit 0bdb8713be40abfe963d9ef625dbb67961068840
Author: Randall Spangler <rspangler@chromium.org>
Date: Thu Mar 08 19:33:26 2018

crossystem: Fix null pointer dereference on VMs

Check the result of VbSharedDataRead() before dereferencing it.

BUG=chromium:789276, chromium:819695 
BRANCH=none
TEST=make runtests

Change-Id: I1b1cc90bdc2fca61a9aad6b02e8b7e1f6a919797
Signed-off-by: Randall Spangler <rspangler@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954712
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/0bdb8713be40abfe963d9ef625dbb67961068840/host/lib/crossystem.c

This does not seem to have been the only problem - looking again

Working on why devserver is catching a SIGTERM and shutting down

[10/Mar/2018:08:09:15] ENGINE Listening for SIGHUP.
[10/Mar/2018:08:09:15] ENGINE Listening for SIGTERM.
[10/Mar/2018:08:09:15] ENGINE Listening for SIGUSR1.
[10/Mar/2018:08:09:15] ENGINE Bus STARTING
[10/Mar/2018:08:09:15] ENGINE Started monitor thread '_TimeoutMonitor'.
[10/Mar/2018:08:09:15] ENGINE Serving on :::8080
[10/Mar/2018:08:09:15] ENGINE Bus STARTED
[10/Mar/2018:08:10:53] ENGINE Caught signal SIGTERM.
[10/Mar/2018:08:10:53] ENGINE Bus STOPPING
[10/Mar/2018:08:10:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 8080)) shut down
[10/Mar/2018:08:10:53] ENGINE Stopped thread '_TimeoutMonitor'.
[10/Mar/2018:08:10:53] ENGINE Bus STOPPED
[10/Mar/2018:08:10:53] ENGINE Bus EXITING
[10/Mar/2018:08:10:53] ENGINE Bus EXITED
[10/Mar/2018:08:10:53] ENGINE Waiting for child threads to terminate...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/db292eb9a421630f2881e33ba969e9bbe5660c17

commit db292eb9a421630f2881e33ba969e9bbe5660c17
Author: Keith Haddow <haddowk@chromium.org>
Date: Sun Mar 11 05:12:30 2018

[moblab] Add sleep before trying to detect USB drive.

On the VM it seems like readlink command will hang if called too
early in the boot, this stops the USB drive being mounted and
results in the dev server crashing.

For now add a sleep - however the real issues is that on VM when
cros-disks is started the filesystem still seems unstable.

BUG= chromium:819695 
TEST=local tests on moblab vm

Change-Id: I1f6125a62f5e29f6b4f16a46f23f50e6c261c60c
Reviewed-on: https://chromium-review.googlesource.com/957835
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>

[modify] https://crrev.com/db292eb9a421630f2881e33ba969e9bbe5660c17/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-external-storage-init.conf

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/5923868cc36379039823cf1fb836b70ec866c41d

commit 5923868cc36379039823cf1fb836b70ec866c41d
Author: Keith Haddow <haddowk@chromium.org>
Date: Mon Mar 12 09:45:25 2018

[moblab] Improve debugging in the upstart scripts

Move all the init script to start with moblab*

Add new upstart handler that logs all the events and return
codes to moblab*

Set -e on scripts so we try to exit as quickly as possible on
script errors

Add ethtool command because -e will not exit when in an if or
in a piped command - but we want it to fail quickly.

BUG= chromium:819695 
TEST=tryjobs on both vm and guado_moblab

Change-Id: Iaba4f11c0ce14c2cbe7e1944bc0f451fe9ad1881
Reviewed-on: https://chromium-review.googlesource.com/957865
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>

[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-host-scheduler-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-gsoffloader_s-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-base-container-init.conf
[rename] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-mobmonitor-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-apache-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-dhcpd-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-network-bridge-init.conf
[rename] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-whining-update-db.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-containers-dir-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-cleanup-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-homedir-init.conf
[add] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/boot-logging-moblab.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-scheduler-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-external-storage-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-gsoffloader-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-database-test-importer.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf
[modify] https://crrev.com/5923868cc36379039823cf1fb836b70ec866c41d/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-database-init.conf

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5bc6f2a2ef1d3c849783565209ffa16af11a9b77

commit 5bc6f2a2ef1d3c849783565209ffa16af11a9b77
Author: Keith Haddow <haddowk@chromium.org>
Date: Mon Mar 19 21:19:07 2018

[chromite] Make moblab-vm important again

Build has been green for some time

BUG= chromium:819695 
TEST=None

Change-Id: I8b0ddcf992b0a06e1d27d45ca5e29eb4f113b4fd
Reviewed-on: https://chromium-review.googlesource.com/961536
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/5bc6f2a2ef1d3c849783565209ffa16af11a9b77/cbuildbot/config_dump.json
[modify] https://crrev.com/5bc6f2a2ef1d3c849783565209ffa16af11a9b77/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bb4fdab05c5a9dccbf93ee2f598d081316c54890

commit bb4fdab05c5a9dccbf93ee2f598d081316c54890
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Tue Mar 20 00:15:24 2018

Roll src/third_party/chromite/ 8d50e94a5..5978d7dd7 (2 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/8d50e94a5bc9..5978d7dd749e

$ git log 8d50e94a5..5978d7dd7 --date=short --no-merges --format='%ad %ae %s'
2018-03-14 dgarrett builder_status_lib: Remove GetBuilderStatusFromCIDB.
2018-03-13 haddowk [chromite] Make moblab-vm important again

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:821986 , chromium:819695 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ie1aad5cf3d65e808393afe189cd50940170a2762
Reviewed-on: https://chromium-review.googlesource.com/969603
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#544225}
[modify] https://crrev.com/bb4fdab05c5a9dccbf93ee2f598d081316c54890/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7

commit 06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7
Author: Keith Haddow <haddowk@chromium.org>
Date: Tue May 01 20:05:56 2018

[moblab] Improve debugging in the upstart scripts

Move all the init script to start with moblab*

Add new upstart handler that logs all the events and return
codes to moblab*

Set -e on scripts so we try to exit as quickly as possible on
script errors

Add ethtool command because -e will not exit when in an if or
in a piped command - but we want it to fail quickly.

BUG= chromium:819695 
TEST=tryjobs on both vm and guado_moblab

Change-Id: Iaba4f11c0ce14c2cbe7e1944bc0f451fe9ad1881
Reviewed-on: https://chromium-review.googlesource.com/957865
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1038130
Commit-Queue: Keith Haddow <haddowk@chromium.org>

[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-host-scheduler-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-gsoffloader_s-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-base-container-init.conf
[rename] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-mobmonitor-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-apache-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-dhcpd-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-network-bridge-init.conf
[rename] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-whining-update-db.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-containers-dir-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-cleanup-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-homedir-init.conf
[add] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/boot-logging-moblab.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-scheduler-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-external-storage-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-gsoffloader-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-database-test-importer.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf
[modify] https://crrev.com/06eb6fb74e6ab16b0f49751135fdc88ac2a44ec7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-database-init.conf