New issue
Advanced search Search tips

Issue 793003 link

Starred by 1 user
Status: WontFix
Owner:
pprabhu@chromium.org
Closed: Dec 2017
Cc:
haddowk@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 725620



Sign in to add a comment

moblab-vm: moblab-database-init fails after reboot

Project Member Reported by pprabhu@chromium.org, Dec 7 2017 
I hope I'm terribly confused:

moblab runs mysqld via the moblab-database-init upstart job.
This upstart job actually tries to:
 pre-start: setup some logs, install mysql db
 exec: run mysqld
 post-start: Drop all tables, recreate db from scratch.

So, if this job were to succeed, moblab would lose its entire AFE DB on every reboot.
But, wait, one step in pre-start is "expected to fail" once the setup is complete, as a comment there indicates.


But that also means that:
  - pre-start fails
  - myslqd is never started
  - the job never reaches the "started" state -- this has downstream effect (moblab-external-storage-init doesn't run, nothing else moblab specific runs)


So, by my reading, once you reboot a moblab device, essential moblab services fail to come up.

We wouldn't notice this in the CQ since we only run a handful of tests after each provision, and don't reboot moblab between them.

But wouldn't partners notice this? Do they **never** restart moblabs between updates? OMG!

Pri 1: because this makes debugging moblab-pre-cq failures difficult. I package the VM images for debugging, but you can't even boot the moblab VM completely.

Comment 1 by pprabhu@chromium.org, Dec 7 2017

Owner: pprabhu@chromium.org
To verify that this also happens on a physical moblab, I locked chromeos2-row1-rack8-host5.
guado_moblab doesn't seem to be affected negatively by this...


localhost ~ # grep 'Starting' /var/log/bootup/moblab-database-init.log 
+ logger -t moblab-database-init Pre Start Starting.
+ logger -t moblab-database-init Starting.

localhost ~ # grep 'Ending' /var/log/bootup/moblab-database-init.log                                                                                                                           
+ logger -t moblab-database-init Ending.


localhost ~ # initctl list | grep moblab
moblab-containers-dir-init stop/waiting
moblab-devserver-init start/running, process 3578
moblab-homedir-init stop/waiting
moblab-apache-init stop/waiting
moblab-devserver-cleanup-init start/running, process 3580
moblab-host-scheduler-init start/running, process 3828
moblab-gsoffloader_s-init start/running, process 3829
moblab-base-container-init stop/waiting
moblab-database-test-importer stop/waiting
moblab-database-init start/running, process 2978
moblab-external-storage-init stop/waiting
moblab-gsoffloader-init start/running, process 3831
moblab-scheduler-init start/running, process 3824
moblab-dhcpd-init stop/waiting
moblab-network-bridge-init stop/waiting


------------------
REBOOT MOBLAB
------------------

localhost ~ # initctl list | grep moblab
moblab-containers-dir-init stop/waiting
moblab-devserver-init start/running, process 3247
moblab-homedir-init stop/waiting
moblab-apache-init stop/waiting
moblab-devserver-cleanup-init start/running, process 3249
moblab-host-scheduler-init start/running, process 3497
moblab-gsoffloader_s-init start/running, process 3502
moblab-base-container-init stop/waiting
moblab-database-test-importer start/running, process 2984
moblab-database-init start/running, process 1615
moblab-external-storage-init stop/waiting
moblab-gsoffloader-init start/running, process 3505
moblab-scheduler-init start/running, process 3498
moblab-dhcpd-init stop/waiting
moblab-network-bridge-init stop/waiting


localhost ~ # grep Starting /var/log/bootup/moblab-database-init.log 
+ logger -t moblab-database-init Pre Start Starting.
+ logger -t moblab-database-init Starting.
+ logger -t moblab-database-init Pre Start Starting.
+ logger -t moblab-database-init Pre Start Starting.
+ logger -t moblab-database-init Starting.

localhost ~ # grep Ending /var/log/bootup/moblab-database-init.log 
+ logger -t moblab-database-init Ending.
+ logger -t moblab-database-init Ending.

-------------------

Something is killing the upstart job mid-stream, and then restarting it. Upon restart, it finds that it can actually run mysql_isntall_db successfully to completion (of course dropping all data before boot to the floor).

Comment 2 by pprabhu@chromium.org, Dec 7 2017

Summary: moblab-vm: moblab-database-init fails after reboot (was: moblab: restarting moblab breaks it. Really.)
(1) This only seems to affect moblab-vm.
On guado_moblab, the pre-start setup of the DB fails, but the exec still happens.

Then, post-start has switches to only migrate DB if needed etc.

Comment 3 by pprabhu@chromium.org, Dec 7 2017

Blocking: 725620

Comment 4 by pprabhu@chromium.org, Dec 7 2017

Status: WontFix (was: Untriaged)
moblab-vm behaved the same as guado_moblab on next reboot.
Something is racy here... but I don't have my finger on what.
►

Sign in to add a comment