I'm rebooting moblab now. 

Matt can you look into this please.

I think the database got deadlocked

apache_error
[Wed Nov 14 11:47:32.086854 2018] [core:error] [pid 32733] [client ::1:53899] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/
[Wed Nov 14 11:47:32.877566 2018] [core:error] [pid 32766] [client ::1:53903] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/
[Wed Nov 14 11:47:33.430662 2018] [core:error] [pid 328] [client ::1:53904] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/
[Wed Nov 14 11:47:33.437313 2018] [core:error] [pid 329] [client ::1:53906] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/
[Wed Nov 14 11:47:34.431645 2018] [core:error] [pid 360] [client ::1:53907] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/
[Wed Nov 14 11:47:34.431870 2018] [core:error] [pid 363] [client ::1:53908] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/
[Wed Nov 14 11:47:34.440557 2018] [core:error] [pid 362] [client ::1:53910] Script timed out before returning headers: frontend.wsgi, referer: http://localhost/afe/


messages:
2018-11-14T10:17:48.008187-08:00 NOTICE moblab-kill-long-running-sql-query[19363]: Killing 9 processes.

mysqld.err:
lots of
--Thread 134640451975936 has waited at dict0dict.cc line 1140 for 298.00 seconds the semaphore:
Mutex at 0x62c79eeb8918 '&dict_sys->mutex', lock var 1
Last time reserved by thread 134635546330880 in file not yet reserved line 0, waiters flag 1
InnoDB: Warning: semaphore wait:

Mengqi, what were the symptoms? Nothing on localhost:80 would load? But Mobmonitor loads fine? Did you end up rebooting?

We have an upstart job to kill long running queries, but looks like it didn't kill these.

Nothing would load on localhost.
Mobmonitor loads find (at least UI seems fine).
I ended up rebooting. 

And the reboot resolved the issue?

What were you doing on the moblab before it froze?

Yes, moblab came back.

I don't remember doing anything. I think it might have been sleeping.