This is a production shard.

akeshet@akeshet:~/chromiumos/src$ atest shard list | grep chromeos-skunk-1
208  chromeos-skunk-1.mtv.corp.google.com    board:auron_paine
akeshet@akeshet:~/chromiumos/src$ atest server list | grep chromeos-skunk-1
Hostname     : chromeos-skunk-1.mtv.corp.google.com

(was there an alert about this shard? I didn't see one

https://viceroy.corp.google.com/chromeos/deputy-view#_VG_lnuPnWCa

I'm perplexed by the failure. There is only 1 chromeos-skunk-1 entry in the database.

There is also a chromeos-skunk1 . We are emitting a wildcard query somehow that is matching both?

Also, why is shard_client emitting a django query at all? Which db is it querying?

I thought shard_client is only supposed to use RPCs to interact with master.

shard_client is hitting the local database, where there are indeed two entries in shard_table;

I don't know why we are even hitting local shard_table, makes no sense, but I will fix this incident by deleting the errant entry.

akeshet@akeshet:~/chromiumos/src/third_party/autotest/files$ autotest-db chromeos-skunk-1.mtv.corp.google.com
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3761212
Server version: 5.5.57-0ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from afe_shards;
+-----+--------------------------------------+
| id  | hostname                             |
+-----+--------------------------------------+
| 207 | chromeos-skunk-1.mtv.corp.google.com |
| 208 | chromeos-skunk-1.mtv.corp.google.com |
+-----+--------------------------------------+
2 rows in set (0.00 sec)

Entry #207 is the wrong one, deleting it.

Innnteresting... one of the DUTs in the shard-db is still stuck on id=207.

mysql> DELETE FROM afe_shards where id=207;
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`chromeos_autotest_db`.`afe_hosts`, CONSTRAINT `hosts_to_shard_ibfk` FOREIGN KEY (`shard_id`) REFERENCES `afe_shards` (`id`))
mysql> SELECT * from afe_hosts where shard_id=207;
+------+----------------------------+--------+----------+---------------+---------+------------+--------------+-----------+-------+--------+----------+-------------+
| id   | hostname                   | locked | synch_id | status        | invalid | protection | locked_by_id | lock_time | dirty | leased | shard_id | lock_reason |
+------+----------------------------+--------+----------+---------------+---------+------------+--------------+-----------+-------+--------+----------+-------------+
| 4256 | chromeos4-row8-rack4-host8 |      0 |     NULL | Repair Failed |       1 |          0 |         NULL | NULL      |     1 |      1 |      207 |             |
+------+----------------------------+--------+----------+---------------+---------+------------+--------------+-----------+-------+--------+----------+-------------+

All other hosts on this shard db have the correct id. I'll just update this host to match.

Have to do the same for afe_jobs...

mysql> select COUNT(*) from afe_jobs where shard_id=207;
+----------+
| COUNT(*) |
+----------+
|    15581 |
+----------+
1 row in set (0.00 sec)

mysql> select COUNT(*) from afe_jobs where shard_id=208;
+----------+
| COUNT(*) |
+----------+
|      120 |
+----------+
1 row in set (0.00 sec)

mysql> 


afe_jobs is pretty messed up, most of the jobs are associated with the incorrect shard id...

Ok, agressive fix time. I'm going to move this board:auron_paine from chromeos-skunk-1 to chromeos-skunk-2

However, since I'm worried about misbehavior from chromeos-skunk-1 I'm first going to remove it from serverdb and shard table. I may even wipe it.

auron_paine moved to chromeos-skunk-2. The new shard is ticking, I expect it should work correctly.

The old shard id exception-looping. I'm going to wipe it's tables to be safe...

(note: some job history may be lost, for jobs that ran on that shard)

akeshet@akeshet:~/chromiumos/chromeos-admin$ ./bin/run_server_task ShardCleanupTask --host_server chromeos-skunk-1.mtv.corp.google.com

^ did a lot of cleanup but also seemed to die in puppet due to Issue 770903

I believe the production issue is solved. Chase-Pending follow up questions:

1) How did the shard's local afe_shards table get inconsistent with master db.
   (theory: this shard was added and removed and re-added to production under a new id)
2) Should db_sentinel notice this inconsistency?
3) Should db_sentinel heal such inconsistency?

sentinel should enforce that every shard's afe_shards table is a subset of the master's

CL under review.

pprabhu to review

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/74496e17ebd0a2ad8bbcb4932c58de069cf3e2c6

commit 74496e17ebd0a2ad8bbcb4932c58de069cf3e2c6
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Tue Oct 24 05:39:05 2017

Either the logging is confusing, or something is not behaving correctly. I see the following message in the sentinel logs:

2017-11-15 12:28:47,727 ERRO| Shard 136 (chromeos-server36.cbf.corp.google.com) does not exist in master DB
2017-11-15 12:28:47,753 INFO| chromeos-server36.cbf.corp.google.com: Done.

However, if I examine atest shard list I see:

136  chromeos-server36.cbf.corp.google.com   board:guado, board:guado_moblab


I see the same message on many other shards too, in sentinel logs.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/d8075d2d36f02209266f72a27104b6cd596ec1cc

commit d8075d2d36f02209266f72a27104b6cd596ec1cc
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Thu Nov 16 07:52:12 2017