I believe the database needs to be poked at and have the entry purged.

Any updates?

rsabanga@rsabanga-glaptop0:~$ atest host create chromeos2-row6-rack3-host5
Operation add_host failed:
    MultipleObjectsReturned: get() returned more than one Host -- it returned 2! Lookup parameters were {'hostname': 'chromeos2-row6-rack3-host5', 'invalid': True}
rsabanga@rsabanga-glaptop0:~$ atest host create chromeos2-row6-rack3-host14
Operation add_host failed:
    MultipleObjectsReturned: get() returned more than one Host -- it returned 2! Lookup parameters were {'hostname': 'chromeos2-row6-rack3-host14', 'invalid': True}
rsabanga@rsabanga-glaptop0:~$ 

> As deployment_test is not currently working for us (crbug.com/638969) we've
> been trying to add hosts manually.

The problem with deployment_test is now bug 646519.  There's
supposed to be a fix for that problem in place.  Please run
update_lab_tools, and test whether deployment_test works.

jrbarnette the main issue is a bit different as it seems there is a duplicate entry in the database which can't be removed via CLI or AFE.

I still need to know whether deployment_test is working.
Also, to be blunt, only deployment_test really matters.  If
`atest host create` fails, that's a low-priority bug.  For a
high-priority bug, it needs to happen with `deployment_test`,
too.

deployment_test cave chromeos2-row6-rack3-host5
Installation output logs in /home/strongj/Documents/2016-09-15T11:09:24.603010-07:00-cave
AFE   version is R53-8530.81.0.
Omaha version is R53-8530.81.0.

============
Repair version for board cave is now R53-8530.81.0.

============
Failures
============
chromeos2-row6-rack3-host5     MultipleObjectsReturned: get() returned more than one Host -- it returned 2! Lookup parameters were {'hostname': 'chromeos2-row6-rack3-host5', 'invalid': True}
Traceback (most recent call last):
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 114, in dispatchRequest
    results['result'] = self.invokeServiceEndpoint(meth, args)
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 154, in invokeServiceEndpoint
    return meth(*args)
  File "/usr/local/autotest/frontend/afe/rpc_handler.py", line 270, in new_fn
    return f(*args, **keyword_args)
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 298, in add_host
    protection=protection).id
  File "/usr/local/autotest/frontend/afe/model_logic.py", line 670, in add_object
    obj.save()
  File "/usr/local/autotest/frontend/afe/models.py", line 611, in save
    super(Host, self).save(*args, **kwargs)
  File "/usr/local/autotest/frontend/afe/model_logic.py", line 1247, in save
    old_object = self.__class__.objects.get(**filters)
  File "/usr/local/autotest/site-packages/django/db/models/manager.py", line 143, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/usr/local/autotest/site-packages/django/db/models/query.py", line 393, in get
    (self.model._meta.object_name, num, kwargs))
MultipleObjectsReturned: get() returned more than one Host -- it returned 2! Lookup parameters were {'hostname': 'chromeos2-row6-rack3-host5', 'invalid': True}


Installation complete:  0 successes, 1 failures.
Logs will be uploaded to gs://chromeos-install-logs/2016-09-15T11:09:24.603010-07:00-cave
Failed to upload logs; failure details are stored in /home/strongj/Documents/2016-09-15T11:09:24.603010-07:00-cave/gs_upload_failure.log.

Deleted: chromeos2-row6-rack3-host5.log 4.0 KB

chromeos2-row6-rack3-host5.log 4.0 KB View Download

These are the problem children:

mysql> select hostname, COUNT(*) c FROM afe_hosts where invalid=1 group by hostname having c > 1;
+-----------------------------+---+
| hostname                    | c |
+-----------------------------+---+
| 172.31.15.168               | 2 |
| 172.31.26.196               | 2 |
| chromeos2-row6-rack3-host10 | 2 |
| chromeos2-row6-rack3-host11 | 2 |
| chromeos2-row6-rack3-host12 | 2 |
| chromeos2-row6-rack3-host13 | 2 |
| chromeos2-row6-rack3-host14 | 2 |
| chromeos2-row6-rack3-host5  | 2 |
| chromeos2-row6-rack3-host6  | 2 |
| chromeos2-row6-rack3-host7  | 2 |
| chromeos2-row6-rack3-host8  | 2 |
| chromeos2-row6-rack3-host9  | 2 |
| chromeos3-row4-rack4-host19 | 2 |
| ping                        | 2 |
+-----------------------------+---+
14 rows in set (0.00 sec)

OK.  I've applied the following changes:

update afe_hosts set hostname="invalid-115" where id=115;
update afe_hosts set hostname="invalid-139" where id=139;
update afe_hosts set hostname="invalid-5687" where id=5687;
update afe_hosts set hostname="invalid-5688" where id=5688;
update afe_hosts set hostname="invalid-5692" where id=5692;
update afe_hosts set hostname="invalid-5690" where id=5690;
update afe_hosts set hostname="invalid-5689" where id=5689;
update afe_hosts set hostname="invalid-3286" where id=3286;
update afe_hosts set hostname="invalid-3287" where id=3287;
update afe_hosts set hostname="invalid-5686" where id=5686;
update afe_hosts set hostname="invalid-5685" where id=5685;
update afe_hosts set hostname="invalid-5691" where id=5691;
update afe_hosts set hostname="invalid-5533" where id=5533;
update afe_hosts set hostname="invalid-3300" where id=3300;

Afterwards, I get this:
mysql> select id, hostname, invalid, COUNT(*) c FROM afe_hosts group by hostname having c > 1;
Empty set (0.03 sec)