I checked the database, it seems the host-scheduler is scheduling a completed job to the host again.

E.g
05/11 09:04:46.853 INFO |  scheduler_models:0530| Assigning host chromeos2-row6-rack2-host4 to entry HQE: 63135235, for job: 62804994 and host: no host has status:Queued

$select * from afe_host_queue_entries where host_id=3246 and job_id>62804934 and job_id<62805000;
+----------+----------+---------+-----------+-----------+--------+----------+---------+----------------------------+-----------------+---------+---------------------+---------------------+
| id       | job_id   | host_id | status    | meta_host | active | complete | deleted | execution_subdir           | atomic_group_id | aborted | started_on          | finished_on         |
+----------+----------+---------+-----------+-----------+--------+----------+---------+----------------------------+-----------------+---------+---------------------+---------------------+
| 63135174 | 62804935 |    3246 | Completed |      NULL |      0 |        1 |       0 | chromeos2-row6-rack2-host4 |            NULL |       0 | 2016-05-10 12:07:54 | 2016-05-10 12:10:53 |
| 63135234 | 62804994 |    3246 | Completed |      NULL |      0 |        1 |       0 | chromeos2-row6-rack2-host4 |            NULL |       0 | 2016-05-10 12:11:20 | 2016-05-10 12:14:27 |
+----------+----------+---------+-----------+-----------+--------+----------+---------+----------------------------+-----------------+---------+---------------------+---------------------+
2 rows in set (0.00 sec)

The job 62804994 has finished yesterday.

It is always failing for the same host-queue-entry 63135234. 

mysql> select * from afe_host_queue_entries where job_id=62804994;
+----------+----------+---------+-----------+-----------+--------+----------+---------+----------------------------+-----------------+---------+---------------------+---------------------+
| id       | job_id   | host_id | status    | meta_host | active | complete | deleted | execution_subdir           | atomic_group_id | aborted | started_on          | finished_on         |
+----------+----------+---------+-----------+-----------+--------+----------+---------+----------------------------+-----------------+---------+---------------------+---------------------+
| 63135234 | 62804994 |    3246 | Completed |      NULL |      0 |        1 |       0 | chromeos2-row6-rack2-host4 |            NULL |       0 | 2016-05-10 12:11:20 | 2016-05-10 12:14:27 |
| 63135235 | 62804994 |    3285 | Completed |      3217 |      0 |        1 |       0 | chromeos2-row6-rack3-host4 |            NULL |       0 | 2016-05-11 09:24:40 | 2016-05-11 09:28:17 |
+----------+----------+---------+-----------+-----------+--------+----------+---------+----------------------------+-----------------+---------+---------------------+---------------------+
2 rows in set (0.00 sec)

For some reason the job gets two host-queue-entries and one of them has meta_host=NULL. I believe this is incorrect. 
meta_host is used for determine jobs scheduled from frontend, so maybe scheduler got confused by it.
I manually updated it to 3217. Let's see if the problem will happen again.

+  tienchang@chromium.org
looks like this job is scheduled by Tien manually from frontend?
http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=62804994

Tien, could you provide some details on how you scheduled the job?

Yes! I created the host via the frontend. I believe this is what I did ...

This was the job:
http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=62804343

Wanting to rule out (or confirm) a failure, I cloned the job. I wanted to chose a specific hostname, but I already clicked "Reuse any specific hosts (default)". No matters, because I could just select a host while creating the job. In the Hostname field, I entered "chromeos2-row6-rack2-host4" while still on the "By browsing hosts" tab. I then selected (confirmed by the yellow highlight) chromeos2-row6-rack2-host4.

Maybe that's why there were 2 queued jobs? After reviewing my steps, I seem horrible at following directions!

tien@ thanks for you information. That's very useful.
I think I know what's happening.

In #4, a job with two hosts are created, which resulting in two hqes
hqe 63135234 -- this one was created with a specific host chromeos2-row6-rack2-host4, and meta_host=NULL

hqe 63135235 -- this one was created with meta_host=3217 (because user selected Reuse any similar hosts), but not specific host.

Our host-scheduler can't deal with such situation.
It tried to assign chromeos2-row6-rack2-host4 to 63135235, because 
 - 63135235 needs a host
 - chromeos2-row6-rack2-host4 is ready/unleased
However,  chromeos2-row6-rack2-host4 was already assigned to 63135234 (as user requested), so the assignment triggers db error.

The problem went away after another dut become available (chromeos2-row6-rack3-host4), and 63135235 finally get this one.

Interesting problem. Fang, thanks for figuring this out!

Potential fix:
 - Ban such use case, that is, if user selects "Reusing any similar hosts", do now allow user to add an additional dut later on in the next tab. 

 - Or if we allow this use case, when user added an additional dut, it should not lead to an hqe with meta_host=NULL being created.

The problem is still there :p I asked Tian to not do it again, but we can't guarantee someone else won't. P3 as not critical, good candidate for Fixit

Supported for all options reopen 

 Issue 634064  has been merged into this issue.

Hey guys we need to raise the priority of this. If Danny can make this mistake at Google, our partners can make this mistake with their MobLabs.

Either:
* Update the host scheduler to properly handle this situation (which is likely the correct action).
* Disable multiple host selection in the AFE (Sounds like we're hiding a problem to me).

will take a look

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/61f5d31b6f6173069e1ad5ce28c739666398e237

commit 61f5d31b6f6173069e1ad5ce28c739666398e237
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Sat Aug 06 00:15:23 2016

[autotest]only accept hosts when both hosts and meta_hosts given for a clone job

When cloning a job via autotest frontend, if user chooses the default
clone action, which is 'Reuse any similar hosts' and then later still
select specific hosts in the next tab, the host scheduler will get
confused about the case and would try to schedule two same jobs to one
host. In this case, the host scheduler will crash because of duplicate entries
for one job in the database. The fix is to only accept the hosts
variables under this situation.

BUG= chromium:611064 
TEST=unittest

Change-Id: Id945a9f8f2b45271c903f1ec038366da47f5d436
Reviewed-on: https://chromium-review.googlesource.com/366795
Commit-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/61f5d31b6f6173069e1ad5ce28c739666398e237/frontend/afe/rpc_interface.py