We need to either disable one of these paladin hwtests, or move it to a dedicated pool. We can't support concurrent CQs within a single pool due to provision thrashing, which is a long standing limitation.

 -> Bernie for input on which course is more desired.

This is tricky, so even if we have a 2x deployment of DUTs (as we do for Kevin) it can still step on itself.

I guess we could split the cq pool (say cq and cq-arcnext) for Kevin and put in a hack so that we run on the appropriate pool?

If we have to actually tear one down I think we could probably replace kevin more easily than kevin-arcnext, we already have bob in the CQ, and if we can get them stable we probably could add in scarlet (dru). 

> This is tricky, so even if we have a 2x deployment of DUTs (as we do for Kevin) it can still step on itself.

Yes, with the current scheduler. Quotascheduler should eventually resolve this.

> I guess we could split the cq pool (say cq and cq-arcnext) for Kevin and put in a hack so that we run on the appropriate pool?

Yep, that's doable if we have enough devices. We currently have 16 devices in pool:cq. We'd need to create a new managed pool so that devices get autorepaired, which is a bit of additional development overhead.

> If we have to actually tear one down I think we could probably replace kevin more easily than kevin-arcnext, we already have bob in the CQ, and if we can get them stable we probably could add in scarlet (dru). 

That's easier than above. Should be just go ahead and do it?

While I don't like the reduction in coverage, if it is necessary for stability we can drop kevin for now if that is the best course of action we have available. 

https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1187546

Is this still relevant? I have a CL languishing for disabling these tests, but not sure if still relevant.

It seemed to me that the recent paladin failed due to similar reasons?

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/19700

I saw this in the log (cc'ing ihf@ who I mentioned this to over hangouts...)

...
09/26 13:35:19.184 INFO |    tradefed_utils:0032| Waiting for cache lock...
09/26 13:39:37.053 INFO |    tradefed_utils:0032| Waiting for cache lock...
09/26 14:26:08.636 ERROR|    tradefed_utils:0047| Permanent lock failure. Trying to break lock.
09/26 14:26:08.637 WARNI|              test:0606| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 567, in _exec
    _cherry_pick_call(self.initialize, *args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 715, in _cherry_pick_call
    return func(*p_args, **p_dargs)
  File "/usr/local/autotest/server/cros/tradefed_test.py", line 124, in initialize
    self._clean_download_cache_if_needed()
  File "/usr/local/autotest/server/cros/tradefed_test.py", line 483, in _clean_download_cache_if_needed
    with tradefed_utils.lock(self._tradefed_cache_lock):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/autotest/server/cros/tradefed_utils.py", line 54, in lock
    raise error.TestFail('Error: permanent cache lock failure.')
TestFail: Error: permanent cache lock failure.

Re #10, sorry, this is issue 885199.

As for the thrashing, I am not convinced the analysis is correct. We already schedule in the same pool builds from different branches. Scheduling kevin and kevin-arcnext in the same pool is no different (and they get the same priority but different spots in the queue). Maybe a race? Now the lab does provision too aggressively, as it tries to grab the whole pool instead of orienting itself on the minimum_duts parameter for instance.