For now, I believe we don't want to wait for the first slice too long, if no DUTs are available, (every DUTs with correct-provisioned build is busy), we want to grab other DUTs to run first. So I'm gonna set the first piece's expirations secs shorter than 5 mins to relieve the reef-paladin and see whether it goes better.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/651694146aed03326b994abc2789f61283fbd241

commit 651694146aed03326b994abc2789f61283fbd241
Author: Xixuan Wu <xixuan@chromium.org>
Date: Wed Aug 22 18:36:37 2018

autotest: Set expiration_secs as the whole task's expiration_secs.

BUG=chromium:876579
TEST=Test it on staging.

Change-Id: I44094203dc514d959f1f56be31cd07fd2efc55e6
Reviewed-on: https://chromium-review.googlesource.com/1185374
Reviewed-by: Xixuan Wu <xixuan@chromium.org>
Commit-Queue: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/651694146aed03326b994abc2789f61283fbd241/venv/skylab_suite/swarming_lib_unittest.py
[modify] https://crrev.com/651694146aed03326b994abc2789f61283fbd241/venv/skylab_suite/suite_runner.py
[modify] https://crrev.com/651694146aed03326b994abc2789f61283fbd241/venv/skylab_suite/swarming_lib.py

After testing, I can't pass "'expiration_secs': 5400" in the task level as it reports:

69879 2018-08-22 18:08:16.411 E: Request to https://chromium-swarm-dev.appspot.com/_ah/api/swarming/v1/tasks/new failed with HTTP status code 400: 400 Client Error: Bad Request for url: https://chromium-swarm-dev.appspot.com/_ah/api/swarming/v1/tasks/new - When using task_slices, do not specify a global expiration_secs

So I just pass-in different expiration secs for each slice temporarily in #2.

 Issue 877097  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/26f2ace0ee533e10d5d695af1e7bd37d98ad8984

commit 26f2ace0ee533e10d5d695af1e7bd37d98ad8984
Author: Xixuan Wu <xixuan@chromium.org>
Date: Thu Aug 23 21:08:03 2018

autotest: Set the provision slice expiration as 85mins (longer).

BUG=chromium:876579
TEST=None

Change-Id: Ie9bf9067e73229abd2eb5b0bc763c948815e1ace
Reviewed-on: https://chromium-review.googlesource.com/1187137
Reviewed-by: Xixuan Wu <xixuan@chromium.org>
Commit-Queue: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/26f2ace0ee533e10d5d695af1e7bd37d98ad8984/venv/skylab_suite/suite_runner.py

Based on observation in  Issue 877097 , I should set the expiration seconds of first slice much longer.

1) If I set the first slice's expiration seconds short, 5mins vs 85mins,

    Imagine that a task is pending for picking up, during the pending, the first slice will soon expire.
    So when this task is matched to one bot, the first slice is already expired? so it's the second slice that's matched to the bot, and the matched bot will get provisioned again.

2) If I set the first slice's expiration seconds long, 85mins vs 5mins, 

    Imagine that a task is pending for picking up, during the pending, the first slice won't expire.
    So when this task is matched to one bot, the first slice is used. Since 'wait_for_capacity' by default is False, if there's no available bots, second slice will be used for matching. This sounds more reasonable.

The current solution is for CQ jobs, the expiration seconds are split by 95% and 5%.

The reason is in #6.

For non-CQ jobs, the expiration seconds are split by 5% and 95%,

as the first slice may expire very soon due to 'wait_for_capacity'=False. So it quickly jumps to the second slice. If the expiration seconds of second slice is short, these non-CQ jobs will soon expire.

As you correctly figured out. I don't think I have any AI here?

I'll take it for now.  The quota scheduler or similar project should take care of the shortcomings here, under the current suites model I don't think we can do much better.