Status changed to "LUCI not WAI" (correctness 99%, speed 141%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 125%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 96%, speed 114%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 131%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 107%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

yep, the latest failure is

Traceback (most recent call last):
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/validate_story_expectation_data:8
    from core import story_expectation_validator
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/story_expectation_validator.py:12
    from core import benchmark_finders
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/benchmark_finders.py:10
    from core import perf_benchmark
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/perf_benchmark.py:8
    from telemetry import benchmark
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/third_party/catapult/telemetry/telemetry/benchmark.py:11
    from telemetry.internal import story_runner
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:15
    from py_utils import memory_debug  # pylint: disable=import-error
  <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/third_party/catapult/common/py_utils/py_utils/memory_debug.py:9
    import psutil
ImportError: No module named psutil

in https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8960173593790663904%2F%2B%2Fsteps%2Fpresubmit%2F0%2Fstdout

where presubmit_support.py is run via 'python', not 'vpython'. bug 784597 should fix the problem

Status changed to "LUCI WAI" (correctness 100%, speed 136%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 115%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 118%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 95%, speed 133%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Latest failures are still due to lack of psutil:

https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8959854993945153552%2F%2B%2Fsteps%2Fpresubmit%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8959864967047535312%2F%2B%2Fsteps%2Fpresubmit%2F0%2Fstdout

Status changed to "LUCI WAI" (correctness 100%, speed 125%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 98%, speed 123%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 119%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 97%, speed 123%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 115%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 96%, speed 117%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 123%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 98%, speed 120%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 121%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "Not enough data to analyze" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/f72286d0b3fbc87acfe7dd94182adcd2d28c20f8

commit f72286d0b3fbc87acfe7dd94182adcd2d28c20f8
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Jan 09 00:46:35 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/5afc2706e858b0d20b1da1711e2042e220fea23f

commit 5afc2706e858b0d20b1da1711e2042e220fea23f
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Jan 09 01:14:41 2018

Status changed to "LUCI not WAI" (correctness 98%, speed 139%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

This is up and running again. Going to let it cook for a day or so to make sure it's all sorted out.

Status changed to "LUCI WAI" (correctness 100%, speed 161%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 158%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 133%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 129%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 132%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 98%, speed 148%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 107%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 105%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 124%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 149%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 144%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 98%, speed 130%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 106%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 96%, speed 109%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 232%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 218%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 115%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 98%, speed 112%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 154%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI not WAI" (correctness 99%, speed 124%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Status changed to "LUCI WAI" (correctness 100%, speed 128%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

Neither I nor Dirk doesn't see reason not to migrate this builder. John, wdyt?

Plan is as follows:
1. Stop LUCI migration A/B testing.
2. Ramp up 50% CQ traffic splitting.
3. After 1-2 days, monitor peak usage and add more machines if necessary.
4. Ramp up to 100%.
5. Declare success :)

1. Stop LUCI migration A/B testing.
  Done.

2. Ramp up 50% CQ traffic splitting.
  https://chromium-review.googlesource.com/#/c/chromium/src/+/891666

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/38feaec854d6cc4a453791c47a60fb404960b79f

commit 38feaec854d6cc4a453791c47a60fb404960b79f
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Jan 29 19:26:31 2018

[CQ.cfg] 50% of chromium_presubmit builds will now run on LUCI.

R=dpranke@chromium.org, jbudorick@chromium.org

No-Try: True
Bug:  731553 
Change-Id: I12cd97bfee4d1cc7bf81f5c990014a27335a1c50
Reviewed-on: https://chromium-review.googlesource.com/891666
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532539}
[modify] https://crrev.com/38feaec854d6cc4a453791c47a60fb404960b79f/infra/config/cq.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/65f11ef409ba28b63d038e923bb04ff5ab3d9e11

commit 65f11ef409ba28b63d038e923bb04ff5ab3d9e11
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Jan 29 20:05:06 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c08b498086a1a7eb389a1afdfe10dfdf34f7ad21

commit c08b498086a1a7eb389a1afdfe10dfdf34f7ad21
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Jan 29 20:06:08 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/090a7c7dbfc9031a2c015ca9965898ce360971b6

commit 090a7c7dbfc9031a2c015ca9965898ce360971b6
Author: Andrii Shyshkalov <tandrii@google.com>
Date: Mon Jan 29 20:19:27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6e8f475479321840a4dd4f9de7f72944e699ddba

commit 6e8f475479321840a4dd4f9de7f72944e699ddba
Author: John Budorick <jbudorick@chromium.org>
Date: Tue Jan 30 14:16:26 2018

Revert "[CQ.cfg] 50% of chromium_presubmit builds will now run on LUCI."

This reverts commit 38feaec854d6cc4a453791c47a60fb404960b79f.

Reason for revert: commit-git-patch flakily fails on LUCI, e.g. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit/21881

Original change's description:
> [CQ.cfg] 50% of chromium_presubmit builds will now run on LUCI.
> 
> R=​dpranke@chromium.org, jbudorick@chromium.org
> 
> No-Try: True
> Bug:  731553 
> Change-Id: I12cd97bfee4d1cc7bf81f5c990014a27335a1c50
> Reviewed-on: https://chromium-review.googlesource.com/891666
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#532539}

TBR=dpranke@chromium.org,tandrii@chromium.org,jbudorick@chromium.org

Change-Id: I5ea0844f22c10335ffb0de5a360d76a792442781
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  731553 
Reviewed-on: https://chromium-review.googlesource.com/893418
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532858}
[modify] https://crrev.com/6e8f475479321840a4dd4f9de7f72944e699ddba/infra/config/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e34f3cb37fe35676402657a572ca041a085fc23c

commit e34f3cb37fe35676402657a572ca041a085fc23c
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Tue Jan 30 15:29:47 2018

Reland "[CQ.cfg] 50% of chromium_presubmit builds will now run on LUCI."

This is a reland of 38feaec854d6cc4a453791c47a60fb404960b79f.
It was reverted because of suspicion of causing red "git commit' failure builds.
However, that was due to a Gerrit bug. Hence, relanding.

Original change's description:
> [CQ.cfg] 50% of chromium_presubmit builds will now run on LUCI.
>
> R=dpranke@chromium.org, jbudorick@chromium.org
>
> No-Try: True
> Bug:  731553 
> Change-Id: I12cd97bfee4d1cc7bf81f5c990014a27335a1c50
> Reviewed-on: https://chromium-review.googlesource.com/891666
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#532539}

Bug:  731553 
No-Try: True
Change-Id: I89844ce9ae99ae2d2fadb19e6d0db5ab2f759415
Reviewed-on: https://chromium-review.googlesource.com/893538
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532883}
[modify] https://crrev.com/e34f3cb37fe35676402657a572ca041a085fc23c/infra/config/cq.cfg

Peak usage has been steady at 3-4, while we have 7 machines. I'm will be bumping to 90%.

#90: does that include yesterday, or is that just looking at this morning?

Yesterday. Except for the initial spike due to the switch -- see issue 806986.

CL: https://chromium-review.googlesource.com/#/c/chromium/src/+/893654

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff826a15755f495064d0b2404ffcb3353e2c4feb

commit ff826a15755f495064d0b2404ffcb3353e2c4feb
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Jan 31 15:52:59 2018

[cq.cfg] Move chromium_presubmit to 100% on LUCI.

R=jbudorick@chromium.org

Bug:  731553 
Change-Id: I064be66068ef0a02e5bab21061112d6174395246
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/893654
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#533281}
[modify] https://crrev.com/ff826a15755f495064d0b2404ffcb3353e2c4feb/infra/config/cq.cfg

8 hours of LUCI in prod, nothing nasty discovered, but pending time is higher than in buildbot (see attached graphs). I'll add 1 more machine to a total of 8 machines.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/48b6bd0aaf5c0dd609372dc838e1286c5d9c0bc2

commit 48b6bd0aaf5c0dd609372dc838e1286c5d9c0bc2
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Feb 01 01:34:13 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c639edbc414b0b276e84ec794d1a78666854b675

commit c639edbc414b0b276e84ec794d1a78666854b675
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Feb 01 01:45:15 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/f6b62f27d48215bc98d77928e90728e755c7abb1

commit f6b62f27d48215bc98d77928e90728e755c7abb1
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Feb 02 14:11:31 2018

Free 7 slaves from chromium_presubmit migrated to LUCI.

R=hinoka

Bug:  731553 
Change-Id: If5721dee9f3f4bfcc630484feb57e0a0373241c4
Reviewed-on: https://chromium-review.googlesource.com/896952
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/f6b62f27d48215bc98d77928e90728e755c7abb1/masters/master.tryserver.chromium.linux/slaves.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/16ab78c0aadc07114925e8d30ddfdbb71cb2a262

commit 16ab78c0aadc07114925e8d30ddfdbb71cb2a262
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Feb 02 18:41:49 2018

All right, the only remaining thing is removing chromium_presubmit builder from buildbot configs.

Do we want to do that now or in bulk later? I'm just thinking about rollback scenarios.

There is no immediate need to this, since bots have already been freed.
Indeed, roll back is easier if we still have config (and that's why I split cleanup into bot reclamation and config removal).

I think there is no harm in keeping buildbot master config around for longer, but IMO we should remove chromium_presubmit from list of trybots in Gerrit UI.

https://chromium-review.googlesource.com/c/chromium/src/+/900497

Status changed to "Not enough data to analyze" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/chromium_presubmit

This is finished now, right? What do we need to do to have the migration app close it? Did it get confused since it's completely done now?

I think Migration app can't distinguish current state from pre-migration state. My understanding is that the app scans what builders are defined in buildbot and if buildbot definition is gone, then the builder is migrated.

Hence, if I land my CL removing builder from master.cfg, the app will close this bug.

nodir@ is this still the case?

Updated decommissioned bots in go/buildbot-decom 

The NextAction date has arrived: 2018-02-19

Still blocked on  issue 800355  

The NextAction date has arrived: 2018-02-23

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/dca71aa8d2a90233c8d9e9fd0f5ee166ea7e9f38

commit dca71aa8d2a90233c8d9e9fd0f5ee166ea7e9f38
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Mar 07 01:54:13 2018

Remove chromium_presubmit builder from buildbot.

Builder is running entirely on LUCI now.

R=hinoka

Bug:  731553 
Change-Id: I18c7f9b1ae4a1021bef3fb082420d55536575470
Reviewed-on: https://chromium-review.googlesource.com/896955
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/dca71aa8d2a90233c8d9e9fd0f5ee166ea7e9f38/masters/master.tryserver.chromium.linux/slaves.cfg
[modify] https://crrev.com/dca71aa8d2a90233c8d9e9fd0f5ee166ea7e9f38/masters/master.tryserver.chromium.linux/master.cfg

I see some chromium_presubmit builds (around 10~15%) have slow bot_update step with log like below.

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit/55413
===Running git fetch origin ===
In directory: /b/swarming/w/ir/cache/builder/chromium_presubmit/src
From /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
   3e9d9d600ae5..30ea8e31a31a  master     -> origin/master
Auto packing the repository for optimum performance.
See "git help gc" for manual housekeeping.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
===Succeeded in 3.1 mins of git fetch origin ===

Why does auto packing run in this step?

Andrii, can you take a quick look? Is this due to the existing git slowdown issues? 

>> Remove chromium_presubmit builder from buildbot.
Shouldn't the migration app have closed this automatically? 

tikuta@ yeah, we (hinoka, jchinlee, myself) noticed this too, but didn't yet have time to do anything. Thanks for reminder! I've filed  issue 822437  to make sure this doesn't get forgotten and because it's not presubmit builder specific.

Status changed to "Migrated"