Do not manually mark this bug as "Fixed" when builder is working as intended (WAI) on LUCI. Migration is not complete until builder is solely on LUCI.

The LUCI migration app will mark the bug status to "Fixed" automatically when the BuildBot builder equivalent is turned down. 

This builder runs on macOS 10.12 in buildbot.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5e6fd861a636da7c5760dfc434e367e70a8da7e1

commit 5e6fd861a636da7c5760dfc434e367e70a8da7e1
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 21 02:17:39 2018

Explicit OS dimension for two dbg mac trybots.

mac_chromium_compile_dbg_ng and mac_chromium_dbg_ng run on Mac 10.12.

R=jchinlee@chromium.org

Bug:  731476 ,  731545 
Change-Id: I573a03bc246ecfce4f6f51d13b4b76af762ec400
Reviewed-on: https://chromium-review.googlesource.com/927826
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/5e6fd861a636da7c5760dfc434e367e70a8da7e1/cr-buildbucket.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/bbb4b0c22e6cfffb02fc022ad6a01b0cf1ce02af

commit bbb4b0c22e6cfffb02fc022ad6a01b0cf1ce02af
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 21 02:37:40 2018

So far -- very good:

Correctness: analyzed 110 build groups, rejected 12 of them: 0 have false failures, 0 have false successes
Speed: analyzed 110 build groups: on average LUCI is 3.96m slower than Buildbot, according to build run durations.


3.96m is fine, because bot_update and compile caches were non-existent, so this is expected. Let's wait and see.

Hm, this difference in compile time persists. I found these two builds (same patch, same base rev) interesting:
LUCI: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/212
BBOT: https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_compile_dbg_ng/610264

Goma stats are mostly the same except total time taken and this:
LUCI: include_processor: total=475257 skipped=352183 total_wait_time=663 total_run_time=13738
BBOT: include_processor: total=96152 skipped=75437 total_wait_time=216 total_run_time=2334

The trace viewer clearly shows that almost all compile tasks on LUCI took 1.5x what they took on buildbot:

LUCI: https://chromium-build-stats.appspot.com/ninja_log/2018/02/22/vm1502-m4/ninja_log.vm1502-m4.chrome-bot.20180222-142519.35308.gz/trace.html
BBOT: https://chromium-build-stats.appspot.com/ninja_log/2018/02/22/vm984-m4/ninja_log.vm984-m4.chrome-bot.20180222-140451.72590.gz/trace.html

Bot specs are the same in: CPU, RAM.
Bots are diff in OS:
 LUCI: 10.12.5  https://chromium-swarm.appspot.com/bot?id=vm1502-m4&show_state=true&sort_stats=total%3Adesc
 BBOT: 10.12.2  https://build.chromium.org/deprecated/tryserver.chromium.mac/buildslaves/vm984-m4

Filed compile-slowdown specific bug separately and asked GoMa team to take a look. issue 815015

Status changed to "LUCI WAI" (correctness 100%, speed 90%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng

Well, I think the diff in compile speed of (LUCI - BBOT) got diluted by builds which skipped compile step altogether. The last couple of build pairs are still different compile-wise.

However, discussing this with goma team in issue 815015, I'm almost convinced this is primarily due to cache hit ratio, which is **currently** lower in LUCI, but will get higher as more mac dbg builds happen on LUCI.

Note, that LUCI and BBOT builds use different checkout path, and hence can't share goma cache, which is code path dependent.

According to http://shortn/_IRfGiccyic on buildbot this builder has:
 * peak trigger is 90 builds in 1 hour window
 * median duration is 16 minutes

According to https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_compile_dbg_ng/ 
 * 115 bots pool is shared by this AND mac_chromium_rel_ng builder

According to http://shortn/_0GO6FEr2Av, mac_chromium_rel_ng builder on buildbot:
 * peak trigger also 90 builds in 1 hour window
 * median duration is 55 minutes during peaks


Therefore, we can roughly divide 115 bots among two builders with ratio of 16 to 55,
which translates to 26 to 89 bots for mac_chromium_compile_dbg_ng to mac_chromium_rel_ng.

Hence, I need 26 mac bots to migrate this builder.

I currently have 8 bots: http://shortn/_TpjHMN7hGl 
I need 18 more bots. Let's ask labs if I can get them.

Filed issue 816721 to see how feasible is to get 18 bots at once soon.

Otherwise, I'll have to bump traffic splitting 0 -> 33 -> 67 -> 100 and migrate ~8 buildbots bots to swarming between bumps.

Status changed to "LUCI not WAI" (correctness 100%, speed 76%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/4f2aecce29b0276d9a930f108ded0031a62ad544

commit 4f2aecce29b0276d9a930f108ded0031a62ad544
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Mar 01 23:15:01 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a8cd25dd4bb7d13bf817e967b3ac4a5a463d8064

commit a8cd25dd4bb7d13bf817e967b3ac4a5a463d8064
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Mar 03 00:29:46 2018

I got remaining macs in issue 817666, so I have all that i need to start migration.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/21f6e6219d5372fac7a1af22d1a5ed48b5d84c71

commit 21f6e6219d5372fac7a1af22d1a5ed48b5d84c71
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Mar 03 00:58:05 2018

Send 25% of CQ tryjobs to LUCI's mac_chromium_compile_dbg_ng.

Also, stop experimenting with mac_chromium_dbg_ng.

R=jchinlee

Bug:  731545 ,  731476 
No-Try: True
Change-Id: I89d3b7eb95f4ec76909101e802f70d2627cf4602
Reviewed-on: https://chromium-review.googlesource.com/947469
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540702}
[modify] https://crrev.com/21f6e6219d5372fac7a1af22d1a5ed48b5d84c71/infra/config/branch/cq.cfg

25% in LUCI - so far so good.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9b9227af05ba7031b24ba61f0d8dbcfe14762400

commit 9b9227af05ba7031b24ba61f0d8dbcfe14762400
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Mar 03 01:21:50 2018

Send 66% of CQ tryjobs to LUCI's mac_chromium_compile_dbg_ng.

R=jchinlee

Bug:  731545 
Change-Id: Ieeb3df10e3649063679cf66a8b9231813714b34b
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/947566
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540705}
[modify] https://crrev.com/9b9227af05ba7031b24ba61f0d8dbcfe14762400/infra/config/branch/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/61cb47822ea823f71c42718913d7720c4f4bdca7

commit 61cb47822ea823f71c42718913d7720c4f4bdca7
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Mar 03 01:46:52 2018

LUCI's mac_chromium_compile_dbg_ng will is now 100% in prod.

R=jchinlee

No-Try: True
Bug:  731545 
Change-Id: I4655d2aaa1f78ff350336743b19627bd4cfc1e2d
Reviewed-on: https://chromium-review.googlesource.com/947572
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540712}
[modify] https://crrev.com/61cb47822ea823f71c42718913d7720c4f4bdca7/infra/config/branch/cq.cfg

All good, 100% in prod.

Turns out skia CQ is using mac_chromium_compile_dbg_ng: https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_compile_dbg_ng/616284

TODO: remove 26 bots from the pool this builder is on.

Status changed to "Low confidence" (correctness 100%, speed 81%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/b2254fbaa9348bda9b716a88997cdd8bcb57aec8

commit b2254fbaa9348bda9b716a88997cdd8bcb57aec8
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Mar 10 07:09:04 2018

tryserver.chromium.mac: remove hardware of migrated dbg builders.

This removes 26 + 2 machines for mac_chromium_compile_dbg_ng (on CQ)
and mac_chromium_dbg_ng (optional) builders, respectively.

R=hinoka@chromium.org

Bug:  731476 ,  731545 
Change-Id: I5012faeee4c6326aea96e8a3fd76f3eb973781c7
Reviewed-on: https://chromium-review.googlesource.com/958086
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/b2254fbaa9348bda9b716a88997cdd8bcb57aec8/masters/master.tryserver.chromium.mac/slaves.cfg

Status changed to "No data" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng

Blocking on issue 824467, because skia CQ is still referencing old builder.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/4e283b027e3df79f2816ae5fbd5c30d8b1a0e3ce

commit 4e283b027e3df79f2816ae5fbd5c30d8b1a0e3ce
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Mar 21 22:47:07 2018

tryserver.chromium.mac: remove _dbg_ mac builders.

* mac_chromium_dbg_ng
* mac_chromium_compile_dbg_ng

R=hinoka@chromium.org

Bug:  731476 ,  731545 
Change-Id: I042f56d56a37802e8e862cd9e97177436f6102bd
Reviewed-on: https://chromium-review.googlesource.com/958090
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/4e283b027e3df79f2816ae5fbd5c30d8b1a0e3ce/masters/master.tryserver.chromium.mac/master.cfg
[modify] https://crrev.com/4e283b027e3df79f2816ae5fbd5c30d8b1a0e3ce/masters/master.tryserver.chromium.mac/slaves.cfg
[modify] https://crrev.com/4e283b027e3df79f2816ae5fbd5c30d8b1a0e3ce/tests/masters_recipes_test.py

And now totally removed.

Status changed to "Migrated"