Assigning to tandrii since he's been working on linux.chromium sandbox test migrations for this. Marking as blocker

ok, fine, i'll own this. I guess it should migrate together with CI's Linux Builder/Tester pair.

Changing which builders we migrate first to dbg builders.

Status changed to "LUCI not WAI" (correctness 78%, speed 148%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 110%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI not WAI" (correctness 98%, speed 111%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 106%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI not WAI" (correctness 97%, speed 112%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 104%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI not WAI" (correctness 99%, speed 114%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 106%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI not WAI" (correctness 99%, speed 139%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 139%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI not WAI" (correctness 99%, speed 117%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 115%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/7bc334f78e27e2f42c68c8a991afca93e517ebab

commit 7bc334f78e27e2f42c68c8a991afca93e517ebab
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jan 05 19:21:39 2018

Status changed to "LUCI not WAI" (correctness 97%, speed 122%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "Not enough data to analyze" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI WAI" (correctness 100%, speed 116%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "LUCI not WAI" (correctness 99%, speed 112%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/27060 uploaded data to the production test results server. I'm not sure why it did that; I didn't really set it up. Does anyone know why it would have done that?

I set the experiment percentage on that bot to 0, so it shouldn't schedule new builds. There are a bunch pending on it now though.

The tryjobs are getting "$recipe_engine/runtime	{"is_experimental": false, "is_luci": true}" set on them. I'm not sure where that comes from...

As I'm reading the migration doc, it looks trybots maybe don't get experimental? 

I am seeing luci.chromium.try runs on the test results dashboard, which seems confusing; if I'm a random developer looking at the dashboard, it looks a lot more red than it actually is. So this looks like a bug to me, but maybe I don't know something.

1. Correct, tryjobs don't get experimental set. You may set experimental manually, if you so want, in cr-buildbucket.cfg.

What's test results dashboard, and why are you looking at it for tryjobs?

tryjobs scheduled by migration app were non-experimental since migration app was created. So far, nobody complained, so let's confirm the problem first. In the mean time, I've set the experimental percentage back to 10.

Note that we've exposed "LUCI is Prod" checkbox for try builders very recently because of  bug 800355 . This might cause some confusion because it is not checked, but builds are not marked experimental. Note sure what to do with this besides setting "experimental" to true for builds scheduled by the migration app...

The only reason I'm concerned about this is that right now some tests are failing consistently on the LUCI version of bots, which is polluting the test results server. It is true that no one has complained though, so maybe it's fine to leave as is.

Also, I found that the "check_network_annotations" step is failing intermittently. It does a subprocess.call(['ninja', ..]) invocation, which fails when depot_tools isn't in path. Is there a standard way to fix this somehow? Relevant code is here: https://cs.chromium.org/chromium/src/tools/clang/pylib/clang/compile_db.py?q=clang/compile_db&sq=package:chromium&l=87

I sent a CL https://chromium-review.googlesource.com/#/c/chromium/src/+/889994 to fix this, to use the third party ninja copy checked in.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/50f07df061226f9b7a22d724ed600ede5082200d

commit 50f07df061226f9b7a22d724ed600ede5082200d
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Jan 30 00:25:57 2018

Use third party ninja in compile_db.py

'ninja' isn't always in PATH on luci builders, so use the copy checked
into third_party/depot_tools

Bug:  731558 
Change-Id: I3f57375dc768d5266e7b80a0324abb3acf48fd49
Reviewed-on: https://chromium-review.googlesource.com/889994
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Daniel Cheng <dcheng@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532692}
[modify] https://crrev.com/50f07df061226f9b7a22d724ed600ede5082200d/tools/clang/pylib/clang/compile_db.py

I'd like to revisit the experimental trybots bit -- why shouldn't trybots that don't contribute to whether the CL passes the CQ be experimental?

(context is https://bugs.chromium.org/p/chromium/issues/detail?id=806983, where LUCI win_chromium_dbg_ng consumed a sufficient amount of our windows swarming capacity to trigger pending time alerts)

there are at least 3 meanings of term "experimental":
1) "CQ experimental": if true, build result does not affect CQ decision
2) "experimental build": if true, build does not make prod side effects, e.g. does not upload test results
3) "experimental percentage": percentage of Buildbot builds that the migration app tries on LUCI

So far, the discussion in this bug was about (1). Comment 82 talks about (2), but (2) seems irrelevant, all builds scheduled by migration app are already marked "CQ experimental" anyway. The cause of long pending time is high value of (3)

I'm working on this. Still some issues with network annotations test failing: https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8955903927256856192%2F%2B%2Fsteps%2Fcheck_network_annotations__with_patch_%2F0%2Fstdout

https://cs.chromium.org/chromium/src/tools/traffic_annotation/auditor/traffic_annotation_auditor.cc?q=traffic_annotation_auditor&sq=package:chromium&dr=C&l=510 is triggering in builds like hhttps://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/27602

I think that https://cs.chromium.org/chromium/src/tools/traffic_annotation/auditor/traffic_annotation_auditor.cc?q=traffic_annotation_auditor&sq=package:chromium&dr=C&l=494 needs to be qualified somehow to use the third_party/depot_tools gn. I don't know C++ to do that as easily, but I'll try to figure it out.

Status changed to "LUCI WAI" (correctness 100%, speed 121%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

I think the app is wrong; I have one more CL that needs to land to fix this. Errors like https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8955903927256856192%2F%2B%2Fsteps%2Fcheck_network_annotations__with_patch_%2F0%2Fstdout still happen flakily I think.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c6faed7f02f46b64bcee78e97e74518a49261308

commit c6faed7f02f46b64bcee78e97e74518a49261308
Author: Stephen Martinis <martiniss@google.com>
Date: Fri Feb 02 00:06:08 2018

I bumped the experiment percentage; I added more capacity in #93 so it should be able to handle the increased load.

replied in https://crbug.com/806948#c4

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/33a46dfe7e9169a52680b1029c232ff5a6a1be4b

commit 33a46dfe7e9169a52680b1029c232ff5a6a1be4b
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Feb 02 19:44:47 2018

traffic annotation: Use checked in version of gn

Bug:  731558 
Change-Id: I4d3eaf86139e140df0c465edfdc84756a490fa38
Reviewed-on: https://chromium-review.googlesource.com/896954
Reviewed-by: Ramin Halavati <rhalavati@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534127}
[modify] https://crrev.com/33a46dfe7e9169a52680b1029c232ff5a6a1be4b/tools/traffic_annotation/auditor/traffic_annotation_auditor.cc

Status changed to "LUCI not WAI" (correctness 99%, speed 124%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

The failure that caused #99 is flaky. It flaked on buildbot and not LUCI. I'll try to look into it.

Status changed to "LUCI WAI" (correctness 100%, speed 106%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

I wasn't able to find anything useful about #99, but it's been WAI for several days, so I think it's safe.

I'd like to switch LUCI over to prod tomorrow, Wednesday February 21st. I'm planning on doing this in the afternoon mountain view time.

jbudorick@ has signed off that this seems like an ok plan.

please, don't flip from 0 to >= 50% traffic splitting at once, because of https://crbug.com/806986#c7. 
If you flip gradually 25%, 50%, 75%, 100% with 1h intervals, that'd be probably be fine. Alternatively, do 20%, 40% at day time, and then if all is well, switch to 100% after 5pm MTV time.

I'll flip gradually and monitor load.

Status changed to "LUCI not WAI" (correctness 99%, speed 114%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

As a note, #105 is a real failure. It looks like a pretty random flake in webkit_layout_tests. AFAIK there's nothing different between the buildbot CQ machines and swarming machines, and the luci ones. Not sure why it failed :/ I'm going to proceed anyways.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/184689f94fc719c4bfc094a92ace0c881c732271

commit 184689f94fc719c4bfc094a92ace0c881c732271
Author: Stephen Martinis <martiniss@google.com>
Date: Wed Feb 21 22:41:26 2018

I'm going to start migrating now. 

Actually, I'm changing strategies. I'm going to frontload a lot of capacity on the bot right now from the spreadsheet with bots allocated for the transition.

I'll then ramp up equivalence, making sure nothing is broken.

After the builder is 100% on LUCI, I'll make a CL to remove the capacity on buildbot, and tell labs to reimage those and dump them into the spreadsheet to replace the bots I took from the spreadsheet.

Actually, how many bots do you need for this?

It looks like the buildbot slaves are shared by many builders on buildbot:
https://build.chromium.org/deprecated/tryserver.chromium.linux/buildslaves

I don't think you'll be able to reclaim them without migrating the other builders unless I'm missing something.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b8ccefe0b6a44b8e26ec4f0775a57d8df884dabc

commit b8ccefe0b6a44b8e26ec4f0775a57d8df884dabc
Author: Stephen Martinis <martiniss@chromium.org>
Date: Wed Feb 21 22:59:45 2018

Bump linux_chromium_rel_ng to 25% LUCI equivalence

Bug:  731558 
Change-Id: I6acb2b31ff95fa20e632191e7bd067579911f6e4
Reviewed-on: https://chromium-review.googlesource.com/927683
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538242}
[modify] https://crrev.com/b8ccefe0b6a44b8e26ec4f0775a57d8df884dabc/infra/config/cq.cfg

Re #110: I used a graph of number of running builds at peak to guess that 68 bots should be sufficient for the builder to run jobs at peak. So, I'll reclaim that many bots from the cq linux machine pool on tryserver.chromium.linux.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a409d8ab166b7fdb4a9ec8e9af6888dcff887331

commit a409d8ab166b7fdb4a9ec8e9af6888dcff887331
Author: Stephen Martinis <martiniss@google.com>
Date: Wed Feb 21 23:03:19 2018

We can estimate how many bots by looking at

machine_time(THIS_BUILDER) / sum(map(machine_time, all_builders_using_these_slaves)

def machine_time(builder):
  return builder.average_cycle_time * builder.daily_builds  / (sum( 
    (avg cycle time * daily builds))

Builds are running on the LUCI builder. They seem to be taking a bit, probably because they're fresh bots that need to download the chromium checkout and gclient runhooks.

A few builds have completed which analyze says needed no compile.

I'm going to wait 15 more minutes, and then bump equivalence again.

I waited a bit longer! 

Going to bump up to 50%. I believe we'll be able to handle the extra load, but we'll be at pretty much full capacity.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f1a74152563660e9c7088abf8c09ba43bddcecf0

commit f1a74152563660e9c7088abf8c09ba43bddcecf0
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Feb 22 00:12:20 2018

Bump linux_chromium_rel_ng to 50% LUCI equivalence

Bug:  731558 
Change-Id: Ibf3ce1e23b699a0bf7c87ce3bb818f5185d444b2
Reviewed-on: https://chromium-review.googlesource.com/929319
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538270}
[modify] https://crrev.com/f1a74152563660e9c7088abf8c09ba43bddcecf0/infra/config/cq.cfg

Everything looks ok. Bumping to 75%.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8671315fbb18b83b6bb70cd2a01843e87cb8079

commit c8671315fbb18b83b6bb70cd2a01843e87cb8079
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Feb 22 01:17:29 2018

Bump linux_chromium_rel_ng to 75% LUCI equivalence

Bug:  731558 
Change-Id: I137731ae2ab7633d00863f9919dfa92f93168323
Reviewed-on: https://chromium-review.googlesource.com/930028
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538293}
[modify] https://crrev.com/c8671315fbb18b83b6bb70cd2a01843e87cb8079/infra/config/cq.cfg

There's a potential issue. https://chromium-review.googlesource.com/c/chromium/src/+/930101/1 appears to be failing only on linux_chromium_rel_ng, which is worrisome. CQ retried the build, so we'll see what happens on the retry.

Actually, the retry worked fine.

jbudorick@ pointed out that there's probably a data race in here somewhere

gn path out-mb/linux_chromium_rel_ng/ //third_party/WebKit/Source/bindings/core/v8:bindings_core_impl //third_party/WebKit/Source/core/html No non-data paths found between these two targets.

I'm going to land the 100% equivalence change.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/090fde99b2378a1c6b25d37c2b4a7e8feb1f00d1

commit 090fde99b2378a1c6b25d37c2b4a7e8feb1f00d1
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Feb 22 01:52:04 2018

Bump linux_chromium_rel_ng to full LUCI equivalence

Bug:  731558 
Change-Id: I8c4d75e00b084adf71ae7dffce29ee3e7ac4b14f
Reviewed-on: https://chromium-review.googlesource.com/929581
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538308}
[modify] https://crrev.com/090fde99b2378a1c6b25d37c2b4a7e8feb1f00d1/infra/config/cq.cfg

Status changed to "LUCI WAI" (correctness 100%, speed 118%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Status changed to "Not enough data to analyze" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

I'm going to decomission slave{111-131, 162-209}-c4. This is 68 machines, which is the number of GCE machines we have assigned to the LUCI builder.

More specifically, list is [111-131), [162-210), where "[" is inclusive and ")" is exclusive.

Status changed to "Low confidence" (correctness 100%, speed 115%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

This has been migrated.

NVM keep it open.

Why ninja -j is different on LUCI and buildbot?
https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/652174 buildbot uses -j150
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/41844 LUCI uses -j50

I specified to use -j150 on linux_chromium_rel_ng buildbot, but why it is not applied to LUCI bot?
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_linux.py?l=26&rcl=0c6a4853998c9890a1c28f876fb3338c63992699

Ah it needs explicit specification. Made CL.
https://chromium-review.googlesource.com/c/chromium/src/+/954766

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/6aa80a1b0790e502ec84af94c1d5393575257885

commit 6aa80a1b0790e502ec84af94c1d5393575257885
Author: Ryan Tseng <hinoka@google.com>
Date: Thu Mar 08 20:27:01 2018

Remove linux x64 builders from buildbot try and CI

These have been migrated to LUCI.

This also decommissions the following bots:
* build36-m1
* vm906-m1
* build38-m1
* vm916-m1
* slave[909-914]-c4
* slave[1010-1021]-c4

Bug:  731434 ,  731552 ,  731558 ,  731429 ,  731558 
Change-Id: I0d9beb1e2be64dc349da449d0bbbf4c3ef7ea3ca
Reviewed-on: https://chromium-review.googlesource.com/943829
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/6aa80a1b0790e502ec84af94c1d5393575257885/masters/master.chromium.linux/master_linux_cfg.py
[modify] https://crrev.com/6aa80a1b0790e502ec84af94c1d5393575257885/masters/master.tryserver.chromium.linux/master.cfg
[modify] https://crrev.com/6aa80a1b0790e502ec84af94c1d5393575257885/tests/masters_recipes_test.py
[modify] https://crrev.com/6aa80a1b0790e502ec84af94c1d5393575257885/masters/master.tryserver.chromium.linux/slaves.cfg
[modify] https://crrev.com/6aa80a1b0790e502ec84af94c1d5393575257885/masters/master.chromium.linux/slaves.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2d86a05a6f2e768bd82e362def9ec23496d02246

commit 2d86a05a6f2e768bd82e362def9ec23496d02246
Author: Takuto Ikuta <tikuta@google.com>
Date: Fri Mar 09 02:43:04 2018

Increase build parallelism for LUCI linux_chromium_rel_ng trybot

I increased ninja -j for buildbot from 50 to 150 in crrev.com/877743
But this increase is not applied to LUCI bot.
Let me enable high build parallelism for LUCI also.

Bug:  731558 
Change-Id: Ibd9ece5954142a67a1632b3d4a4a391e4d6f7805
Reviewed-on: https://chromium-review.googlesource.com/954766
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542003}
[modify] https://crrev.com/2d86a05a6f2e768bd82e362def9ec23496d02246/infra/config/global/cr-buildbucket.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/8bf9106d3950bf78bbcad86c2dd23c3dddeab596

commit 8bf9106d3950bf78bbcad86c2dd23c3dddeab596
Author: Ryan Tseng <hinoka@google.com>
Date: Tue Mar 13 00:24:32 2018

Status changed to "No data" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.linux/builders/linux_chromium_rel_ng

Stephen, is there still something to be done here?

I have no idea. I tried closing this manually and people said I shouldn't, so I left it open, saying the migration app would close it. But now it isn't? 

Stephen: thanks. We'll be updating instructions soon to close bugs, but we have to first fix one thing in migration app.

I've sent a PSA about this. Let's close bugs

Thanks Stephen for fixing this!

Status changed to "Migrated"