Do we need to build and run tests, or is it enough to just do a compile?

Is this more or less important than:
1) Running tests on both iOS9 and iOS10.
2) Running tests on both 32bit and 64bit.
3) Running tests on both device and simulator.

I think building is probably enough as all the error we've seen for the moment are compilation error (unsupported flags or incorrect warnings). A codegen bug is possible, but it is more likely to be present in Chromium version of clang than Xcode version of clang (more project use Xcode version of clang to ship app on iOS so I would consider it more tested).

Can anyone on CC list take this bug?

Eugene: I was waiting for confirmation that Xcode clang must be a separate bot and that we can't convert the existing bots which use Chromium clang. I have a rationale for that so I can start on this.

They need to be separate bots. The goal is two have two separate set of bots, one set building with Chromium clang while the other set build with Xcode clang. We need both set as the two compiler support different set of parameters, debug diagnostics and bugs and we want to have both building without error so that we can switch from one to the other quickly if needed.

Currently we use Chromium clang for development and Xcode clang for official builders and only see issues with Chromium clang when building Official, thus it immediately escalate to P0. But we want to keep Chromium clang for development as it has better diagnostics and supports Chromium plugins.

In the future, if/when we switch to Chromium clang for Official builds, we want to still build with Xcode clang to have the possibility to switch back to it quickly in case a critical feature is released in a new iOS SDK that is only available through the use of Xcode clang.

Honestly, tests are essentially free right now because the Swarming pool upstream has so many more Macs suitable for Chrome for iOS than we actually need. I'll set it up with tests, but please feel free to adjust or remove them.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/71d162c4b325fd8729aeb6f7aff859841e658295

commit 71d162c4b325fd8729aeb6f7aff859841e658295
Author: smut <smut@google.com>
Date: Sat Nov 05 02:14:14 2016

Replace ios-simulator-swarming with ios-simulator-xcode-clang

ios-simulator-swarming is a deprecated experimental try bot (all iOS bots use Swarming now).

BUG= 661612 

Review-Url: https://codereview.chromium.org/2476023002

[modify] https://crrev.com/71d162c4b325fd8729aeb6f7aff859841e658295/masters/master.tryserver.chromium.mac/master.cfg
[modify] https://crrev.com/71d162c4b325fd8729aeb6f7aff859841e658295/masters/master.tryserver.chromium.mac/slaves.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/927bb1bd67f74429dcc04336dbb246f61448e3eb

commit 927bb1bd67f74429dcc04336dbb246f61448e3eb
Author: smut <smut@google.com>
Date: Sat Nov 05 02:20:15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/778993dda4a5c8c4a15df8d8048165aa28b80ada

commit 778993dda4a5c8c4a15df8d8048165aa28b80ada
Author: smut <smut@google.com>
Date: Sat Nov 05 03:24:42 2016

Replace ios-simulator-swarming bot with ios-simulator-xcode-clang

ios-simulator-swarming is a deprecated CQ experiment. All iOS bots use Swarming now.

BUG= 661612 

Review-Url: https://codereview.chromium.org/2482503003
Cr-Commit-Position: refs/heads/master@{#430140}

[modify] https://crrev.com/778993dda4a5c8c4a15df8d8048165aa28b80ada/infra/config/cq.cfg
[rename] https://crrev.com/778993dda4a5c8c4a15df8d8048165aa28b80ada/ios/build/bots/chromium.mac/ios-simulator-xcode-clang.json
[modify] https://crrev.com/778993dda4a5c8c4a15df8d8048165aa28b80ada/tools/mb/mb_config.pyl

Re comment #7, does that mean that it's also basically free to run 32-bit tests and iOS9 tests on a non-main waterfall, if we just set up a few machines for each that send jobs to swarming?

Are the requirements much higher if we wanted to put 32-bit or iOS9 tests onto the CQ?

It's free for ios-simulator-xcode-clang because it's a 50% CQ experiment so it doesn't even double our usage. Sergey's current capacity estimates suggest to me that we can afford to double our utilization without adding any additional capacity.

I could turn off tests on ios-simulator-xcode-clang and instead turn on your choice of 64-bit iOS 9, 32-bit iOS 10, 32-bit iOS 9, @3x iOS 10, or @3x iOS 9 tests on CQ instead. In the long term I would like to be able to provide capacity to test on all of these platforms, because it's what we do on downstream CQ.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/45a56f69537b8fb0bd7f1767c3aafb75929b3b81

commit 45a56f69537b8fb0bd7f1767c3aafb75929b3b81
Author: smut <smut@google.com>
Date: Tue Nov 15 23:46:54 2016

Add ios-device-xcode-clang, reshuffle slaves for iOS try bots

Current capacity estimates put us at 20 needed for compilation-only builders.

BUG= 661612 

Change-Id: I7ace6da9ee1e8334a60fb891cdcf11283b4c7eeb
Reviewed-on: https://chromium-review.googlesource.com/411913
Commit-Queue: smut <smut@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/45a56f69537b8fb0bd7f1767c3aafb75929b3b81/masters/master.tryserver.chromium.mac/master.cfg
[modify] https://crrev.com/45a56f69537b8fb0bd7f1767c3aafb75929b3b81/masters/master.tryserver.chromium.mac/slaves.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/60835da990f9379e9208b257f30f0b56a2d722fc

commit 60835da990f9379e9208b257f30f0b56a2d722fc
Author: smut <smut@google.com>
Date: Wed Nov 16 06:50:15 2016

Add config for ios-device-xcode-clang bot

And remove additional_compile_targets to match ios-simulator and ios-simulator-xcode-clang.

BUG= 661612 

Review-Url: https://codereview.chromium.org/2509503002
Cr-Commit-Position: refs/heads/master@{#432413}

[add] https://crrev.com/60835da990f9379e9208b257f30f0b56a2d722fc/ios/build/bots/chromium.mac/ios-device-xcode-clang.json
[modify] https://crrev.com/60835da990f9379e9208b257f30f0b56a2d722fc/ios/build/bots/chromium.mac/ios-device.json
[modify] https://crrev.com/60835da990f9379e9208b257f30f0b56a2d722fc/ios/build/bots/chromium.mac/ios-simulator-cronet.json
[modify] https://crrev.com/60835da990f9379e9208b257f30f0b56a2d722fc/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/859306826d74201482d4414ff38b4278bc72c883

commit 859306826d74201482d4414ff38b4278bc72c883
Author: smut <smut@google.com>
Date: Wed Nov 16 22:17:42 2016

Sorry for the delay, it's taken awhile to get accurate capacity estimations even just to set up CQ experiments. ios-simulator-xcode-clang and ios-device-xcode-clang are now available on tryserver.chromium.mac:
https://uberchromegw.corp.google.com/i/tryserver.chromium.mac/builders/ios-simulator-xcode-clang
https://uberchromegw.corp.google.com/i/tryserver.chromium.mac/builders/ios-device-xcode-clang

We're still awaiting more capacity to make them part of the CQ.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/74e4cbd04cae50b3ee58567152f4340730359b5c

commit 74e4cbd04cae50b3ee58567152f4340730359b5c
Author: smut <smut@google.com>
Date: Fri Nov 18 01:40:31 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/a9cf4b3bd5d408197c6769b4c786ca1f0d343c37

commit a9cf4b3bd5d408197c6769b4c786ca1f0d343c37
Author: smut <smut@google.com>
Date: Thu Nov 17 23:59:13 2016

Add ios-device-xcode-clang, ios-simulator-xcode-clang to chromium.mac

BUG= 661612 

Change-Id: I5833367b163f86f12bc30aee29bfec379d2d5d3a
Reviewed-on: https://chromium-review.googlesource.com/412288
Commit-Queue: smut <smut@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/a9cf4b3bd5d408197c6769b4c786ca1f0d343c37/masters/master.chromium.mac/master_ios_cfg.py
[modify] https://crrev.com/a9cf4b3bd5d408197c6769b4c786ca1f0d343c37/masters/master.chromium.mac/slaves.cfg
[modify] https://crrev.com/a9cf4b3bd5d408197c6769b4c786ca1f0d343c37/scripts/slave/gatekeeper.json
[modify] https://crrev.com/a9cf4b3bd5d408197c6769b4c786ca1f0d343c37/tests/masters_recipes_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8e88707603804ae8d9bf60fec2b2d7e82f1ed6f

commit c8e88707603804ae8d9bf60fec2b2d7e82f1ed6f
Author: smut <smut@google.com>
Date: Fri Nov 18 03:28:10 2016

Disable tests on ios-simulator-xcode-clang

Tests are mostly redundant with ios-simulator, and are eating Swarming capacity we could use to put additional tests on ios-simulator. We're mostly interested in the compilation differences for this builder.

BUG= 661612 

Review-Url: https://codereview.chromium.org/2512843003
Cr-Commit-Position: refs/heads/master@{#433062}

[modify] https://crrev.com/c8e88707603804ae8d9bf60fec2b2d7e82f1ed6f/ios/build/bots/chromium.mac/ios-simulator-xcode-clang.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1117c2c39b529a46f4b0c7ea9bcd4d790ac2f7cd

commit 1117c2c39b529a46f4b0c7ea9bcd4d790ac2f7cd
Author: smut <smut@google.com>
Date: Fri Nov 18 06:51:36 2016

Add ios-device-xcode-clang as a 50% CQ experiment

BUG= 661612 
R=dpranke@chromium.org

Review URL: https://codereview.chromium.org/2509363004 .

Cr-Commit-Position: refs/heads/master@{#433138}

[modify] https://crrev.com/1117c2c39b529a46f4b0c7ea9bcd4d790ac2f7cd/infra/config/cq.cfg

\o/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/10cd9c8f136d972ff0c679fd83d49d4f94d4d5c4

commit 10cd9c8f136d972ff0c679fd83d49d4f94d4d5c4
Author: smut <smut@google.com>
Date: Tue Dec 06 01:20:39 2016

Reallocated iOS slaves on tryserver.chromium.mac

ios-simulator compiles and tests, which has never required more than 15 in the past 30 days. Allocating 30.

ios-device, ios-device-xcode-clang, and ios-simulator-xcode-clang compile only, which has never required more than 10 in the past 30 days. Allocating 16 each.

BUG= 661612 

Change-Id: I54ea031379d19ff2042f1d35c7c8f526b169a967
Reviewed-on: https://chromium-review.googlesource.com/416214
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Commit-Queue: smut <smut@google.com>

[modify] https://crrev.com/10cd9c8f136d972ff0c679fd83d49d4f94d4d5c4/masters/master.tryserver.chromium.mac/slaves.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/156367b700b4579a0efd7e860b1b38bbdba1b457

commit 156367b700b4579a0efd7e860b1b38bbdba1b457
Author: smut <smut@google.com>
Date: Wed Dec 07 22:28:14 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dbc9ca652bbe74844e95837854238720df43fc14

commit dbc9ca652bbe74844e95837854238720df43fc14
Author: smut <smut@google.com>
Date: Sat Dec 10 02:45:50 2016

Promote ios-device-xcode-clang, ios-simulator-xcode-clang to the commit queue

BUG= 661612 

Review-Url: https://codereview.chromium.org/2550063003
Cr-Commit-Position: refs/heads/master@{#437733}

[modify] https://crrev.com/dbc9ca652bbe74844e95837854238720df43fc14/infra/config/cq.cfg