I don't particularly like the idea of requiring 10.12 to compile, but that's better discussed in  bug 624049 , so I'll take it as a given for purposes of this bug (and it's not my call anyway, obviously).

Assuming this bug wasn't an issue, when would you otherwise be ready for this switch?

That said, assuming we can only build on 10.12, we have a few options:

1) Run the webkit_tests working under swarming ( bug 524758 ). I do not know of an ETA for this (tansell@ ?).

2) Change the recipes so that they can archive and upload the builds and then trigger separate test-only builds on the platform. We don't usually do this, but I believe kbr@ made this work at some point for GPU tests. I don't know how much work this is but I don't think it's a huge amount (we should ask kbr@). This would probably require us to rework the rebaseline-cl logic a bit, but possibly not too much.

3) Only run tryjobs on 10.12, and require us to keep rebaseline-o-matic around and keep the "rebaseline after commit" process. We've been trying hard lately to get rid of this flow and the need for rebaseline-o-matic at all, so this would be a big step backwards for us and I would really not like this option.

4) Only run webkit_tests on 10.12, and stop trying to get coverage of the other platforms. I don't know how much benefit having mac-platform-specific layout test results bring these days. My general impression is that the value is much lower than it was when we still supported 10.6.

5) Drop Mac support for anything less than 10.12 :). I'm assuming this is a non-starter.

If we think the coverage is important enough, then (1) is the ideal answer, and (2) is the next-best choice, I think. Obviously if we think the tests aren't providing enough value, though, we should just stop running them.

I don't own or regularly work on Blink infra any more, so I'm de-assigning myself and copying other people that do ...

(2)'s supported by the chromium recipe. (All the GPU-specific recipes were deleted earlier this year.) If the machines are configured as a builder/tester pair and Swarming's not in use, then the build will be archived by the builder and downloaded and extracted on the tester. See the machines like "Linux Release (New Intel)" in tools/build/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py which are run this way.

Thanks, Ken! That makes things sound easy enough, then.

Thanks Ken. I'll take it from here.

After investigation, I think (1) is the best way forward.

The problem with (2) is that it doesn't support the following use case without significant changes:
1) Compile with patch on 10.12 machine.
2) Run tests on 10.9 machine.
3) If there are failed tests...
  3a) Recompile without patch on 10.12 machine
  3b) Run tests without patch on 10.9 machine.
4) Aggregate information from (1-2) and (3a-3b) to determine final results.

This use case is perfectly supported by swarming, and is how swarming + trybots works right now. Let me figure out what the status is of swarming + webkit_tests.

Good point Erik. Correct, I'd forgotten about retries without the patch.

Apart from the rebaselining case, it's usually pretty rare that a test passes on the latest o/s but fails on an older version. Arguably we don't even really care, since our policy is to only test one version of a platform in the CQ.

So, since the main reason for supporting the cross-version case is for generating baselines, perhaps we don't need to care about retrying without the patch?

(I agree that getting swarming to work would be ideal, I'm just looking for a fallback plan).

Chatted with tansell@. They're trying to finish by end of quarter, but there have been some unforeseen difficulties. Exact timeline is unclear, earliest estimate is end of November. I'll look into the details of a short-term solution, along the lines of what dpranke@ proposed in c#9.

Actually, how much do you care about performance / run time? Is this just for rebaselines?

I'm just about to send out a CL which allows run-webkit-tests to be isolated. Once that is done - you can run them on swarming, but is kind of slow to upload the files into isolate and download them on the other side (IE it'll take longer than running the tests themselves).

If you are okay with the running taking 2-3 hours, that might be a short term solution?

Looking at the last week, load on these trybots appears to be pretty low, 

https://build.chromium.org/p/tryserver.blink/builders/mac10.9_blink_rel?numbuilds=100
https://build.chromium.org/p/tryserver.blink/builders/mac10.9_blink_dbg
https://build.chromium.org/p/tryserver.blink/builders/mac10.10_blink_rel
https://build.chromium.org/p/tryserver.blink/builders/mac10.11_blink_rel

Temporarily increasing the run time seems fine to me.

Attached file is notes to myself.

There are 8 blink-related steps: 
A) webkit_lint, webkit_python_tests
B) blink_heap_unittests, blink_platform_unittests, webkit_unit_tests, wtf_unittests
C) webkit_tests, archive_webkit_tests_results

The first two should not depend on OS version, and are not swarming compatible. The next four are already swarming-compatible. The last two are not yet swarming compatible. 

The main waterfall is in an inconsistent state and will also have to be fixed.
[WebKit Mac10.9]: Compiles locally, dispatches (B) tests via swarming.
[WebKit Mac10.11 (retina)]: Compiles locally, runs all tests locally.
[WebKit Mac10.10] and [WebKit Mac10.11]: Use binary from [WebKit Mac Builder], run all tests locally.

We should fix this so that all builders use the binary from [WebKit Mac Builder], and then run tests locally. [I don't care whether (B) tests get dispatched to swarming or not].

Aside: Tests (A) and (B) come from testing/buildbot/chromium.webkit.json. Tests (C) come from scripts/slave/recipe_modules/chromium_tests/chromium_webkit.py.

The tryserver also runs all 8 tests, but it picks them up from _trybot_steps_internal. We can make these tests swarming by making sure they get instantiated with the right parameters, and have the appropriate dimensions set. 

I'm not sure why the tests are specified in three different locations...

Deleted: blink tryserver notes.txt 10.3 KB

blink tryserver notes.txt 10.3 KB View Download

The fact that Mac10.9 compiles locally is left over from some bug we were troubleshooting at some point. There's no good reason for it now.

We have tests specified in three different locations for annoying reasons :).

1) All other things being equal, everything should be listed in the JSON scripts.

However,

2) You can't currently list webkit_tests in the JSON scripts, because we don't have a "generator" for that kind of test in the recipes. We should just fix this (it's not more than a few lines of code).

The tryservers are special because we want to run webkit_tests in the CQ only for changes to a subset of directories, and not for any change that affects content_shell. This is because the tests aren't swarmed and hence running them locally would slow down the CQ too much. Once we have swarming support, this reason (hopefully) goes away.

In addition, based on the CL I looked at from tansell@, he's planning to switch away from the BlinkTest step class to the IsolatedScriptTest class when things are swarmed, in which case you wouldn't need to write the new generator either.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/5ffef937190e0b93938defbde7c4d6325fa8c10a

commit 5ffef937190e0b93938defbde7c4d6325fa8c10a
Author: erikchen <erikchen@chromium.org>
Date: Fri Oct 28 20:34:39 2016

Main waterfall blink testers should not compile locally.

There are already dedicated builders for compilation.

BUG= 659270 

Review-Url: https://codereview.chromium.org/2457173002

[modify] https://crrev.com/5ffef937190e0b93938defbde7c4d6325fa8c10a/scripts/slave/recipe_modules/chromium_tests/chromium_webkit.py
[modify] https://crrev.com/5ffef937190e0b93938defbde7c4d6325fa8c10a/scripts/slave/recipes/chromium.expected/full_chromium_webkit_WebKit_Mac10_11__retina_.json
[modify] https://crrev.com/5ffef937190e0b93938defbde7c4d6325fa8c10a/scripts/slave/recipes/chromium.expected/full_chromium_webkit_WebKit_Mac10_9.json
[modify] https://crrev.com/5ffef937190e0b93938defbde7c4d6325fa8c10a/scripts/slave/recipes/chromium.expected/full_chromium_webkit_WebKit_Mac_Builder.json

Just chatted with tansell@. Theoretically, once 524758 is finished, we should just be able to deploy 10.12 onto the relevant bots and everything will just work. 

LayoutTests is now running regularly on swarming but just not on the Mac builder. There are currently ~121 unexpected failures when using swarming for LayoutTests on Mac.

Hey tansell, you mentioned that we now run Blink Layout Tests on 10.12 on swarming for macOS: https://bugs.chromium.org/p/chromium/issues/detail?id=730936

However, the blocker for this bug is still open. What's the status?

We have yet to deploy Layout Tests on swarming for Mac OS on the try servers, but we have successfully got it running on the Random Order FYI bots (see https://bugs.chromium.org/p/chromium/issues/detail?id=717344). We hope to look at deploying this in the next couple of weeks.

BTW The Layout Tests have been running using 10.12 locally for a while now (at least on mac_chromium_rel_ng). I'm unclear if we still run them on 10.9 too?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/0ad2bb432a30741156395fa048b1bacdbf6faf71

commit 0ad2bb432a30741156395fa048b1bacdbf6faf71
Author: Erik Chen <erikchen@chromium.org>
Date: Fri Jun 16 22:55:48 2017

Turn off mac hermetic toolchain for blink trybots.

The trybots are still running OSes < 10.12, so won't support the new hermetic
toolchain. This will be fixed when we support swarming for webkit tests on
macOS, at which point all the trybots can be updated to 10.12.

Bug:659270,  624049 
Change-Id: I60f12237584099cc38c2e59ff853a5d65238816a
Reviewed-on: https://chromium-review.googlesource.com/538876
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/trybots.py
[add] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/webkit_mac_trybot_no_hermetic_toolchain.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/use_v8_patch_on_blink_trybot.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.py
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/use_skia_patch_on_blink_trybot.json
[add] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.expected/webkit_mac_trybot_no_hermetic_toolchain.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/0ad2bb432a30741156395fa048b1bacdbf6faf71

commit 0ad2bb432a30741156395fa048b1bacdbf6faf71
Author: Erik Chen <erikchen@chromium.org>
Date: Fri Jun 16 22:55:48 2017

Turn off mac hermetic toolchain for blink trybots.

The trybots are still running OSes < 10.12, so won't support the new hermetic
toolchain. This will be fixed when we support swarming for webkit tests on
macOS, at which point all the trybots can be updated to 10.12.

Bug:659270,  624049 
Change-Id: I60f12237584099cc38c2e59ff853a5d65238816a
Reviewed-on: https://chromium-review.googlesource.com/538876
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/trybots.py
[add] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/webkit_mac_trybot_no_hermetic_toolchain.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/use_v8_patch_on_blink_trybot.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.py
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/use_skia_patch_on_blink_trybot.json
[add] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.expected/webkit_mac_trybot_no_hermetic_toolchain.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.py

My plan is to deploy Layout Tests on Swarming for Mac on Monday. (I had hope to do it last Thursday but converting the remaining Linux FYI bots took longer than I thought.)

Will update this bug when it has been deployed.

Ping ?

The CQ trybots are running the layout tests under swarming now. 

I don't think *all* of the trybots (or all of the waterfall bots, for that matter) are doing so yet.

The waterfall bots are not all yet running layout tests under swarming.

There is some issue with running layout tests with Mac 10.9 on swarming that I have yet to figure out.

With stuff running on swarming, there is no reason we can't have a single Mac 10.12 builder which triggers layout tests running on both Mac 10.9 and Mac 10.12

To say that another way;

We are now in the able to potentially run layout tests on Mac 10.9 and Mac 10.12 from a single Mac 10.12 compile *but* we need to figure out what is going on with Mac 10.9 and the swarming pool.

I had the Mac 10.12 compile with Mac 10.9 testing configuration running on the Mac Random Order bot for a while.

As far as I can see, we don't run the Mac 10.9 configuration as part of the try bots in the commit queue, so it currently isn't a priority to figure out what is going on there.

I think the configuration you are after is actually currently running on the Mac Random Order bot - you can see it here -> https://luci-milo.appspot.com/buildbot/chromium.fyi/WebKit%20Mac%20-%20RandomOrder/

This bot builds on Mac 10.12 and then runs Layout Tests on the following configurations;
 * (none) GPU on Mac on Mac-10.9.5
 * (none) GPU on Mac on Mac-10.10.5
 * (none) GPU on Mac on Mac-10.11.6
 * (none) GPU on Mac Retina on Mac-10.11.6
 * Intel GPU on Mac Retina on Mac-10.12.5
 * Intel GPU on Mac on Mac-10.12.5

Great! So once that's green, we can run it everywhere?

I've moved all the bots except;
 * (none) GPU on Mac on Mac-10.9.5
 * (none) GPU on Mac on Mac-10.9.5 (dbg)
 * (none) GPU on Mac Retina on Mac-10.11.6

Mac-10.9.5 is having random failures on swarming that I'm investigating.
We also don't have enough bots for "Mac Retina on Mac-10.11.6", so we are going to look at moving to "Mac Retina on Mac-10.12.5".

Is this why https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.11%20%28retina%29 is offline at the moment?

For context, these bots are the only bots still using the 10.10 SDK, which causes developer confusion every now and then. It'd be nice if we could stop supporting building with the 10.10 sdk.

That bot should not be offline and is unrelated to my change.

The last thing to be converted to run LayoutTests on swarming is Mac10.9 - pending on  https://crbug.com/736257 

I believe this is done, many thanks to tansell@ and dpranke@ for making this happen!