get layout tests running properly on 10.12 and 10.13 bots |
||||||||||||
Issue descriptionCurrent the "Mac 10.13 Tests" bot is actually running the layout tests on 10.12, and we don't have them running at all on "Mac 10.12 Tests". I think this is largely because we didn't have the capacity to run them on 10.13 in the CQ and mac_chromium_rel_ng mirrors Mac 10.13 Tests. We do have the layout tests running on "Chromium Mac 10.13" on chromium.fyi. We should straighten this out.
,
Jun 20 2018
,
Jun 21 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/5b28a5f82940c11f921a1f7fc9762db59167795f commit 5b28a5f82940c11f921a1f7fc9762db59167795f Author: Dirk Pranke <dpranke@chromium.org> Date: Thu Jun 21 01:45:03 2018
,
Jun 21 2018
Okay, I'm going to use the "quarantining" we're doing for browser_tests as an excuse to shuffle capacity around temporarily and get this fixed. My best guess is that we need ~120 swarming bots to handle the layout tests at peak. It looks like we have ~70 that I can temporarily take out of the 10.13 Chrome-quarantine pool, and we have another ~70 in the 'unassigned' pool in -m9 that we can probably move over. So, if I land the quarantine CL, I should be able to then land another CL to run the layout tests at at least 50% on 10.13 and stop running them (in CQ) on 10.12 immediately. Once the other 70 are added, we can probably ramp up to 100%, and then return all of the now-unneeded capacity in 10.12 to labs, in maybe 3-4 CLs total. I will proceed with this strategy. 1. CL to move 70 bots back to pool:Chrome: https://crrev.com/i/643349 2. CL to switch the layout tests to 10.13@50% in the CQ: https://crrev.com/c/1109458 3. Bug to reassign VMs: 845874
,
Jun 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/52f13686047d3bff273e6848b956fccaf79c3cb5 commit 52f13686047d3bff273e6848b956fccaf79c3cb5 Author: Dirk Pranke <dpranke@chromium.org> Date: Thu Jun 21 03:36:02 2018 Move the CQ Mac layout tests from 10.12 to 10.13. Currently the Mac10.13 Tests bot runs the layout tests on 10.12; this is done because we didn't have enough capacity on 10.13 to run them. This CL is one of a series that is attempting to fix this. In this particular CL, this switches the test step from 100% 10.12 to 50% 10.13, and also updates the chromium.clang ToTMac and 'Chromium Mac 10.13' bots to get around the unneeded hacks. A subsequent CL will ramp this to 100% (non-experimental) once we're sure we have the capacity. BUG=853356 Change-Id: I4a772f4bd82f427fde1931417dd60d0f80a8d40a Reviewed-on: https://chromium-review.googlesource.com/1109458 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#569147} [modify] https://crrev.com/52f13686047d3bff273e6848b956fccaf79c3cb5/testing/buildbot/chromium.clang.json [modify] https://crrev.com/52f13686047d3bff273e6848b956fccaf79c3cb5/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/52f13686047d3bff273e6848b956fccaf79c3cb5/testing/buildbot/chromium.mac.json [modify] https://crrev.com/52f13686047d3bff273e6848b956fccaf79c3cb5/testing/buildbot/test_suite_exceptions.pyl
,
Jun 21 2018
I realized belatedly that I should just be using the traffic-splitting features in dimension sets for this and we wouldn't need the experimental stuff and hoop-jumping at all. I'm going to revert the above change and switch to dimension sets and the custom trigger script tomorrow morning. We should still reconfigure the machines, though, as I think it's probably the best/fastest way to do the transition.
,
Jun 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c2917e9f46e6c4c516955d710c293eec99ded3d1 commit c2917e9f46e6c4c516955d710c293eec99ded3d1 Author: Dirk Pranke <dpranke@chromium.org> Date: Thu Jun 21 04:28:53 2018 Revert "Move the CQ Mac layout tests from 10.12 to 10.13." This reverts commit 52f13686047d3bff273e6848b956fccaf79c3cb5. Reason for revert: I should just use the custom trigger script and dimension sets instead ... Original change's description: > Move the CQ Mac layout tests from 10.12 to 10.13. > > Currently the Mac10.13 Tests bot runs the layout tests on 10.12; > this is done because we didn't have enough capacity on 10.13 to > run them. This CL is one of a series that is attempting to fix this. > > In this particular CL, this switches the test step from 100% > 10.12 to 50% 10.13, and also updates the chromium.clang ToTMac > and 'Chromium Mac 10.13' bots to get around the unneeded hacks. > > A subsequent CL will ramp this to 100% (non-experimental) once > we're sure we have the capacity. > > BUG=853356 > > Change-Id: I4a772f4bd82f427fde1931417dd60d0f80a8d40a > Reviewed-on: https://chromium-review.googlesource.com/1109458 > Reviewed-by: John Budorick <jbudorick@chromium.org> > Commit-Queue: Dirk Pranke <dpranke@chromium.org> > Cr-Commit-Position: refs/heads/master@{#569147} TBR=dpranke@chromium.org,tikuta@chromium.org,jbudorick@chromium.org Change-Id: I8cd37858ad7fe16a43009168d0d16a0535dac7e5 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 853356 Reviewed-on: https://chromium-review.googlesource.com/1108619 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#569153} [modify] https://crrev.com/c2917e9f46e6c4c516955d710c293eec99ded3d1/testing/buildbot/chromium.clang.json [modify] https://crrev.com/c2917e9f46e6c4c516955d710c293eec99ded3d1/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/c2917e9f46e6c4c516955d710c293eec99ded3d1/testing/buildbot/chromium.mac.json [modify] https://crrev.com/c2917e9f46e6c4c516955d710c293eec99ded3d1/testing/buildbot/test_suite_exceptions.pyl
,
Jun 21 2018
,
Jul 9
Any progress on this? I got "Max shard duration: 0:32:24.477730 (shard #3)" in the bot. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/89002
,
Jul 9
tikuta@: that's not a failure. The overhead of mapping isolated inputs on Mac is a bit high (there was an issue but I can't find it atm). 20min overhead + 12min execution is normal. I think the status is that tests are mostly running correctly but we need more capacity (issue 854874).
,
Jul 10
#10, yes, I know that it is not failure. But 20min overhead is too large, it can even become more than 30 mins. https://bugs.chromium.org/p/chromium/issues/detail?id=851355 I want to see whether migration to 10.13 fixes the issue or not.
,
Jul 19
Any progress on this? We see very slow webkit_layout_tests step yet in recent 24 hour. 6075s https://ci.chromium.org/b/8940641820329762592 5764s https://ci.chromium.org/b/8940642426814556672 5208s https://ci.chromium.org/b/8940641752973382016 5070s https://ci.chromium.org/b/8940643357325917792
,
Jul 28
I got sidetracked when I was working on it (on Jun 21) and then I went on vacation, so this kinda fell on the floor. I've picked it back up and will try to land it this weekend or early next week.
,
Jul 31
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ca0a1abeb0bf5a4504c4d88eea9aec025029a54d commit ca0a1abeb0bf5a4504c4d88eea9aec025029a54d Author: Dirk Pranke <dpranke@chromium.org> Date: Tue Jul 31 21:06:43 2018 Reconfigure webkit_layout_tests on Mac10.13 Tests. At one point we were capacity-constrained on the Mac CQ, and so in order to switch running the tests from 10.9 to something newer, I was forced to split the tests between 10.12 and 10.13. Really, they should all be on 10.13 since the CQ is configured to mirror the 'Mac 10.13 Tests' bot. This CL starts to fix that by configuring traffic splitting using the multi-dimension trigger script; this should start shifting the layout tests to 10.13. Eventually we'll rebalance the pools and run them exclusively on 10.13 when we have enough capacity. R=jbudorick@chromium.org, kbr@chromium.org BUG=853356 NOTRY=true Change-Id: I549c5e5b9e149d78606e692679d03e055be14ab7 Reviewed-on: https://chromium-review.googlesource.com/1111030 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#579567} [modify] https://crrev.com/ca0a1abeb0bf5a4504c4d88eea9aec025029a54d/testing/buildbot/chromium.clang.json [modify] https://crrev.com/ca0a1abeb0bf5a4504c4d88eea9aec025029a54d/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/ca0a1abeb0bf5a4504c4d88eea9aec025029a54d/testing/buildbot/chromium.mac.json [modify] https://crrev.com/ca0a1abeb0bf5a4504c4d88eea9aec025029a54d/testing/buildbot/test_suite_exceptions.pyl
,
Aug 6
,
Sep 22
not sure how we're going to do this at this point; someone needs to look to see if we can make the load-splitting work and/or figure out how to get more capacity. @jbudorick - I'm going to punt this over to you now to figure out how to move forward, let me know if you want to talk about it.
,
Oct 2
We're going to try to make the load-splitting work.
,
Oct 2
Since all our mac pools regularly hit 100% utilization during peak, we'll have split the load gradually while upgrading the 10.12 pool in batches to avoid overburdening any one pool. Here are some capacity graphs for pool:Chrome: 10.12 minis: http://shortn/_3pqxphfL44 10.13 VMs: http://shortn/_JsjtwON0I8 10.13 minis: http://shortn/_lr7OKzeQTD
,
Oct 4
,
Oct 19
,
Oct 19
,
Nov 2
ping? What is current status of this bug?
,
Nov 2
tikuta: I'm going to upload another CL for 871453 today -- once that goes in we can start with the load splitting.
,
Nov 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6822d5b7a05ea873c7744473a8e5940738a60f6e commit 6822d5b7a05ea873c7744473a8e5940738a60f6e Author: Brad Hall <bradhall@google.com> Date: Tue Nov 20 17:57:00 2018 Add 10.13 as an optional dimension for layout tests This will allow them to try 10.13 first and fall back to 10.12 if there aren't any hosts available. Bug: 853356 Change-Id: I0fd82376ebb2c4e501a78360bbbc675c90ee7d34 No-Try: true Reviewed-on: https://chromium-review.googlesource.com/c/1330693 Commit-Queue: Brad Hall <bradhall@google.com> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#609743} [modify] https://crrev.com/6822d5b7a05ea873c7744473a8e5940738a60f6e/testing/buildbot/chromium.mac.json [modify] https://crrev.com/6822d5b7a05ea873c7744473a8e5940738a60f6e/testing/buildbot/test_suite_exceptions.pyl
,
Nov 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6a8502885d85d83da10129f376bd83cf31cc5cbf commit 6a8502885d85d83da10129f376bd83cf31cc5cbf Author: Brad Hall <bradhall@google.com> Date: Mon Nov 26 21:34:07 2018 Revert "Add 10.13 as an optional dimension for layout tests" This reverts commit 6822d5b7a05ea873c7744473a8e5940738a60f6e. Reason for revert: https://crbug.com/908551 Original change's description: > Add 10.13 as an optional dimension for layout tests > > This will allow them to try 10.13 first and fall back to 10.12 if there > aren't any hosts available. > > Bug: 853356 > Change-Id: I0fd82376ebb2c4e501a78360bbbc675c90ee7d34 > No-Try: true > Reviewed-on: https://chromium-review.googlesource.com/c/1330693 > Commit-Queue: Brad Hall <bradhall@google.com> > Reviewed-by: John Budorick <jbudorick@chromium.org> > Cr-Commit-Position: refs/heads/master@{#609743} TBR=jbudorick@chromium.org,bradhall@google.com # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 853356 Change-Id: I2d620fd3bec6b43ceda004de3925154a24986d8e Reviewed-on: https://chromium-review.googlesource.com/c/1351503 Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#610950} [modify] https://crrev.com/6a8502885d85d83da10129f376bd83cf31cc5cbf/testing/buildbot/chromium.mac.json [modify] https://crrev.com/6a8502885d85d83da10129f376bd83cf31cc5cbf/testing/buildbot/test_suite_exceptions.pyl
,
Dec 5
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5f3baa794ec4e97c5d6e8f79bc329d7fe623c960 commit 5f3baa794ec4e97c5d6e8f79bc329d7fe623c960 Author: Brad Hall <bradhall@google.com> Date: Wed Dec 05 23:18:02 2018 Reland "Add 10.13 as an optional dimension for layout tests" > Add 10.13 as an optional dimension for layout tests > > This will allow them to try 10.13 first and fall back to 10.12 > if there aren't any hosts available. Bug: 853356 Change-Id: I2a23e3abacc747d888a88af1e0f27df235a60512 Reviewed-on: https://chromium-review.googlesource.com/c/1363647 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Brad Hall <bradhall@google.com> Cr-Commit-Position: refs/heads/master@{#614165} [modify] https://crrev.com/5f3baa794ec4e97c5d6e8f79bc329d7fe623c960/testing/buildbot/chromium.mac.json [modify] https://crrev.com/5f3baa794ec4e97c5d6e8f79bc329d7fe623c960/testing/buildbot/test_suite_exceptions.pyl
,
Jan 18
(4 days ago)
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a05aae00b991ec291f54a6d43f4782a276bd0ffa commit a05aae00b991ec291f54a6d43f4782a276bd0ffa Author: John Budorick <jbudorick@chromium.org> Date: Fri Jan 18 14:54:45 2019 Move layout tests entirely to Mac 10.13. We've been seeing a relatively consistent pattern at peak on mac_chromium_rel_ng: pending times creep toward 20 minutes, then skyrocket to 60 minutes as layout test tasks get dumped into a relatively limited 10.12.6 pool. This should alleviate load on the 10.12.6 pool (allowing us to upgrade more of it) and hopefully will result in fewer task expirations. No-Try: true Tbr: bradhall@chromium.org,kbr@chromium.org Bug: 853356 Change-Id: I7248fb7d0578e1dcf343da2b21ab47a575bd4c1e Reviewed-on: https://chromium-review.googlesource.com/c/1420306 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#624121} [modify] https://crrev.com/a05aae00b991ec291f54a6d43f4782a276bd0ffa/testing/buildbot/chromium.mac.json [modify] https://crrev.com/a05aae00b991ec291f54a6d43f4782a276bd0ffa/testing/buildbot/test_suite_exceptions.pyl |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by dpranke@chromium.org
, Jun 16 2018