mac_chromium_rel_ng is overloaded |
|||||||
Issue description*** This is a public tracking bug for the CQ outage. *** mac_chromium_rel_ng - https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng is experiencing delays due to an issue with layout tests. At this time, layout tests are turned off in this builder, but the builder still has a backlog and still needs to recover. The incident is tracked internally at http://shortn/_7Q4dIiSjL5 . I'll post public updates to this bug.
,
Nov 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d655201369a7f1c12ca20820bc98a94e47389749 commit d655201369a7f1c12ca20820bc98a94e47389749 Author: Sergey Berezin <sergeyberezin@google.com> Date: Tue Nov 27 23:50:41 2018 [ios] Set swarming task priorities to 30 for CQ bots Priority 29 was introduced accidentally during LUCI migration, and was later fixed. All the other Chromium CQ isolated tests now run at the default priority 30. Updating the priority to prevent ios-* builder from starving other Mac try bots. Bug: 908929 Change-Id: I7a70de756c4b63bac0a5bd55b9fa6ad285f9aea5 Reviewed-on: https://chromium-review.googlesource.com/c/1352583 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org> Cr-Commit-Position: refs/heads/master@{#611425} [modify] https://crrev.com/d655201369a7f1c12ca20820bc98a94e47389749/ios/build/bots/chromium.mac/ios-simulator-cronet.json [modify] https://crrev.com/d655201369a7f1c12ca20820bc98a94e47389749/ios/build/bots/chromium.mac/ios-simulator-full-configs.json [modify] https://crrev.com/d655201369a7f1c12ca20820bc98a94e47389749/ios/build/bots/chromium.mac/ios-simulator.json [modify] https://crrev.com/d655201369a7f1c12ca20820bc98a94e47389749/ios/build/bots/chromium.mac/ios-slimnav.json [modify] https://crrev.com/d655201369a7f1c12ca20820bc98a94e47389749/ios/build/bots/chromium.mac/ios-webview.json
,
Nov 27
We've identified the issue of incorrect priorities in ios-simulator which starve mac_chromium_rel_ng tests. https://crrev.com/c/1352583 should fix it, and hopefully mac_chromium_rel_ng will start clearing its backlog. This may, however, slow down ios-simulator.
,
Nov 28
Issue 908948 has been merged into this issue.
,
Nov 28
The pending queue cleared, lowering priority. I'll keep watching the builder tomorrow.
,
Nov 28
Issue 908551 has been merged into this issue.
,
Nov 28
The builder seems healthy, along with its "competitor" ios-simulator. Lowering the priority even more, will check it again throughout the day and close if it looks healthy.
,
Nov 28
We probably want to leave this as a P1 until we've restored the layout tests to mac_chromium_rel_ng.
,
Nov 29
The bot indeed performed well today, I think we're ready for trying the layout tests again (once the branch is done).
,
Nov 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ca57f95acfe72a1a5bff7cfbcedb6f4a25858469 commit ca57f95acfe72a1a5bff7cfbcedb6f4a25858469 Author: John Budorick <jbudorick@chromium.org> Date: Thu Nov 29 02:45:17 2018 Reenable webkit_layout_tests on Mac10.12 Tests. Note that this does not affect mac_chromium_rel_ng. Bug: 908929 Change-Id: I4834ae78bb0e037a2def2fd73762b5a90c0ef949 Reviewed-on: https://chromium-review.googlesource.com/c/1354131 Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#612001} [modify] https://crrev.com/ca57f95acfe72a1a5bff7cfbcedb6f4a25858469/testing/buildbot/chromium.mac.json [modify] https://crrev.com/ca57f95acfe72a1a5bff7cfbcedb6f4a25858469/testing/buildbot/test_suite_exceptions.pyl
,
Dec 5
What's the ETA for getting mac_chromium_rel_ng to run webkit_layout_tests again?
,
Dec 5
Branch has been cut, I think it's time to add them back. jbudorick: is it safe to add the layout tests back?
,
Dec 5
Great, thanks!
,
Dec 5
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d30f3083f96d5998e23e741e835fd21d26cc6c22 commit d30f3083f96d5998e23e741e835fd21d26cc6c22 Author: Brad Hall <bradhall@google.com> Date: Wed Dec 05 20:07:15 2018 Reenable webkit_layout_tests on Mac10.13 Bug: 908929 Change-Id: Ica3cb1ee2b0b3c53259d02a6059f0124c7381e3e Reviewed-on: https://chromium-review.googlesource.com/c/1363643 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Brad Hall <bradhall@google.com> Cr-Commit-Position: refs/heads/master@{#614071} [modify] https://crrev.com/d30f3083f96d5998e23e741e835fd21d26cc6c22/testing/buildbot/chromium.mac.json [modify] https://crrev.com/d30f3083f96d5998e23e741e835fd21d26cc6c22/testing/buildbot/test_suite_exceptions.pyl
,
Dec 7
The incident is now closed, postmortem (internal) is at go/chops-pm-108. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by sergeybe...@chromium.org
, Nov 27