Request for optional layout test try bot (LayoutNG) |
||||||||
Issue descriptionWe'd like to set up an optional try bot that runs the blink layout tests with a specific flag. This would be for testing LayoutNG which is a new layout system under development. Rather than use a virtual test suite, which would increase cycle time for everyone, we'd like to use the same approach as the for slimming paint v2 (detailed in issue 622865). This bot would be used by the layout team to track pass rates and to guard against regressions. I'd expect this try job to be triggered around 10-20 times per day. A waterfall bot is not needed at this time. Given the low rate I'd expect a single bot to be sufficient and we only need one platform. The specific platform doesn't really matter (as long as it is one of linux, win, mac, chrome).
,
Apr 3 2017
,
Apr 3 2017
Yep! I set up the linux_layout_tests_slimming_paint_v2 before; this involved: 1. File an issue to request a VM (like issue 626379 ) 2. Update src/tools/mb/mb_config.pyl (http://crrev.com/2156633003) 3. Update slaves.cfg, master.cfg and config in chromium_tests recipe module in build repo (http://crrev.com/2153503006) 4. File an issue for a master restart.
,
Apr 3 2017
So, which specific flag would run-webkit-tests be run with for this bot?
,
Apr 3 2017
--enable-blink-features=LayoutNG
,
Apr 5 2017
CLs for steps 2 and 3: https://codereview.chromium.org/2804893002 https://chromium-review.googlesource.com/c/469254/
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/72a4c8afaf07ec0832bc9554e914628de1d67cd5 commit 72a4c8afaf07ec0832bc9554e914628de1d67cd5 Author: qyearsley <qyearsley@chromium.org> Date: Thu Apr 06 02:16:30 2017 Update mb_config.pyl for linux_layout_tests_layout_ng try bot. Related build-repo change: https://chromium-review.googlesource.com/c/469254/ The plan would be to commit this first, then the build repo CL, then request a master restart. BUG= 706183 Review-Url: https://codereview.chromium.org/2804893002 Cr-Commit-Position: refs/heads/master@{#462328} [modify] https://crrev.com/72a4c8afaf07ec0832bc9554e914628de1d67cd5/tools/mb/mb_config.pyl
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/437b20aeedbc9304008fe8d929f28a464e99aee1 commit 437b20aeedbc9304008fe8d929f28a464e99aee1 Author: Quinten Yearsley <qyearsley@google.com> Date: Thu Apr 06 16:44:08 2017 Add config for linux_layout_tests_layout_ng try bot. Bug: 706183 Change-Id: I64ca5b17e534f298ba02b0104ea86148aa10f5cf Reviewed-on: https://chromium-review.googlesource.com/469254 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Quinten Yearsley <qyearsley@chromium.org> [add] https://crrev.com/437b20aeedbc9304008fe8d929f28a464e99aee1/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_linux_layout_tests_layout_ng.json [add] https://crrev.com/437b20aeedbc9304008fe8d929f28a464e99aee1/scripts/slave/recipes/chromium.expected/full_chromium_fyi_WebKit_Linux_layout_ng_Dummy_Builder.json [modify] https://crrev.com/437b20aeedbc9304008fe8d929f28a464e99aee1/scripts/slave/recipe_modules/chromium_tests/trybots.py [modify] https://crrev.com/437b20aeedbc9304008fe8d929f28a464e99aee1/masters/master.tryserver.chromium.linux/master.cfg [modify] https://crrev.com/437b20aeedbc9304008fe8d929f28a464e99aee1/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py [modify] https://crrev.com/437b20aeedbc9304008fe8d929f28a464e99aee1/masters/master.tryserver.chromium.linux/slaves.cfg
,
Apr 7 2017
The try bot now exists: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_layout_tests_layout_ng Now we can try it out and see if it works as expected :-D
,
Apr 7 2017
Awesome, thank you.
,
Apr 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ab81cf5d84d85f10c48b1c02dc9e8999df8928a3 commit ab81cf5d84d85f10c48b1c02dc9e8999df8928a3 Author: eae <eae@chromium.org> Date: Mon Apr 10 06:14:52 2017 Trigger linux_layout_tests_layout_ng for LayoutNG changes Change PRESUBMIT.py for third_party/WebKit to trigger the LayoutNG test bot, linux_layout_tests_layout_ng, which runs all layout tests with NG. R=thakis@chromium.org BUG= 706183 Review-Url: https://codereview.chromium.org/2806593005 Cr-Commit-Position: refs/heads/master@{#463177} [modify] https://crrev.com/ab81cf5d84d85f10c48b1c02dc9e8999df8928a3/third_party/WebKit/PRESUBMIT.py
,
Apr 10 2017
The bot appears to be working fine, thank you, however it's running the tests with the wrong flag, it needs to be --additional-driver-flag=--enable-blink-features=LayoutNG (i.e. the --additional-driver-flag= prefix needs to be added). I should have pointed that out in the request, my bad. Sorry about the confusion and extra work.
,
Apr 10 2017
NP, simple change: https://chromium-review.googlesource.com/473326
,
Apr 11 2017
The step of webkit_tests in linux_layout_tests_layout_ng is failing, and thus blocking commits when it is automatically added to CQ_INCLUDE_TRYBOTS. Is this intended (the tests are still failing and we can skip the bot)?
,
Apr 11 2017
The adding of this bot to CQ_INCLUDE_TRYBOTS was added in CL https://codereview.chromium.org/2806593005. Recent build results: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_layout_tests_layout_ng/builds/42 It looks like there are about 1000 tests that behave differently under LayoutNG, including: - some crashes - some image diffs (vertical shift) - some whitespace diffs in JS tests - possibly other differences? eae@, should we temporarily revert the presubmit change until the "expected failures" are listed out and have bugs filed? Apart from filing bugs for the above categories of mismatches, it might be helpful to add a flag-specific expectations file in LayoutTests/FlagExpectations, then burn down that list.
,
Apr 11 2017
I'm temporarily reverting the PRESUBMIT.PY change http://crrev.com/2814913002
,
Apr 11 2017
,
Apr 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/51ce668aebf388516f2349a0b55bc622c14e3f4c commit 51ce668aebf388516f2349a0b55bc622c14e3f4c Author: glebl <glebl@chromium.org> Date: Tue Apr 11 23:56:26 2017 Revert "Trigger linux_layout_tests_layout_ng for LayoutNG changes" This reverts commit ab81cf5d84d85f10c48b1c02dc9e8999df8928a3. Reason for revert: linux_layout_tests_layout_ng is blocking Chromium Presubmit Queue. Examples: - http://crrev.com/2810833003 - http://crrev.com/2805283003 - https://chromium-review.googlesource.com/c/474547/ linux_layout_tests_layout_ng bot is not designed to pass all LayoutTests. It should be configured as optional. BUG= 706183 Review-Url: https://codereview.chromium.org/2814913002 Cr-Commit-Position: refs/heads/master@{#463844} [modify] https://crrev.com/51ce668aebf388516f2349a0b55bc622c14e3f4c/third_party/WebKit/PRESUBMIT.py
,
Apr 12 2017
That was the plan. The bot should only have been triggered for layout/ng changes, was that not the case?
,
Apr 21 2017
I think that this bug should be closed now since the builder exists; we should file a separate bug for other issues, I think. Note: In theory, what I'd expect to happen normally is that all the tests should be run, and then tests that failed should be re-run. If a test fails both with and without a patch, then it shouldn't count as a failure for that patch. So, it *should* be OK to use add this bot as an extra try bot in PRESUBMIT. In theory. But, it's probably still be a good idea to add flag-specific layout test expectations for --enable-blink-features=LayoutNG.
,
Apr 21 2017
Opened new bug for current problems with this try bot: bug 714203 .
,
May 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6e27c23b503ebc4cef57d104f40950aae9dc1bcc commit 6e27c23b503ebc4cef57d104f40950aae9dc1bcc Author: kojii <kojii@chromium.org> Date: Sun May 14 08:25:28 2017 Re-land: Trigger linux_layout_tests_layout_ng for LayoutNG changes This is a re-land of r463177, after 23k lines were added to: LayoutTests/FlagExpectations/enable-blink-features=LayoutNG Change PRESUBMIT.py for third_party/WebKit to trigger the LayoutNG test bot, linux_layout_tests_layout_ng, which runs all layout tests with NG. R=thakis@chromium.org, eae@chromium.org, glebl@chromium.org BUG= 706183 , 714203 Review-Url: https://codereview.chromium.org/2854783002 Cr-Commit-Position: refs/heads/master@{#471625} [modify] https://crrev.com/6e27c23b503ebc4cef57d104f40950aae9dc1bcc/third_party/WebKit/PRESUBMIT.py |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by jparent@chromium.org
, Apr 3 2017Components: -Infra>Client>Chrome Blink>Infra