New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 650747 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocking:
issue 707891



Sign in to add a comment

Support --gtest_filter, --gtest_repeat, --test-launcher-retry-limit in run-webkit-tests for FindIt

Project Member Reported by jeffcarp@chromium.org, Sep 27 2016

Issue description

To allow FindIt to run the Blink layout tests, the run-webkit-tests command needs to support these 3 parameters:

- [ ] --gtest_filter
- [x] --gtest_repeat
- [ ] --test-launcher-retry-limit

Open questions:

- [x] We've already added support for 2 gtest parameters as env variables (https://crrev.com/2350853003), can we also accept these as env variables or must we support them as command line arguments? [yes, must support cmd line]
 
Components: Blink>Infra

Comment 2 by st...@chromium.org, Sep 27 2016

To match with base/test/launcher-based gtest (https://cs.chromium.org/chromium/src/base/test/launcher/), they have to be command line arguments.
Description: Show this description
Description: Show this description
Description: Show this description

Comment 6 by st...@chromium.org, Apr 4 2017

Blocking: 707891
Hey Shuotao, are there any docs I can reference for these flags? I'm pretty certain I know what --gtest_repeat needs to do but am less clear on the others.

Also what is the relative priority of this? Is running run-webkit-tests in FindIt blocked on moving all platforms to swarming?
Project Member

Comment 8 by bugdroid1@chromium.org, Jun 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/68444af30e59f730704616740707f77c90b27850

commit 68444af30e59f730704616740707f77c90b27850
Author: Jeff Carpenter <jeffcarp@chromium.org>
Date: Wed Jun 21 01:31:49 2017

Add support for --gtest_repeat to run-webkit-tests

The context for this change is to allow FindIt, which only speaks GTest,
to run run-webkit-tests.

Bug:  650747 
Change-Id: Ic4930cda309395f1a19a619a701140909f6f296c
Reviewed-on: https://chromium-review.googlesource.com/541757
Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#481064}
[modify] https://crrev.com/68444af30e59f730704616740707f77c90b27850/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
[modify] https://crrev.com/68444af30e59f730704616740707f77c90b27850/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py

Description: Show this description
Owner: robertma@chromium.org
Status: Assigned (was: Available)
Labels: -Pri-3 Pri-2
Status: Started (was: Assigned)
One more flag: --gtest_also_run_disabled_tests

CL up: https://crrev.com/c/893683
Project Member

Comment 12 by bugdroid1@chromium.org, Jan 31 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/01637921aef09b8582164ebabdeb317f9bc43208

commit 01637921aef09b8582164ebabdeb317f9bc43208
Author: Robert Ma <robertma@chromium.org>
Date: Wed Jan 31 20:49:45 2018

[r-w-t] Add --gtest_also_run_disabled_tests as an alias for --skipped=ignore

When both options are given, --gtest_also_run_disabled_tests takes
precedence.

Also add a TODO to change the execution order of --gtest_repeat
(--repeat-each).

Bug:  650747 
Change-Id: I6159ccc3975a979570517a54dab5461234b81f4e
Reviewed-on: https://chromium-review.googlesource.com/893683
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#533395}
[modify] https://crrev.com/01637921aef09b8582164ebabdeb317f9bc43208/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
[modify] https://crrev.com/01637921aef09b8582164ebabdeb317f9bc43208/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
[modify] https://crrev.com/01637921aef09b8582164ebabdeb317f9bc43208/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py

Two CLs up for the two remaining flags:

- [.] --gtest_filter (CL: https://crrev.com/c/900125)
- [x] --gtest_repeat
- [.] --test-launcher-retry-limit (CL: https://crrev.com/c/895812)
- [x] --gtest_also_run_disabled_tests
Cc: chanli@chromium.org
Project Member

Comment 15 by bugdroid1@chromium.org, Feb 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/be1a84101cd8b278dc57b1c4ac24ec89ee298cba

commit be1a84101cd8b278dc57b1c4ac24ec89ee298cba
Author: Robert Ma <robertma@chromium.org>
Date: Tue Feb 13 08:24:09 2018

[r-w-t] Add --test-launcher-retry-limit and remove --retry-failures

--test-launcher-retry-limit is added as an alias of --num-retries.
Besides, the legacy --retry-failures flag is removed as no one is using
it any more. --no-retry-failures is kept for now as a logical shorthand
for --num-retries=0. The behaviours (both w/ and w/o the flags) are not
changed.

Furthermore, improve the unit tests by fixing a few wrong assertions of
filesystem (they are always true), removing some unrelated flags in a
few tests, and splitting the test for --clobber-old-results out of the
test for retrying.

Bug:  650747 
Change-Id: I1617e1eee36339eb7df4d55d832a0e54af204c0c
Reviewed-on: https://chromium-review.googlesource.com/895812
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536291}
[modify] https://crrev.com/be1a84101cd8b278dc57b1c4ac24ec89ee298cba/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
[modify] https://crrev.com/be1a84101cd8b278dc57b1c4ac24ec89ee298cba/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
[modify] https://crrev.com/be1a84101cd8b278dc57b1c4ac24ec89ee298cba/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py

Project Member

Comment 16 by bugdroid1@chromium.org, Feb 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2dde4abe519fdf6d4531aaa1a8e8e8b0b022b797

commit 2dde4abe519fdf6d4531aaa1a8e8e8b0b022b797
Author: Robert Ma <robertma@chromium.org>
Date: Tue Feb 13 08:33:13 2018

[r-w-t] Add --gtest_filter for FindIt

The argument takes a colon(:)-separated list of tests to run. Wildcards
are NOT supported (as FindIt does not use wildcards). In fact, the code
simply splits the list by colons and adds them as positional arguments.

Bug:  650747 
Change-Id: I9c4f863f17d15eec869649a0d7cb2d9c37932ea6
Reviewed-on: https://chromium-review.googlesource.com/900125
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536292}
[modify] https://crrev.com/2dde4abe519fdf6d4531aaa1a8e8e8b0b022b797/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
[modify] https://crrev.com/2dde4abe519fdf6d4531aaa1a8e8e8b0b022b797/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py

Status: Fixed (was: Started)
chanli@ all done!
Project Member

Comment 18 by bugdroid1@chromium.org, Apr 26 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5653652498e1275cda59da9d06ef4b58e173da5b

commit 5653652498e1275cda59da9d06ef4b58e173da5b
Author: Robert Ma <robertma@chromium.org>
Date: Thu Apr 26 15:49:19 2018

[blinkpy] Fix the semantics of --gtest_repeat in r-w-t

gtest's --gtest_repeat runs all the tests in iterations, i.e. its
execution order looks like ABCABCABC. Hence, the correct alias for this
flag in r-w-t should be --iterations instead of --repeat-each (the
latter repeats each test individually, like AAABBBCCC).

Bug:  650747 
Change-Id: I9abe19b89951d7b4b655c7a3118ae8d1c2654daf
Reviewed-on: https://chromium-review.googlesource.com/1029180
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554038}
[modify] https://crrev.com/5653652498e1275cda59da9d06ef4b58e173da5b/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py
[modify] https://crrev.com/5653652498e1275cda59da9d06ef4b58e173da5b/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py
[modify] https://crrev.com/5653652498e1275cda59da9d06ef4b58e173da5b/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests_unittest.py

Project Member

Comment 19 by bugdroid1@chromium.org, Oct 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6e58bdb9bc67b99300645edbd41d71437701d9b0

commit 6e58bdb9bc67b99300645edbd41d71437701d9b0
Author: Robert Ma <robertma@chromium.org>
Date: Wed Oct 10 21:18:40 2018

Reland "[blinkpy] Fix the semantics of --gtest_repeat in r-w-t"

This is a reland of 5653652498e1275cda59da9d06ef4b58e173da5b

It was accidentally reverted (perhaps due to a bad merge) in
https://crrev.com/c/1029430 .

Original change's description:
> [blinkpy] Fix the semantics of --gtest_repeat in r-w-t
>
> gtest's --gtest_repeat runs all the tests in iterations, i.e. its
> execution order looks like ABCABCABC. Hence, the correct alias for this
> flag in r-w-t should be --iterations instead of --repeat-each (the
> latter repeats each test individually, like AAABBBCCC).
>
> Bug:  650747 
> Change-Id: I9abe19b89951d7b4b655c7a3118ae8d1c2654daf
> Reviewed-on: https://chromium-review.googlesource.com/1029180
> Reviewed-by: Chan Li <chanli@chromium.org>
> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
> Commit-Queue: Robert Ma <robertma@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#554038}

Bug:  650747 
Change-Id: Ia41ad977605007727bfed7f3f1394b545588426c
Reviewed-on: https://chromium-review.googlesource.com/c/1274109
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#598511}
[modify] https://crrev.com/6e58bdb9bc67b99300645edbd41d71437701d9b0/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py
[modify] https://crrev.com/6e58bdb9bc67b99300645edbd41d71437701d9b0/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py
[modify] https://crrev.com/6e58bdb9bc67b99300645edbd41d71437701d9b0/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests_unittest.py

Sign in to add a comment