New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 3
Type: Bug



Sign in to add a comment
link

Issue 912323: iOS EG test sharding using XCodebuild v10

Reported by monufriienko@google.com, Dec 5 Project Member

Issue description

Implement iOS EG test sharding to run tests using XCodebuild v10
 

Comment 1 by monufriienko@google.com, Dec 5

Status: Started (was: Untriaged)

Comment 5 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd7d1b5a709c73a5c3171df3e96ba9971749880b

commit cd7d1b5a709c73a5c3171df3e96ba9971749880b
Author: git config --global <monufriienko@chromium.org>
Date: Fri Dec 14 20:12:43 2018

Added runner for sharding egtests .

Runner runs test using command:

xcodebuild test-without-building -xctestrun file.xctestrun \
      -parallel-testing-enabled YES -parallel-testing-worker-count %d% \
      -destination "destination"  -resultBundlePath %output_path%

that allows to split tests between few clones of simulator
(`destination` parameter) and stores an entire output of the command
(session and diagnostic logs) in specified folder(`resultBundlePath`).

Change-Id: I0a50680bb6946d227ab19719c16cbecffc3f8456
Bug: 912323
Reviewed-on: https://chromium-review.googlesource.com/c/1330204
Commit-Queue: Maksym Onufriienko <monufriienko@chromium.org>
Reviewed-by: Justin Cohen <justincohen@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616802}
[modify] https://crrev.com/cd7d1b5a709c73a5c3171df3e96ba9971749880b/ios/build/bots/scripts/run.py
[modify] https://crrev.com/cd7d1b5a709c73a5c3171df3e96ba9971749880b/ios/build/bots/scripts/test_runner.py
[modify] https://crrev.com/cd7d1b5a709c73a5c3171df3e96ba9971749880b/ios/build/bots/scripts/test_runner_test.py
[add] https://crrev.com/cd7d1b5a709c73a5c3171df3e96ba9971749880b/ios/build/bots/scripts/xcodebuild_runner.py
[add] https://crrev.com/cd7d1b5a709c73a5c3171df3e96ba9971749880b/ios/build/bots/scripts/xcodebuild_runner_test.py

Comment 6 by bugdroid1@chromium.org, Jan 8

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/39ecb197c8932f7e2951c9e186d9de69c98af1b6

commit 39ecb197c8932f7e2951c9e186d9de69c98af1b6
Author: git config --global <monufriienko@chromium.org>
Date: Tue Jan 08 19:39:20 2019

Added test_args and test_env to xcodebuildrunner.

Bug: 912323
Change-Id: I48a2905b5660c936eb59c730f69b37197be080c2
Reviewed-on: https://chromium-review.googlesource.com/c/1398313
Commit-Queue: Maksym Onufriienko <monufriienko@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#620838}
[modify] https://crrev.com/39ecb197c8932f7e2951c9e186d9de69c98af1b6/ios/build/bots/scripts/xcodebuild_runner.py

Comment 7 by sergeybe...@chromium.org, Jan 9

Moving a discussion from CL: https://crrev.com/c/1398391/8/ios/build/bots/chromium.fyi/ios12-sdk-simulator.json#5

sergeyberezin:
nit: ironically, Xcode 10.1 is now in GM, so probably no point in testing its beta? :-) But reverting the CL should be the easiest thing to do here indeed.

lindsayw:
Well, we'd not want to revert the whole CL if we want to keep the sharding for any tests that are working as we want, and on the other hand I don't want to have to sort through each line of the file in order to roll this bot to the next xcode beta version and get back the coverage we expect to be in place. 

It is true we don't have a current xcode beta right now, but that is very temporary so todo's should be in place for putting things back together.

Comment 8 by sergeybe...@chromium.org, Jan 9

Given how it's currently implemented, we can't mix EG tests with sharding and other types of tests on the same bot. So IMHO if this experiment is successful, the easiest thing to do is to revert this CL, and add EG test sharding to production bots dedicated to EG tests separately.

Or, we can fix the recipe to treat EG tests and other tests differently, so they both work together nicely.

Comment 9 by bugdroid1@chromium.org, Jan 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/12b3dedcbe39a8af3b3af51624a9e2e654112296

commit 12b3dedcbe39a8af3b3af51624a9e2e654112296
Author: git config --global <monufriienko@chromium.org>
Date: Mon Jan 14 19:09:52 2019

Rename "xcode_parallelization" parameter, use underscore instead of dash.

Dash was added in
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1404137

Bug:912323
Change-Id: I061fc29cccefd76fe82273dcc0ed2b3bb4d7f8b1
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/c/1409702
Commit-Queue: Maksym Onufriienko <monufriienko@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>

[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/explain.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/webrtc/ios.expected/no_tests.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/timed_out.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/xparallel_run_and_skip.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/clobber_build.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/icu_patch.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/expiration_test.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/expired.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/xcode_build_version.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/api.py
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/basic.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/no_exit_code.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/gn.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/test_failure.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/infra_failure.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/basic.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/max_runtime_test.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/xcode_build_version_luci.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/deprecate_xcode_version.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/errors.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/unified_builder_tester.expected/basic.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/unified_builder_tester.expected/goma_canary.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/use_trusted_cert.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/fyi.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/unified_builder_tester.expected/goma.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/clobber_checkout.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/perf_test.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/webrtc/ios.expected/basic.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/additional_compile_targets.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/webrtc/chromium_ios.expected/basic_goma_build.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/use_wpr_tools.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/unified_builder_tester.expected/basic_experimental.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipe_modules/ios/examples/full.expected/device_check_false.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/no_tests.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/webrtc/ios.expected/trybot.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/no_compilation.json
[modify] https://crrev.com/12b3dedcbe39a8af3b3af51624a9e2e654112296/scripts/slave/recipes/ios/try.expected/parent.json

Comment 10 by bugdroid1@chromium.org, Jan 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2b883374d5f51543b0b76e799af04013dca80eba

commit 2b883374d5f51543b0b76e799af04013dca80eba
Author: git config --global <monufriienko@chromium.org>
Date: Mon Jan 14 20:36:43 2019

Added eg_cq_tests_ios12_simulator_fyi_experiment to run ios12-sdk-simulator.FYI

Also updated ios12-sdk-simulator.FYI to run only EG-tests

Bug: 912323
Change-Id: I5831e98eaa25846cfeefbcf08eb00412fdf3ec08
Reviewed-on: https://chromium-review.googlesource.com/c/1398391
Commit-Queue: Maksym Onufriienko <monufriienko@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Reviewed-by: Justin Cohen <justincohen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#622585}
[modify] https://crrev.com/2b883374d5f51543b0b76e799af04013dca80eba/ios/build/bots/chromium.fyi/ios12-sdk-simulator.json
[modify] https://crrev.com/2b883374d5f51543b0b76e799af04013dca80eba/ios/build/bots/scripts/run.py
[add] https://crrev.com/2b883374d5f51543b0b76e799af04013dca80eba/ios/build/bots/tests/eg_cq_tests_ios12_simulator_fyi_experiment.json

Comment 11 by bugdroid, Feb 14 (3 days ago)

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/685b3c338b7d6b9d7eb609f2f82355213eb577a7

commit 685b3c338b7d6b9d7eb609f2f82355213eb577a7
Author: git config --global <monufriienko@chromium.org>
Date: Thu Feb 14 03:40:47 2019

Change priority to 35 for fyi/ios12-sdk-simulator.json

Bug: 912323
Change-Id: I3381adad6df31cd96bc72fe24b8d34cc47c817f2
Reviewed-on: https://chromium-review.googlesource.com/c/1471330
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Commit-Queue: Maksym Onufriienko <monufriienko@chromium.org>
Cr-Commit-Position: refs/heads/master@{#632063}
[modify] https://crrev.com/685b3c338b7d6b9d7eb609f2f82355213eb577a7/ios/build/bots/chromium.fyi/ios12-sdk-simulator.json

Sign in to add a comment