ios_chrome_ui_egtests /DownloadManagerTestCase is flaky on bots for parallel tests when shards=5 |
||||||||||||||
Issue descriptionchrome-mobile-test-engprod@ is working on xcode-parallelization feature to run EG tests crbug/912323 Initial change in /recipe_module was submit https://chromium-review.googlesource.com/c/chromium/src/+/1330204 and now we are testing the xcode-runner https://chromium-review.googlesource.com/c/chromium/src/+/1370830/7 during test runs on 'ios-simulator-full-configs' try-bot we've observed flakiness of DownloadManagerTestCase for test runs when number of running in parallel simulators is 5+. On local Mac laptop issue was not reproduced. Passed test output for shards=3 http://go/DownloadManagerTestCase_iphone7_ios12.1_3shards Failed test output for shards = 5 http://go/DownloadManagerTestCase-iphone7-12.1-failures
,
Dec 11
The bug says that the failure is not reproducible locally. With that I wonder if this is really a test issue or the infrastructure problem (EG framework or bots infra)?
,
Dec 12
monufriienko@ how did you test this locally? Did you invoke the test the same way the bots do (not from their python script, but from the same xcode invocation).
,
Dec 12
Eugene, Justin, I did a run with shards=5 locally and DownloadManagerTestCase passed (there were 5 other failures). And then I increased number of shards to 6 and ran 2 times. For both shards=6 runs DownloadManagerTestCase failed 2 times (for each run different tests failed). For local run I used command below, xcodebuild test-without-building -xctestrun egtestui.xctestrun -parallel-testing-enabled YES -destination 'platform=iOS Simulator,OS=12.1,name=iPhone 7' -parallel-testing-worker-count 6 where egtestui.xctestrun is a file in an attach (please update %CHROMIUM_SRC% to real chromium/src dir) Also I've attached session log with stack_trace and screenshots of failures
,
Dec 12
Thanks Maxym! How important is this bug? Realistically I don't have much time to work on something other than slim-navigation-manager in the next month or two. But if this bug has hight priority, I will have to take a look.
,
Dec 13
Marking as assigned since Eugene is assessing severity. Seems like if this flake only occurs under specific testing conditions that it is more likely an infra related issue though?
,
Dec 14
Also when we have 7 shards? https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-slimnav/844
,
Dec 14
#7 is a timeout, not a test failure. The tests actually pass. |
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by monufriienko@chromium.org
, Dec 11