Various tests are flaky (timeout) on the "ios-simulator" bot |
||||||||||||||||||||||||
Issue descriptionhttps://uberchromegw.corp.google.com/i/chromium.mac/builders/ios-simulator/builds/16338 This was ios_net_unittests (iPad Air 2 iOS 10.0). The stdout is: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.mac%2Fios-simulator%2F16338%2F%2B%2Frecipes%2Fsteps%2Fios_net_unittests__iPad_Air_2_iOS_10.0_%2F0%2Fstdout And it ends like this: +-------------------------------------------------------------------------------+ | Shard 0 https://chromium-swarm.appspot.com/user/task/355feddae6e80c10 | +-------------------------------------------------------------------------------+ Device wiped. /b/s/w/irheAvLd/src/out/Debug-iphonesimulator/iossim -d iPad Air 2 -s 10.0 /b/s/w/irheAvLd/src/out/Debug-iphonesimulator/ios_net_unittests.app User defaults from command line: IDETestRunSpecificationPath = /var/folders/p4/py6nlvcx4fs5s3fg51_pzqmc0000gm/T/DE98128C-EBA4-4FF1-A9C9-543627ACC7E1 +-------------------------------------------------------------------------------+ | End of shard 0 Pending: 8.7s Duration: 1202.7s Bot: build216-m4 Exit: -15 | +-------------------------------------------------------------------------------+ Total duration: 1202.7s step returned non-zero exit code: 241
,
Apr 18 2017
Issue 712512 has been merged into this issue.
,
Apr 18 2017
Issue 712368 has been merged into this issue.
,
Apr 21 2017
Detected 3 new flakes for test/step "ios_net_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFpb3NfbmV0X3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
Apr 21 2017
The same behaviour could be observed for components_unittests (iPad Air 2 iOS 10.0) on chromium.mac/ios-simulator: https://uberchromegw.corp.google.com/i/chromium.mac/builders/ios-simulator/builds/16791 Exactly the same log: https://chromium-swarm.appspot.com/task?id=35a8377904064d10&refresh=10&show_raw=1 It really seems to be an issue of chromium.mac/builders/ios-simulator.
,
Apr 21 2017
,
Apr 27 2017
Detected 13 new flakes for test/step "crypto_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGwsSBUZsYWtlIhBjcnlwdG9fdW5pdHRlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Apr 28 2017
smut: is this an infra bug, or a bug in the test? I'm assigning to pkl in hopes of finding an owner and fixing this as it's contributing to flake on the waterfall.
,
Apr 28 2017
It's not really in the test, but it's also not really in the part of the infrastructure that I own. iossim [1] invocations are hanging. [1] https://chromium.googlesource.com/chromium/src/+/master/testing/iossim/
,
Apr 29 2017
,
May 1 2017
Removing sheriff label since there's nothing for sheriff to do here.
,
May 1 2017
Re: #c9: if it's in between tests and the "infrastructure" (i.e. the machine and related services like swarming), then it's likely a problem with the Apple layer (?). IMHO we (Ops) still should own this, even if it's hard to investigate it at the moment.
,
May 5 2017
Detected 3 new flakes for test/step "ios_net_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFpb3NfbmV0X3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
May 5 2017
,
May 8 2017
Removing from sheriff queue (again)
,
May 10 2017
Detected 4 new flakes for test/step "ios_net_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFpb3NfbmV0X3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
May 11 2017
Removing from sheriff queue (again).
,
May 24 2017
Detected 3 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
May 26 2017
Removing from sheriff queue (again). +Infra can this get looked at?
,
May 26 2017
The root issue (see #0) is not infra, we're shelling out to iossim and it hangs. The issue in #18 seems to be a legitimate failure in PaymentRequestViewControllerTest.TestSignedInStringFormatting, not flake.
,
May 27 2017
Detected 179 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
May 27 2017
Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.
,
May 27 2017
Sergey, can you please find an owner for this and assign this issue to them so that we can stop bugging sheriffs?
,
May 29 2017
Detected 7 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
May 30 2017
Bumping up to P1 as it's frequent.
,
May 30 2017
Issue 725979 has been merged into this issue.
,
May 30 2017
,
May 30 2017
Issue 727491 has been merged into this issue.
,
May 30 2017
Detected 88 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
May 31 2017
Detected 3 new flakes for test/step "ios_net_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFpb3NfbmV0X3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
May 31 2017
Issue 728056 has been merged into this issue.
,
Jun 1 2017
Detected 3 new flakes for test/step "url_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGAsSBUZsYWtlIg11cmxfdW5pdHRlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Jun 2 2017
Detected 45 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 2 2017
Why did we think this was related to iossim? (I now see this error consistently on downstream iOS9 device bots. Ping me for links if needed.) It fails with the same error code: WARNING:root:collect_cmd had non-zero return code: 241 Do we know what that means? I can't find anything specific using code search. I traced it to the result value from vield_results inside swarming.py https://cs.chromium.org/chromium/infra/luci/client/swarming.py?q=swarming.p&l=609 (Web search comes up with a SQL error?)
,
Jun 2 2017
Sorry, you can likely ignore my comment. I was able to repo this locally and will report back if I see anything related to this.
,
Jun 3 2017
Detected 3 new flakes for test/step "url_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGAsSBUZsYWtlIg11cmxfdW5pdHRlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Jun 4 2017
Detected 3 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 5 2017
Detected 4 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 5 2017
CC - liaoyuke@ has been investigating this.
,
Jun 6 2017
Detected 40 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 7 2017
Detected 3 new flakes for test/step "crypto_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGwsSBUZsYWtlIhBjcnlwdG9fdW5pdHRlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Jun 7 2017
Detected 5 new flakes for test/step "boringssl_crypto_tests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIQsSBUZsYWtlIhZib3Jpbmdzc2xfY3J5cHRvX3Rlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Jun 7 2017
Detected 5 new flakes for test/step "ios_net_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFpb3NfbmV0X3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 7 2017
This is currently marked as an "Infra-Troopers" bug, but it has an ownder. Is trooper intervention needed here?
,
Jun 7 2017
No, we have identified the root cause of part of the problems and the fix is on the way. Thanks for checking in!
,
Jun 8 2017
Detected 3 new flakes for test/step "url_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGAsSBUZsYWtlIg11cmxfdW5pdHRlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Jun 9 2017
Detected 29 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 12 2017
Detected 3 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 14 2017
Issue 706464 has been merged into this issue.
,
Jun 14 2017
Issue 730703 has been merged into this issue.
,
Jun 14 2017
,
Jun 14 2017
Issue 713156 has been merged into this issue.
,
Jun 14 2017
,
Jun 16 2017
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 19 2017
Detected 3 new flakes for test/step "boringssl_crypto_tests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIQsSBUZsYWtlIhZib3Jpbmdzc2xfY3J5cHRvX3Rlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Jun 19 2017
Detected 3 new flakes for test/step "google_apis_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIAsSBUZsYWtlIhVnb29nbGVfYXBpc191bml0dGVzdHMM. This message was posted automatically by the chromium-try-flakes app.
,
Jun 20 2017
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 21 2017
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 23 2017
Detected 3 new flakes for test/step "ios_chrome_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRpb3NfY2hyb21lX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 23 2017
,
Jun 27 2017
Detected 3 new flakes for test/step "google_apis_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIAsSBUZsYWtlIhVnb29nbGVfYXBpc191bml0dGVzdHMM. This message was posted automatically by the chromium-try-flakes app.
,
Jun 29 2017
Detected 3 new flakes for test/step "google_apis_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIAsSBUZsYWtlIhVnb29nbGVfYXBpc191bml0dGVzdHMM. This message was posted automatically by the chromium-try-flakes app.
,
Jun 29 2017
Detected 3 new flakes for test/step "gfx_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGAsSBUZsYWtlIg1nZnhfdW5pdHRlc3RzDA. This message was posted automatically by the chromium-try-flakes app.
,
Aug 30 2017
Can we run the following when we hit a suspected timeout? `printf '\n' | xcrun simctl diagnose -b --output=/path/to/foo` And then collect `/path/to/foo.tar.gz` and send it to Apple?
,
Sep 7 2017
It's Apple's bug in their provided tools and now it has been fixed in Xcode 9. Tests are not hanging anymore after switching to Xcode 9! https://build.chromium.org/p/chromium.mac/builders/ios-simulator?numbuilds=200 |
||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||
Comment 1 by s...@google.com
, Apr 7 2017Labels: -Infra-Troopers