New issue
Advanced search Search tips

Issue 786965 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 2
Type: Bug



Sign in to add a comment

ios-simulator-full-configs green while some tests fails

Project Member Reported by sdefresne@chromium.org, Nov 20 2017

Issue description

See for example this run:
https://build.chromium.org/p/chromium.mac/builders/ios-simulator-full-configs/builds/663

CredentialManagerEGTest/* tests from ios_chrome_integration_egtests have failed, but ios_chrome_integration_egtests is still green and whole test run is considered as a success; downstream bots are correctly marked as failed, compare with:

https://uberchromegw.corp.google.com/i/internal.bling.main/builders/iphone11-device-x64/builds/1179



 

Comment 1 by baxley@chromium.org, Nov 20 2017

This is because retry-on-failure is enabled upstream. This is used for most upstream test suites.

So the fact that it's green, isn't a bug. However, it would be far better to not rerun tests on failure. Unfortunately I don't think our tests/code/infrastructure is reliable enough to simply turn this on (see recent chromium-dev thread about flake and retries). It also happens on ios-simulator with unit tests:
https://build.chromium.org/p/chromium.mac/builders/ios-simulator/builds/25899/steps/ios_chrome_unittests%20%28iPhone%206s%20Plus%20iOS%2011.0%29

Here are a couple of options - I think we should go for #2, though it's the harder one.
1. Rerun failed tests downstream - this will make it consistent, but mask failures.
2. Turn off retry-on-failure upstream - this is a much harder thing to accomplish, but is the direction I think we should go in.
Here retry-on-failure really mess up because the test only fail if run as part of the full suite. If run individually (or skipping all tests coming before) they pass. I think this is because they depend on some global state that is messed up by another test run before :-/

Comment 3 by pkl@chromium.org, Nov 20 2017

Components: Tests
Labels: -Pri-1 Pri-2
Status: Assigned (was: Untriaged)
huangml: Please reassign if this isn't yours.
I think CredentialManagerEGTest fails because "--enable-features=SlimNavigationManager" is not passed in to integration_egtest suite. Same test with the flag is working fine on chromium.fyi.  But not sure why they'll pass on retries.
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.fyi%2Fios-simulator%2F13049%2F%2B%2Frecipes%2Fsteps%2Fios_chrome_integration_egtests__iPhone_6s_iOS_11.0_%2F0%2Fstdout
Can it be related to issue 788201's root cause (yet to be determined)?
Cc: -baxley@chromium.org
Owner: ----
Status: Available (was: Assigned)

Sign in to add a comment