New issue
Advanced search Search tips

Issue 815189 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 2
Type: Bug



Sign in to add a comment

Some tests failing due to testing::WaitUntilConditionOrTimeout timeouts

Project Member Reported by sdefresne@chromium.org, Feb 23 2018

Issue description

Some tests started to fail in the last couple of days due to timeout in testing::WaitUntilConditionOrTimeout. This is not reproducible at 100% on bots, but on developer machines it seems to trigger pretty reliably.

One such failure on bot is here:

https://ci.chromium.org/buildbot/tryserver.chromium.mac/ios-simulator/401854

The bot is green, but tests have failed (this may be why this has not been reported). The test fails with the following error:

[ RUN      ] SyncCreatePassphraseCollectionViewControllerTest.TestOnStateChanged
../../ios/chrome/browser/ui/settings/sync_create_passphrase_collection_view_controller_unittest.mm:192: Failure
Value of: testing::WaitUntilConditionOrTimeout( testing::kWaitForUIElementTimeout, ^bool() { return [nav_controller_ topViewController] != sync_controller; })
  Actual: false
Expected: true
[  FAILED  ] SyncCreatePassphraseCollectionViewControllerTest.TestOnStateChanged (17 ms)

Some other tests fails when they try to inject load some inline HTML. This causes a recursive stack overflow (or at least a really deep stack), and was reported as https://bugs.chromium.org/p/chromium/issues/detail?id=814709, however, it is believed to just be another manifestation of the same issue as the logs presents the following messages (see the same bot):

[ RUN      ] TextToSpeechListenerTest.ValidAudioDataTest
../../ios/web/public/test/web_test_with_web_state.mm:194: Failure
Value of: WaitUntilConditionOrTimeout(kWaitForJSCompletionTimeout, ^{ return execution_completed; })
  Actual: false
Expected: true
Google Test trace:
../../ios/web/public/test/web_test_with_web_state.mm:180: 0;
../../ios/web/public/test/web_test_with_web_state.mm:194: Failure
Value of: WaitUntilConditionOrTimeout(kWaitForJSCompletionTimeout, ^{ return execution_completed; })
  Actual: false
Expected: true
Google Test trace:
../../ios/web/public/test/web_test_with_web_state.mm:180: 0;
../../ios/web/public/test/web_test_with_web_state.mm:194: Failure
Value of: WaitUntilConditionOrTimeout(kWaitForJSCompletionTimeout, ^{ return execution_completed; })
  Actual: false
Expected: true
Google Test trace:
../../ios/web/public/test/web_test_with_web_state.mm:180: 0;
../../ios/web/public/test/web_test_with_web_state.mm:194: Failure
Value of: WaitUntilConditionOrTimeout(kWaitForJSCompletionTimeout, ^{ return execution_completed; })
  Actual: false
Expected: true
...
[  FAILED  ] TextToSpeechListenerTest.ValidAudioDataTest (103642 ms)

I don't see recent changes in those bots, so I'm wondering whether this may be due to a change in scheduling in //base. Can sherrif try to bisect and find when this error started to occur?


 
This appears to have broken in https://ci.chromium.org/buildbot/chromium.mac/ios-simulator/29858, according to the bot logs.
Status: Started (was: Assigned)
I bet this is https://chromium-review.googlesource.com/c/chromium/src/+/924759. 

If we mock out NSDate but don't put the original implementation back, then WaitUntilConditionOrTimeout() could end up failing because it's trying to call those NSDate APIs.

I can reproduce locally by running the entire ios_chrome_unittests test suite, or by passing --gtest_filter=ReauthenticationModuleTest*:TextToSpeechListenerTest*.

The failing tests pass on retry because they're run by themselves in a separate process.

The OCMock docs say that you have to dealloc the mock object in order to restore the original implementations.  I bet that's not happening because the mock is being autoreleased but we're not subclassing PlatformTest to give ourselves an autorelease pool.
Cc: ioanap@chromium.org sdefresne@chromium.org
FYI this was pretty easy to track down because:

1) The flakiness dashboard showed that some tests were failing 100% of the time.
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=ios_chrome_unittests&showExpectations=true&tests=TextToSpeechListenerTest.ValidAudioDataTest

2) From there it was easy enough to go back through bot logs to find the first run in which these tests started failing.
https://ci.chromium.org/buildbot/chromium.mac/ios-simulator/29858

3) And then go through the set of commits looking for anything that might be related.


Kudos to sdefresne for noticing this.  If we had sat on it for a week, the bot logs might have been deleted and then it would've been a pain to track down.
Looks like we should revert that CL and investigate how we can safely decouple the implementation from the real NSDate (maybe through injection) instead of relying on mocking some class methods.
Subclassing PlatformTest fixes the issue, so that's what I've done in https://chromium-review.googlesource.com/c/chromium/src/+/936222.

Further decoupling from NSDate seems like a reasonable idea, but it's not urgent anymore.
Project Member

Comment 6 by bugdroid1@chromium.org, Feb 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd061a082fcf108b2fb2898886a6155cda2f02b2

commit bd061a082fcf108b2fb2898886a6155cda2f02b2
Author: Rohit Rao <rohitrao@chromium.org>
Date: Sun Feb 25 21:11:37 2018

[ios] Makes ReauthenticationModuleTest inherit from PlatformTest.

These tests use OCMClassMock() to mock out an NSDate class method.  The
OCMock documentation states that behavior is undefined if multiple
objects try to mock the same class simultaneously.  The documentation
also states that the mock object must be deallocated in order to restore
the original method implementations.

I suspect that the autorelease pool is never being drained and therefore
the mock object stays alive past the end of these tests.  This is
calling subsequent tests to fail, because NSDate no longer returns the
correct date.

Inheriting from PlatformTest wraps each test in an autorelease pool,
which appears to be enough to get the tests passing again.

BUG= 815189 
TEST=TextToSpeechListenerTest.ValidAudioDataTest passes on the bots.

Cq-Include-Trybots: master.tryserver.chromium.mac:ios-simulator-cronet;master.tryserver.chromium.mac:ios-simulator-full-configs
Change-Id: I0d8383b0d1f476ff2ab9840fff384f7b4aeb2dc3
Reviewed-on: https://chromium-review.googlesource.com/936222
Reviewed-by: Justin Cohen <justincohen@chromium.org>
Commit-Queue: Rohit Rao <rohitrao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#539050}
[modify] https://crrev.com/bd061a082fcf108b2fb2898886a6155cda2f02b2/ios/chrome/browser/ui/settings/reauthentication_module_unittest.mm

I remember adding a presubmit check to fix this in the downstream tree. Found out that it was reverted, so let's add a check upstream then.
Status: Fixed (was: Started)

Sign in to add a comment