New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 913946 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug-Regression



Sign in to add a comment

SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds failing on Mac

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Dec 11

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of fhorschig@google.com

The components_unittests test SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds started failing on : 
- Mac10.12 Tests
- Mac10.13 Tests
- Mac ASan 64 Tests (1)

Latest builds:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20ASan%2064%20Tests%20%281%29/47396
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests/7598
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.12%20Tests/17521

All three failures occured for the first time when https://crrev.com/c/1361863 landed.

Log:
[ RUN      ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds
[3879:775:1211/054135.026607:2584810315049:WARNING:syncer_proto_util.cc(342)] Error posting from syncer: Response Code (bogus on error): -1 Content-Length (bogus on error): -1 Server Status: CONNECTION_UNAVAILABLE
../../components/sync/engine_impl/sync_scheduler_impl_unittest.cc:1780: Failure
Expected equality of these values:
  1
  success_counter.times_called()
    Which is: 0
Stack trace:
0   components_unittests                0x000000010b16b23b testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop(int) + 91
1   components_unittests                0x000000010b16abf9 testing::internal::AssertHelper::operator=(testing::Message const&) const + 89
2   components_unittests                0x000000010a4c5c7d syncer::SyncSchedulerImplTest_ScheduleClearServerData_FailsRetriesSucceeds_Test::TestBody() + 1741

[  FAILED  ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds (71 ms)
[ RUN      ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds
[4746:775:1211/054222.252934:2632037742964:WARNING:syncer_proto_util.cc(342)] Error posting from syncer: Response Code (bogus on error): -1 Content-Length (bogus on error): -1 Server Status: CONNECTION_UNAVAILABLE
../../components/sync/engine_impl/sync_scheduler_impl_unittest.cc:1780: Failure
Expected equality of these values:
  1
  success_counter.times_called()
    Which is: 0
Stack trace:
0   components_unittests                0x000000010b00023b testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop(int) + 91
1   components_unittests                0x000000010afffbf9 testing::internal::AssertHelper::operator=(testing::Message const&) const + 89
2   components_unittests                0x000000010a35ac7d syncer::SyncSchedulerImplTest_ScheduleClearServerData_FailsRetriesSucceeds_Test::TestBody() + 1741

[  FAILED  ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds (101 ms)
[ RUN      ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds
[4756:775:1211/054224.383358:2634168216914:WARNING:syncer_proto_util.cc(342)] Error posting from syncer: Response Code (bogus on error): -1 Content-Length (bogus on error): -1 Server Status: CONNECTION_UNAVAILABLE
../../components/sync/engine_impl/sync_scheduler_impl_unittest.cc:1780: Failure
Expected equality of these values:
  1
  success_counter.times_called()
    Which is: 0
Stack trace:
0   components_unittests                0x000000010645223b testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop(int) + 91
1   components_unittests                0x0000000106451bf9 testing::internal::AssertHelper::operator=(testing::Message const&) const + 89
2   components_unittests                0x00000001057acc7d syncer::SyncSchedulerImplTest_ScheduleClearServerData_FailsRetriesSucceeds_Test::TestBody() + 1741

[  FAILED  ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds (114 ms)
[ RUN      ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds
[4757:775:1211/054224.788026:2634572894413:WARNING:syncer_proto_util.cc(342)] Error posting from syncer: Response Code (bogus on error): -1 Content-Length (bogus on error): -1 Server Status: CONNECTION_UNAVAILABLE
../../components/sync/engine_impl/sync_scheduler_impl_unittest.cc:1780: Failure
Expected equality of these values:
  1
  success_counter.times_called()
    Which is: 0
Stack trace:
0   components_unittests                0x0000000110a0d23b testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop(int) + 91
1   components_unittests                0x0000000110a0cbf9 testing::internal::AssertHelper::operator=(testing::Message const&) const + 89
2   components_unittests                0x000000010fd67c7d syncer::SyncSchedulerImplTest_ScheduleClearServerData_FailsRetriesSucceeds_Test::TestBody() + 1741

[  FAILED  ] SyncSchedulerImplTest.ScheduleClearServerData_FailsRetriesSucceeds (85 ms)
 
Cc: fdoray@chromium.org gab@chromium.org w...@chromium.org
Components: Internals>TaskScheduler
Labels: -Pri-2 OS-Mac Pri-1 Type-Bug-Regression
Owner: alexclarke@chromium.org
Surprisingly, the MSAN bot doesn't show the test failure in the most recent run:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20ASan%2064%20Tests%20%281%29/47397

Hmm, I'll wait for the next non-MSAN build to see whether this was a very unlikely flake before reverting https://crrev.com/c/1361863.

Meanwhile: alexclarke, do you know whether this would make sense to be related to your change?
Project Member

Comment 3 by bugdroid1@chromium.org, Dec 11

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b1525775f8eb45cd60230ff8329e99167011d979

commit b1525775f8eb45cd60230ff8329e99167011d979
Author: Friedrich Horschig [CET] <fhorschig@chromium.org>
Date: Tue Dec 11 14:45:00 2018

Revert "[Reland #3] Use the SequenceManager in ScopedTaskEnvironment"

This reverts commit d1317fc30ce4e9ef08c5197c0f67767dfe585df4.

Reason for revert: SyncSchedulerImplTest is failing and this is the most likely culprit. More information in the linked bug.

Bug: 913946

Original change's description:
> [Reland #3] Use the SequenceManager in ScopedTaskEnvironment
> 
> A reland of https://crrev.com/c/1324391
> 
> This is necessary because we want content::TestBrowserThreadBundle to
> own a BrowserUIThreadScheduler, but that also owns a ScopedTaskEnvironment
> and you can't have two SequenceManagers on the same thread.
> 
> This patch allows ScopedTaskEnvironment to optionally work with an
> externally owned SequenceManager solving the problem.
> 
> This implements https://docs.google.com/document/d/1y08C6JQ9Yta3EQXzwIqqIIKHq9500WV6CWFZzZfDx7I/edit?usp=drivesdk,
> 
> We now have the ability to mock time on the UI and IO threads.
> 
> RE HttpServiceTest change HttpServiceTest.MultipleRequests was failing
> mysteriously on fuschia on the bots only (not locally). IO_MOCK_TIME seems
> to have fixed this.
> 
> 
> TBR=gab@chromium.org,fdoray@chromium.org,wez@chromium.org
> 
> Bug: 863341, 891670, 708584
> Change-Id: I95444a0a50f8b577c5fae62c12c8423c7e8f21d6
> Reviewed-on: https://chromium-review.googlesource.com/c/1361863
> Commit-Queue: Alex Clarke <alexclarke@chromium.org>
> Reviewed-by: Alex Clarke <alexclarke@chromium.org>
> Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#615496}

TBR=wez@chromium.org,gab@chromium.org,fdoray@chromium.org,skyostil@chromium.org,alexclarke@chromium.org

Change-Id: If5ff0362758b5270eb3113d2fd6eeeb7facc37c3
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 863341, 891670, 708584
Reviewed-on: https://chromium-review.googlesource.com/c/1371811
Reviewed-by: Friedrich Horschig [CET] <fhorschig@chromium.org>
Commit-Queue: Friedrich Horschig [CET] <fhorschig@chromium.org>
Cr-Commit-Position: refs/heads/master@{#615520}
[modify] https://crrev.com/b1525775f8eb45cd60230ff8329e99167011d979/base/test/scoped_task_environment.cc
[modify] https://crrev.com/b1525775f8eb45cd60230ff8329e99167011d979/base/test/scoped_task_environment.h
[modify] https://crrev.com/b1525775f8eb45cd60230ff8329e99167011d979/base/test/scoped_task_environment_unittest.cc
[modify] https://crrev.com/b1525775f8eb45cd60230ff8329e99167011d979/webrunner/net_http/http_service_unittest.cc

Could well be related. Unfortunately the Mac & iOS message pumps don't behave the same way the other ones do which is causing us no end of trouble.
 Issue 913957  has been merged into this issue.
FYI: the MSAN builder hasn't shown another failure and Mac 10.13 also cycled green BEFORE the revert landed:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests/7600

If Mac 10.12 does the same, this could really have been an occurrence of three flakes with very extraordinary timing.
Labels: -Sheriff-Chromium
Status: Assigned (was: Available)
Mac 10.12 failed continuously until the revert was in. So the revert seemed to have effect and the original CL might just increase the flakiness a lot.

I did not notice any other test that broke by the original CL and cycled green with the revert. And neither did findit. Removing from sheriff queue.
Project Member

Comment 8 by bugdroid1@chromium.org, Dec 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2ded172c0daa913d3d6e0b118d86ed38210cf06f

commit 2ded172c0daa913d3d6e0b118d86ed38210cf06f
Author: Alex Clarke <alexclarke@chromium.org>
Date: Thu Dec 13 12:55:20 2018

[Reland #4] Use the SequenceManager in ScopedTaskEnvironment

A reland of https://crrev.com/c/1324391

This is necessary because we want content::TestBrowserThreadBundle to
own a BrowserUIThreadScheduler, but that also owns a ScopedTaskEnvironment
and you can't have two SequenceManagers on the same thread.

This patch allows ScopedTaskEnvironment to optionally work with an
externally owned SequenceManager solving the problem.

This implements https://docs.google.com/document/d/1y08C6JQ9Yta3EQXzwIqqIIKHq9500WV6CWFZzZfDx7I/edit?usp=drivesdk,

We now have the ability to mock time on the UI and IO threads.

HttpServiceTest had been changed because HttpServiceTest.MultipleRequests was
failing mysteriously on fuchsia on the bots only (not locally). IO_MOCK_TIME
seems to have fixed this.

SyncSchedulerImplTest has been changed to use MOCK_TIME because it was
flaky on Mac. This change required base::TimeTicks::Now() to be overridden.
These tests should now run faster which I understand was a historical problem
(see  https://crbug.com/402212 ).

The chromeos test changes are needed because C++ requires a default constructor
to const construct an object.  ScopedTaskEnvironment has a template vararg
constructor which due to a defect in the C++ standard isn't recognized for this
purpose.  C++ will happily let you default construct a non-const object using
it however.

TBR=khorimoto@chromium.org, wez@chromium.org, jkrcal@chromium.org

Bug: 863341, 891670, 913946, 911928
Change-Id: Ib7e58223b531cf85aded11b75bd09c47a0993391
Reviewed-on: https://chromium-review.googlesource.com/c/1371899
Reviewed-by: Alex Clarke <alexclarke@chromium.org>
Reviewed-by: Jan Krcal <jkrcal@chromium.org>
Reviewed-by: Gabriel Charette <gab@chromium.org>
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Commit-Queue: Alex Clarke <alexclarke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616280}
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/base/test/scoped_task_environment.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/base/test/scoped_task_environment.h
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/base/test/scoped_task_environment_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/nearby/count_down_latch_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/nearby/settable_future_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/tether/asynchronous_shutdown_object_container_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/tether/connection_preserver_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/tether/crash_recovery_manager_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/tether/host_scanner_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/components/tether/wifi_hotspot_disconnector_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/multidevice_setup/account_status_change_delegate_notifier_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/multidevice_setup/multidevice_setup_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/multidevice_setup/multidevice_setup_service_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/multidevice_setup/privileged_host_device_setter_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/active_connection_manager_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/authenticated_channel_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/ble_advertiser_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/ble_listener_operation_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/ble_synchronizer_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/client_connection_parameters_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/connect_to_device_operation_base_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/connection_attempt_base_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/multiplexed_channel_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/pending_ble_initiator_connection_request_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/pending_ble_listener_connection_request_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/pending_connection_manager_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/pending_connection_request_base_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/public/cpp/client/client_channel_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/public/cpp/client/secure_channel_client_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/secure_channel_service_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/chromeos/services/secure_channel/single_client_message_proxy_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/components/sync/engine_impl/sync_scheduler_impl_unittest.cc
[modify] https://crrev.com/2ded172c0daa913d3d6e0b118d86ed38210cf06f/webrunner/net_http/http_service_unittest.cc

Sign in to add a comment