Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Issue 658619 "components_unittests" is flaky
Starred by 2 users Project Member Reported by chromium...@appspot.gserviceaccount.com, Oct 23 Back to list
Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 1
Type: Bug


Sign in to add a comment
"components_unittests" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww.



This flaky test/step was previously tracked in issue 637193.
 
Owner: smut@chromium.org
@smut is this the same issue of https://bugs.chromium.org/p/chromium/issues/detail?id=637139
Cc: baxley@chromium.org sergeybe...@chromium.org smut@chromium.org kkhorimoto@chromium.org sdefresne@chromium.org justincohen@chromium.org rohitrao@chromium.org
Labels: OS-iOS
Owner: ----
No it's a different issue. The failures appear to be actual test cases failing:
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/builds/92735
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/builds/92702
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/builds/92027

+cc Chrome for iOS eng for triage.
Owner: baxley@chromium.org
Status: Assigned
Who should triage these components_unittests failures?
Tests that have failed recently:

NetworkTimeTrackerTest.UpdateFromNetworkNetworkError (twice)
SyncBackendHostTest.FirstTimeSync
DataReductionProxySettingsTest.TestDaysSinceEnabled
DataReductionProxyInterceptorWithServerTest.TestBypass
DataReductionProxyInterceptorWithServerTest.TestNoBypass

Owner: pkl@chromium.org
Was just entering a comment, but dmazzoni beat me to it.
The three failure runs linked above, are from different tests...
SyncBackendHostTest.FirstTimeSync
DataReductionProxySettingsTest.TestDaysSinceEnabled 
NetworkTimeTrackerTest.UpdateFromNetworkNetworkError

pkl@ - these tests are still running, so what is our triage process for cases like this? We've had process for disabled tests, but these aren't disabled.
Are these tests (3 of 4 anyway) even relevant for iOS? We don't even support DataReductionProxy.

Re #6: We should probably stop running data_reduction_proxy unit tests on iOS then.
Project Member Comment 8 by bugdroid1@chromium.org, Oct 25
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/59da38fdd532f99ab0f9020f861da209178ae334

commit 59da38fdd532f99ab0f9020f861da209178ae334
Author: sdefresne <sdefresne@chromium.org>
Date: Tue Oct 25 19:04:44 2016

Disable data_reduction_proxy unit tests on iOS.

With the switch to WKWebView, Chrome on iOS code cannot use data
reduction proxy. Disable the tests on iOS as they are flaky and
make components_unittests falky on iOS.

BUG=658619

Review-Url: https://codereview.chromium.org/2450003002
Cr-Commit-Position: refs/heads/master@{#427432}

[modify] https://crrev.com/59da38fdd532f99ab0f9020f861da209178ae334/components/BUILD.gn

Does that cover all of the tests? Should we disable the remaining ones?

No, this only cover the DataReductionProxy tests.

We should probably not disable the other tests as they test features that are used by Chrome on iOS.
OK, disabling these tests:

SyncBackendHostTest.FirstTimeSync
NetworkTimeTrackerTest.UpdateFromNetworkNetworkError


Project Member Comment 12 by bugdroid1@chromium.org, Oct 25
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/05431a027ee7ee6ea854c011347c4250d1b40783

commit 05431a027ee7ee6ea854c011347c4250d1b40783
Author: dmazzoni <dmazzoni@chromium.org>
Date: Tue Oct 25 22:14:54 2016

Disable two components_unittests that fail on ios

SyncBackendHostTest.FirstTimeSync
NetworkTimeTrackerTest.UpdateFromNetworkNetworkError

BUG=658619
TBR=pkl@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2449853005
Cr-Commit-Position: refs/heads/master@{#427507}

[modify] https://crrev.com/05431a027ee7ee6ea854c011347c4250d1b40783/components/network_time/network_time_tracker_unittest.cc
[modify] https://crrev.com/05431a027ee7ee6ea854c011347c4250d1b40783/components/sync/driver/glue/sync_backend_host_impl_unittest.cc

Labels: -Sheriff-Chromium
No further flakes since the tests were disabled, so dropping Sheriff-Chromium label and trusting that this'll get resolved. CC zea@chromium.org who is in OWNERS for both affected tests.
Cc: zea@chromium.org
zea: Can someone on the sync team take this instead?
(really cc'ing zea@)
Blockedon: 659705 659706
Filed bug 659705 for the Sync side, and bug 659706 for the network time side.
Cc: pkl@chromium.org
Owner: ----
Status: Available
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Blockedon: 661144
Filed issue 661144 for a flaky component_unittests.
Labels: -Sheriff-Chromium
Labels: Sheriff-Chromium
Detected 6 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
Labels: Sheriff-Chromium
Detected 5 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
RequestCoordinatorTest.PauseAndResumeObserver on IOS.
Labels: -Sheriff-Chromium
Filed https://bugs.chromium.org/p/chromium/issues/detail?id=663311 for RequestCoordinatorTest.PauseAndResumeObserver
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Blockedon: 682091
Labels: -Sheriff-Chromium
Seems to be timing out now.

Filed https://bugs.chromium.org/p/chromium/issues/detail?id=682091
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Cc: stale-flakes-reports@google.com
Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.
Cc: -stale-flakes-reports@google.com
chromium-try-flakes is able to file bugs for individual tests, but for this the components_unittests test launcher needs to upload results in a JSON format to test-results.appspot.com. The format is documented here: https://www.chromium.org/developers/the-json-test-results-format. Furthermore, looks like components_unittests are GTest-based and we do have code converting GTest JSON results output to the format needed by the test-results app.

Sana, would you happen know who is the owner for this test launcher?
It's me, but we have a bunch of components_unittests timeouts which wouldn't result in anything uploaded to test-results.appspot.com anyways.
If it's whole launcher timeout, then I also think this bug is the correct place to report it, but I see a bunch of comments above where people have been disabling/fixing specific flaky tests. Sheriffs are typically unhappy when such an issue keeps returning to the queue since the comment history gets long and it becomes harder and harder for the sheriff to find all the information related to the actual recent flakes. If, however, individual issues were filed for each flaky test, then it would be much easier to follow the context.
Labels: -Sheriff-Chromium
(Last flakes were 4 days ago; removing sheriff label again.)
Just discovered an example of sheriffs not being happy about bugs files for entire test suites: http://crbug.com/682496.
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
Can somebody assign this to himself so we can avoid getting the Sheriff-Chromium readded over and over? (at least I think that makes it stop).
There's 6040 tests in components_unittests and they are owned by 265 different OWNERs (the number of different OWNERS for //components).

It looks like the tests that are failing are different each time (at least each time this bug is pinged). I think that chromium-try-flakes should try to use individual bugs for each individual test and then we'll be able to find specific OWNERS to look at them. As it is, this notification is sadly mostly noise (too many tests, too many potential OWNERS).
Sylvain, please see comment #29 above. We need to find an OWNER who will implement uploading to test-results server and then chromium-try-flakes will be able to file bugs for individual tests. I'd be happy to assist explaining the format and which endpoints to use, but would need help from someone who actually knows how this test suite runs.
You'll want to ask sergeyberezin@ or smut@ then.
Starting last week, we are uploading test results for components_unittests, but only for iOS, not for other platforms.

Examples are at http://test-results.appspot.com/testfile?builder=ios-simulator-iPhone%206s-10.0&master=chromium.mac&testtype=components_unittests

What else needs to be changed to have the bot file bugs for individual tests?
Ah. That's interesting bit of information. That's probably because you are using a flake builder name... and chromium-try-flakes is looking for "ios-simulator".
s/flake builder name/fake builder name/
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
Blockedon: 690607
Labels: Sheriff-Chromium
Detected 10 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Project Member Comment 48 by bugdroid1@chromium.org, Feb 9
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f8458d55b3619c6820b9095adf63d8fa2aa7e17b

commit f8458d55b3619c6820b9095adf63d8fa2aa7e17b
Author: bsep <bsep@chromium.org>
Date: Thu Feb 09 23:15:21 2017

Disable JsTranslateManagerTest.PerformancePlaceholder due to flakiness.

TBR=droger@chromium.org
BUG=658619

Review-Url: https://codereview.chromium.org/2687193002
Cr-Commit-Position: refs/heads/master@{#449454}

[modify] https://crrev.com/f8458d55b3619c6820b9095adf63d8fa2aa7e17b/components/translate/ios/browser/js_translate_manager_unittest.mm

Labels: -Sheriff-Chromium
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Cc: stale-flakes-reports@google.com
Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
Cc: -stale-flakes-reports@google.com dpranke@chromium.org
Update: the reason why iOS team has decided to use a fake builder name is because there is no way for them to distinguish tests run on different platforms as platform name is stripped away during the step name normalization. I wrote a proposal to add "config" field into test results format to address use-cases like this, but it may take a while before this is implemented.

Having said that, besides iOS, there are many other builders that are running component_unittests. We need to find owner(s) who will implement uploading of test results for those builders as well. They don't have to do it immediately, but having them assigned to this issue will stop spamming sheriffs.

+dpranke, can you please help to find an owner for this?
Cc: estaab@chromium.org
components_unittests have been uploading flaky test results on non-iOS platforms for a long, long time. I don't think that's the problem here?

buildername should be `ios-simulator`, not something fake.

However, it is a problem that we might have the same buildername on multiple waterfalls, and also a problem that we might run the same step name multiple times in different configurations on the same builder.

We should add `mastername` to the JSON results format to handle the former case.

We should probably add another field to handle the additional case, something like `test_configuration` or something. We could change the test/step name from "components_unittests" to "components_unittests (iPhone 5 iOS 10.0)" but I think we know that there are other places in the data pipelines that don't handle a combinatorial explosion of step names, and so it's probably better to break things up. But that might require more work in the flakiness tooling, too.
Sergiy put together some notes on adding configuration support (internal only):
https://docs.google.com/document/d/1rFS9r9dIgTOgHnm84vtAxMtDAUZG1KfHk4Mle1ssS90/edit?ts=58a17b80
Labels: -Sheriff-Chromium
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
> components_unittests have been uploading flaky test results on non-iOS platforms for a long, long time. I don't think that's the problem here?

Indeed, I can see some results here: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=components_unittests%20(with%20patch). I am somewhat surprised that there are just a handful of tests in the list. By looking at https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/120432/steps/components_unittests%20%28with%20patch%29%20on%20Android/logs/stdio, I think many more tests should be reported.

> buildername should be `ios-simulator`, not something fake.

There is no other short-term way to differentiate results from different steps running on different platforms. This happens because any platform information is stripped away from the step name during normalization, e.g. "component_unittests (iPhone 5 iOS 10.0)" becomes "component_unittests". Without placing platform information into the builder name, each step run would be overriding results from other steps. Step normalization was introduced to get rid of non-consistent bits of information that developers (including me) were adding to the step name, e.g. number of seconds a step was waiting as pending on swarming.

I wasn't happy with fake builder names either because it broke tools like chromium-try-flakes and links on Flakiness Dashboard, but due to lack of a better alternative, I've accepted this as a reasonable short-term solution.

> However, it is a problem that we might have the same buildername on multiple waterfalls, and also a problem that we might run the same step name multiple times in different configurations on the same builder.
> We should add `mastername` to the JSON results format to handle the former case.

We do not have master in the JSON format (not sure why?), but scripts do have to specify it when uploading, therefore it should not be an issue. What is called "builder" in Flakiness Dashboard is actually "master:build".

> We should probably add another field to handle the additional case, something like `test_configuration` or something.

Yes, that's my proposal that Erik has linked.

> We could change the test/step name from "components_unittests" to "components_unittests (iPhone 5 iOS 10.0)" but I think we know that there are other places in the data pipelines that don't handle a combinatorial explosion of step names, and so it's probably better to break things up. But that might require more work in the flakiness tooling, too.

As mentioned above, this wouldn't work since step normalization would strip this information away.
Speaking of the broken links on Flakiness Dashboard, see issue 691931.
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Detected 6 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
Labels: -Sheriff-Chromium
As far as I can tell, there's nothing for the sheriffs to do here right now, so removing the sheriff label.
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
I'm not really sure what I can do as a sheriff and I noticed that the previous sheriff removed the sheriff label so I'll do the same...if this is wrong, please let me know...
Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Detected 4 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.
Owner: olivierrobin@chromium.org
Status: Assigned
FromReadingListLocal seems flaky:
https://chromium-swarm.appspot.com/task?id=349c311d02088710
https://chromium-swarm.appspot.com/task?id=349f2a754b133510

[ RUN      ] ReadingListEntry.FromReadingListLocal
../../components/reading_list/ios/reading_list_entry_unittest.cc:355: Failure
The difference between delta.InMillisecondsRoundedUp() and 0 is 16, which exceeds 10, where
delta.InMillisecondsRoundedUp() evaluates to -16,
0 evaluates to 0, and
10 evaluates to 10.
[  FAILED  ] ReadingListEntry.FromReadingListLocal (28 ms)

[ RUN      ] ReadingListEntry.FromReadingListLocal
../../components/reading_list/ios/reading_list_entry_unittest.cc:355: Failure
The difference between delta.InMillisecondsRoundedUp() and 0 is 12, which exceeds 10, where
delta.InMillisecondsRoundedUp() evaluates to -12,
0 evaluates to 0, and
10 evaluates to 10.
[  FAILED  ] ReadingListEntry.FromReadingListLocal (31 ms)
Labels: -Sheriff-Chromium
Removing Sheriff-Chromium since this has an owner.
Status: Started
Project Member Comment 72 by bugdroid1@chromium.org, Mar 1
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9d86f02c7bc5dd777badfb7c1e675a87e434e5d6

commit 9d86f02c7bc5dd777badfb7c1e675a87e434e5d6
Author: olivierrobin <olivierrobin@chromium.org>
Date: Wed Mar 01 11:04:28 2017

Fix ReadingListEntry flaky unittests.

BUG=658619

Review-Url: https://codereview.chromium.org/2722163002
Cr-Commit-Position: refs/heads/master@{#453902}

[modify] https://crrev.com/9d86f02c7bc5dd777badfb7c1e675a87e434e5d6/components/reading_list/ios/reading_list_entry_unittest.cc

Sign in to add a comment