@smut is this the same issue of https://bugs.chromium.org/p/chromium/issues/detail?id=637139

No it's a different issue. The failures appear to be actual test cases failing:
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/builds/92735
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/builds/92702
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/builds/92027

+cc Chrome for iOS eng for triage.

Who should triage these components_unittests failures?

Tests that have failed recently:

NetworkTimeTrackerTest.UpdateFromNetworkNetworkError (twice)
SyncBackendHostTest.FirstTimeSync
DataReductionProxySettingsTest.TestDaysSinceEnabled
DataReductionProxyInterceptorWithServerTest.TestBypass
DataReductionProxyInterceptorWithServerTest.TestNoBypass

Was just entering a comment, but dmazzoni beat me to it.
The three failure runs linked above, are from different tests...
SyncBackendHostTest.FirstTimeSync
DataReductionProxySettingsTest.TestDaysSinceEnabled 
NetworkTimeTrackerTest.UpdateFromNetworkNetworkError

pkl@ - these tests are still running, so what is our triage process for cases like this? We've had process for disabled tests, but these aren't disabled.

Are these tests (3 of 4 anyway) even relevant for iOS? We don't even support DataReductionProxy.

Re #6: We should probably stop running data_reduction_proxy unit tests on iOS then.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/59da38fdd532f99ab0f9020f861da209178ae334

commit 59da38fdd532f99ab0f9020f861da209178ae334
Author: sdefresne <sdefresne@chromium.org>
Date: Tue Oct 25 19:04:44 2016

Disable data_reduction_proxy unit tests on iOS.

With the switch to WKWebView, Chrome on iOS code cannot use data
reduction proxy. Disable the tests on iOS as they are flaky and
make components_unittests falky on iOS.

BUG= 658619 

Review-Url: https://codereview.chromium.org/2450003002
Cr-Commit-Position: refs/heads/master@{#427432}

[modify] https://crrev.com/59da38fdd532f99ab0f9020f861da209178ae334/components/BUILD.gn

Does that cover all of the tests? Should we disable the remaining ones?

No, this only cover the DataReductionProxy tests.

We should probably not disable the other tests as they test features that are used by Chrome on iOS.

OK, disabling these tests:

SyncBackendHostTest.FirstTimeSync
NetworkTimeTrackerTest.UpdateFromNetworkNetworkError

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/05431a027ee7ee6ea854c011347c4250d1b40783

commit 05431a027ee7ee6ea854c011347c4250d1b40783
Author: dmazzoni <dmazzoni@chromium.org>
Date: Tue Oct 25 22:14:54 2016

Disable two components_unittests that fail on ios

SyncBackendHostTest.FirstTimeSync
NetworkTimeTrackerTest.UpdateFromNetworkNetworkError

BUG= 658619 
TBR=pkl@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2449853005
Cr-Commit-Position: refs/heads/master@{#427507}

[modify] https://crrev.com/05431a027ee7ee6ea854c011347c4250d1b40783/components/network_time/network_time_tracker_unittest.cc
[modify] https://crrev.com/05431a027ee7ee6ea854c011347c4250d1b40783/components/sync/driver/glue/sync_backend_host_impl_unittest.cc

No further flakes since the tests were disabled, so dropping Sheriff-Chromium label and trusting that this'll get resolved. CC zea@chromium.org who is in OWNERS for both affected tests.

zea: Can someone on the sync team take this instead?
(really cc'ing zea@)

Filed  bug 659705  for the Sync side, and bug 659706 for the network time side.

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Filed  issue 661144  for a flaky component_unittests.

Detected 6 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 5 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

RequestCoordinatorTest.PauseAndResumeObserver on IOS.

Filed https://bugs.chromium.org/p/chromium/issues/detail?id=663311 for RequestCoordinatorTest.PauseAndResumeObserver

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Seems to be timing out now.

Filed https://bugs.chromium.org/p/chromium/issues/detail?id=682091

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.

chromium-try-flakes is able to file bugs for individual tests, but for this the components_unittests test launcher needs to upload results in a JSON format to test-results.appspot.com. The format is documented here: https://www.chromium.org/developers/the-json-test-results-format. Furthermore, looks like components_unittests are GTest-based and we do have code converting GTest JSON results output to the format needed by the test-results app.

Sana, would you happen know who is the owner for this test launcher?

It's me, but we have a bunch of components_unittests timeouts which wouldn't result in anything uploaded to test-results.appspot.com anyways.

If it's whole launcher timeout, then I also think this bug is the correct place to report it, but I see a bunch of comments above where people have been disabling/fixing specific flaky tests. Sheriffs are typically unhappy when such an issue keeps returning to the queue since the comment history gets long and it becomes harder and harder for the sheriff to find all the information related to the actual recent flakes. If, however, individual issues were filed for each flaky test, then it would be much easier to follow the context.

(Last flakes were 4 days ago; removing sheriff label again.)

Just discovered an example of sheriffs not being happy about bugs files for entire test suites:  http://crbug.com/682496 .

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Can somebody assign this to himself so we can avoid getting the Sheriff-Chromium readded over and over? (at least I think that makes it stop).

There's 6040 tests in components_unittests and they are owned by 265 different OWNERs (the number of different OWNERS for //components).

It looks like the tests that are failing are different each time (at least each time this bug is pinged). I think that chromium-try-flakes should try to use individual bugs for each individual test and then we'll be able to find specific OWNERS to look at them. As it is, this notification is sadly mostly noise (too many tests, too many potential OWNERS).

Sylvain, please see comment #29 above. We need to find an OWNER who will implement uploading to test-results server and then chromium-try-flakes will be able to file bugs for individual tests. I'd be happy to assist explaining the format and which endpoints to use, but would need help from someone who actually knows how this test suite runs.

You'll want to ask sergeyberezin@ or smut@ then.

Starting last week, we are uploading test results for components_unittests, but only for iOS, not for other platforms.

Examples are at http://test-results.appspot.com/testfile?builder=ios-simulator-iPhone%206s-10.0&master=chromium.mac&testtype=components_unittests

What else needs to be changed to have the bot file bugs for individual tests?

Ah. That's interesting bit of information. That's probably because you are using a flake builder name... and chromium-try-flakes is looking for "ios-simulator".

s/flake builder name/fake builder name/

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 10 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f8458d55b3619c6820b9095adf63d8fa2aa7e17b

commit f8458d55b3619c6820b9095adf63d8fa2aa7e17b
Author: bsep <bsep@chromium.org>
Date: Thu Feb 09 23:15:21 2017

Disable JsTranslateManagerTest.PerformancePlaceholder due to flakiness.

TBR=droger@chromium.org
BUG= 658619 

Review-Url: https://codereview.chromium.org/2687193002
Cr-Commit-Position: refs/heads/master@{#449454}

[modify] https://crrev.com/f8458d55b3619c6820b9095adf63d8fa2aa7e17b/components/translate/ios/browser/js_translate_manager_unittest.mm

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.

Update: the reason why iOS team has decided to use a fake builder name is because there is no way for them to distinguish tests run on different platforms as platform name is stripped away during the step name normalization. I wrote a proposal to add "config" field into test results format to address use-cases like this, but it may take a while before this is implemented.

Having said that, besides iOS, there are many other builders that are running component_unittests. We need to find owner(s) who will implement uploading of test results for those builders as well. They don't have to do it immediately, but having them assigned to this issue will stop spamming sheriffs.

+dpranke, can you please help to find an owner for this?

components_unittests have been uploading flaky test results on non-iOS platforms for a long, long time. I don't think that's the problem here?

buildername should be `ios-simulator`, not something fake.

However, it is a problem that we might have the same buildername on multiple waterfalls, and also a problem that we might run the same step name multiple times in different configurations on the same builder.

We should add `mastername` to the JSON results format to handle the former case.

We should probably add another field to handle the additional case, something like `test_configuration` or something. We could change the test/step name from "components_unittests" to "components_unittests (iPhone 5 iOS 10.0)" but I think we know that there are other places in the data pipelines that don't handle a combinatorial explosion of step names, and so it's probably better to break things up. But that might require more work in the flakiness tooling, too.

Sergiy put together some notes on adding configuration support (internal only):
https://docs.google.com/document/d/1rFS9r9dIgTOgHnm84vtAxMtDAUZG1KfHk4Mle1ssS90/edit?ts=58a17b80

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

> components_unittests have been uploading flaky test results on non-iOS platforms for a long, long time. I don't think that's the problem here?

Indeed, I can see some results here: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=components_unittests%20(with%20patch). I am somewhat surprised that there are just a handful of tests in the list. By looking at https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/120432/steps/components_unittests%20%28with%20patch%29%20on%20Android/logs/stdio, I think many more tests should be reported.

> buildername should be `ios-simulator`, not something fake.

There is no other short-term way to differentiate results from different steps running on different platforms. This happens because any platform information is stripped away from the step name during normalization, e.g. "component_unittests (iPhone 5 iOS 10.0)" becomes "component_unittests". Without placing platform information into the builder name, each step run would be overriding results from other steps. Step normalization was introduced to get rid of non-consistent bits of information that developers (including me) were adding to the step name, e.g. number of seconds a step was waiting as pending on swarming.

I wasn't happy with fake builder names either because it broke tools like chromium-try-flakes and links on Flakiness Dashboard, but due to lack of a better alternative, I've accepted this as a reasonable short-term solution.

> However, it is a problem that we might have the same buildername on multiple waterfalls, and also a problem that we might run the same step name multiple times in different configurations on the same builder.
> We should add `mastername` to the JSON results format to handle the former case.

We do not have master in the JSON format (not sure why?), but scripts do have to specify it when uploading, therefore it should not be an issue. What is called "builder" in Flakiness Dashboard is actually "master:build".

> We should probably add another field to handle the additional case, something like `test_configuration` or something.

Yes, that's my proposal that Erik has linked.

> We could change the test/step name from "components_unittests" to "components_unittests (iPhone 5 iOS 10.0)" but I think we know that there are other places in the data pipelines that don't handle a combinatorial explosion of step names, and so it's probably better to break things up. But that might require more work in the flakiness tooling, too.

As mentioned above, this wouldn't work since step normalization would strip this information away.

Speaking of the broken links on Flakiness Dashboard, see issue 691931.

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 6 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.

As far as I can tell, there's nothing for the sheriffs to do here right now, so removing the sheriff label.

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

I'm not really sure what I can do as a sheriff and I noticed that the previous sheriff removed the sheriff label so I'll do the same...if this is wrong, please let me know...

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 4 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.

FromReadingListLocal seems flaky:
https://chromium-swarm.appspot.com/task?id=349c311d02088710
https://chromium-swarm.appspot.com/task?id=349f2a754b133510

[ RUN      ] ReadingListEntry.FromReadingListLocal
../../components/reading_list/ios/reading_list_entry_unittest.cc:355: Failure
The difference between delta.InMillisecondsRoundedUp() and 0 is 16, which exceeds 10, where
delta.InMillisecondsRoundedUp() evaluates to -16,
0 evaluates to 0, and
10 evaluates to 10.
[  FAILED  ] ReadingListEntry.FromReadingListLocal (28 ms)

[ RUN      ] ReadingListEntry.FromReadingListLocal
../../components/reading_list/ios/reading_list_entry_unittest.cc:355: Failure
The difference between delta.InMillisecondsRoundedUp() and 0 is 12, which exceeds 10, where
delta.InMillisecondsRoundedUp() evaluates to -12,
0 evaluates to 0, and
10 evaluates to 10.
[  FAILED  ] ReadingListEntry.FromReadingListLocal (31 ms)

Removing Sheriff-Chromium since this has an owner.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9d86f02c7bc5dd777badfb7c1e675a87e434e5d6

commit 9d86f02c7bc5dd777badfb7c1e675a87e434e5d6
Author: olivierrobin <olivierrobin@chromium.org>
Date: Wed Mar 01 11:04:28 2017

Fix ReadingListEntry flaky unittests.

BUG= 658619 

Review-Url: https://codereview.chromium.org/2722163002
Cr-Commit-Position: refs/heads/master@{#453902}

[modify] https://crrev.com/9d86f02c7bc5dd777badfb7c1e675a87e434e5d6/components/reading_list/ios/reading_list_entry_unittest.cc

Detected 19 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app.

These new tests are not Reading List

Detected 3 new flakes for test/step "components_unittests". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21wb25lbnRzX3VuaXR0ZXN0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.

It looks like lots of different problems have been attributed to this issue over time. The recent failures are not all the same, but all seem like infra problems. For the 3 reported in #75, 2 are "Unexpected infrastructure failure" on iPhone 6s iOS 9.0 and iPhone 6s iOS 10.0 respectively. stdout says "93171 2017-04-04 21:49:45.110 E: Internal error!" close to the end.

The third is "Test timed out" on iPad Air 2 iOS 10.0. I can't see anything interesting in the stdout.

Adding Infra-Troopers and setting to untriaged to get some attention on this.

Closing because this bug is an accumulation of different problems and is not actionable.
New individual bugs will be created for future issues.

Closing the issue won't help chromium-try-flakes start distinguishing between flake types. Besides, the new issue will be file for sheriffs again. IMHO the best solution here would be to distinguish between purple and red failures and file separate bugs for troopers and sheriffs correspondingly.

Unfortunately this is not easy to implement in current chromium-try-flakes architecture and therefore will likely not be done until we can find someone who has the time to rewrite the app using event_mon: https://docs.google.com/document/d/1NKRbelRG1N7LqpaBydohZAgynRzD6-cRj5RWlep5UDg.