These are failing in the "browser_tests (with patch)" test for tryserver.chromium.mac/builders/mac_chromium_rel_ng.

We should not be reporting test flakiness when a CL test fails. If someone tries their failing patch mulitple times then of course it will appear flaky but is in fact WAI. It would make more sense to report flakiness in on "browser_tests (without patch)" instead.

 Issue 715102  has been merged into this issue.

 Issue 715027  has been merged into this issue.

 Issue 715026  has been merged into this issue.

 Issue 715025  has been merged into this issue.

 Issue 715024  has been merged into this issue.

 Issue 715023  has been merged into this issue.

 Issue 715022  has been merged into this issue.

 Issue 715021  has been merged into this issue.

 Issue 714669  has been merged into this issue.

 Issue 714566  has been merged into this issue.

 Issue 714565  has been merged into this issue.

 Issue 714564  has been merged into this issue.

 Issue 714563  has been merged into this issue.

 Issue 714562  has been merged into this issue.

 Issue 714561  has been merged into this issue.

 Issue 714560  has been merged into this issue.

 Issue 714559  has been merged into this issue.

Dirk, can you find an owner for this? This is annoying for sheriffs.

Detected 3 new flakes for test/step "ErrorPageAutoReloadTest.ManualReloadNotSuppressed". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyPAsSBUZsYWtlIjFFcnJvclBhZ2VBdXRvUmVsb2FkVGVzdC5NYW51YWxSZWxvYWROb3RTdXBwcmVzc2VkDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

I think the assessment here is wrong.

One of the patches identified as a flake is here: https://codereview.chromium.org/2844383003/#ps1 . If you click on that, you'll see the patch eventually landed (the same patch). In other words the exact same patch landed on multiple tries. Same thing here: https://codereview.chromium.org/2849433002/#ps80001 . Looking at the logs for the failures I see similar errors:

from https://codereview.chromium.org/2849433002/#ps80001

[ RUN      ] ErrorPageTest.StaleCacheStatus
[42966:7939:0427/142412.419628:WARNING:notification_platform_bridge_mac.mm(493)] AlertNotificationService: XPC connection invalidated.
[42966:52995:0427/142412.433535:WARNING:mac_util.mm(222)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.sLj5BW/ddZz0MI/Default/History': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn?t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50)
[42966:52995:0427/142412.450287:WARNING:mac_util.mm(222)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.sLj5BW/ddZz0MI/Default/Favicons': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn?t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50)
[42971:771:0427/142412.553092:WARNING:vt_video_decode_accelerator_mac.cc(178)] Failed to create VTDecompressionSession: Error Domain=NSOSStatusErrorDomain Code=-8973 "The operation couldn?t be completed. (OSStatus error -8973.)" (codecOpenErr) (-8973)
[42971:771:0427/142412.554287:WARNING:vt_video_decode_accelerator_mac.cc(200)] Failed to create hardware VideoToolbox session
[42971:771:0427/142412.626740:ERROR:vt_video_encode_accelerator_mac.cc(495)]  VTCompressionSessionCreate failed: -12908
[42966:29955:0427/142413.056787:WARNING:simple_synchronous_entry.cc(1051)] Could not open platform files for entry.
../../chrome/browser/net/errorpage_browsertest.cc:994: Failure
Value of: IsDisplayingText(browser(), GetShowSavedButtonLabel())
  Actual: false
Expected: true

from https://codereview.chromium.org/2844383003/#ps1

[ RUN      ] ErrorPageAutoReloadTest.ManualReloadNotSuppressed
[1854:5123:0427/145859.009604:WARNING:notification_platform_bridge_mac.mm(493)] AlertNotificationService: XPC connection invalidated.
[1854:52235:0427/145859.008761:WARNING:mac_util.mm(222)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.5c82lU/d8h2f4t/Default/History': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn?t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50)
[1854:52235:0427/145859.073200:WARNING:mac_util.mm(222)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.5c82lU/d8h2f4t/Default/Favicons': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn?t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50)
[1865:771:0427/145859.170371:WARNING:vt_video_decode_accelerator_mac.cc(178)] Failed to create VTDecompressionSession: Error Domain=NSOSStatusErrorDomain Code=-8973 "The operation couldn?t be completed. (OSStatus error -8973.)" (codecOpenErr) (-8973)
[1865:771:0427/145859.171577:WARNING:vt_video_decode_accelerator_mac.cc(200)] Failed to create hardware VideoToolbox session
[1865:771:0427/145859.248148:ERROR:vt_video_encode_accelerator_mac.cc(495)]  VTCompressionSessionCreate failed: -12908
../../chrome/browser/net/errorpage_browsertest.cc:1136: Failure
Value of: IsDisplayingText(browser(), l10n_util::GetStringUTF8( IDS_ERRORPAGES_SUGGESTION_CHECK_CONNECTION_HEADER))
  Actual: true
Expected: false

I don't know enough about this code to know if the ERRORS/WARNING are relevant to the failure.

I'm assigning to Elly based on the blame logs in errorpage_browsertest.cc, removing infra from this bug and moving to Dirk to the cc in case he wants to ponder some of the earlier comments.

I'm going to also remove Sherriff-Chromium. If this pops again today I'll start disabling these tests.

Note that this was originally for ErrorPageTest.StaleCacheStatus, and there don't appear to be recent failures, so maybe it could be closed.

I'm not sure if my understanding is correct here, but I think that chromium-try-flakes only reports flakiness if the test sometimes passes and sometimes fails with the same patch (same base revision, same patch, different result) - and in general we *do* want to analyse "with patch" steps. Is that not right?

Detected 5 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "info_collection_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLQsSBUZsYWtlIiJpbmZvX2NvbGxlY3Rpb25fdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "gpu_process_launch_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiVncHVfcHJvY2Vzc19sYXVuY2hfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "depth_capture_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKwsSBUZsYWtlIiBkZXB0aF9jYXB0dXJlX3Rlc3RzICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "pixel_test (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIgsSBUZsYWtlIhdwaXhlbF90ZXN0ICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "gpu_process_launch_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiVncHVfcHJvY2Vzc19sYXVuY2hfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "hardware_accelerated_feature_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyOgsSBUZsYWtlIi9oYXJkd2FyZV9hY2NlbGVyYXRlZF9mZWF0dXJlX3Rlc3RzICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "gpu_process_launch_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiVncHVfcHJvY2Vzc19sYXVuY2hfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "maps_pixel_test (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJwsSBUZsYWtlIhxtYXBzX3BpeGVsX3Rlc3QgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "pixel_test (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIgsSBUZsYWtlIhdwaXhlbF90ZXN0ICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "hardware_accelerated_feature_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyOgsSBUZsYWtlIi9oYXJkd2FyZV9hY2NlbGVyYXRlZF9mZWF0dXJlX3Rlc3RzICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "maps_pixel_test (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJwsSBUZsYWtlIhxtYXBzX3BpeGVsX3Rlc3QgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "pixel_test (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyIgsSBUZsYWtlIhdwaXhlbF90ZXN0ICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Over to network for triage - I'm not even sure if the original flaky test is still flaky.

All those alerts look bogus.  pixel_test?  This is not a pixel test.  The "maps_pixel_test" link shows cases where all tests failed (And this is also still not a pixel_test).   Same for hardware_accelerated_feature_tests.  Looks like something is wrong with our infrastructure.  Would file a bug for it, but I have no idea what infrastructure this is, or who owns it.