Hmm, Linux ChromiumOS Tests (dbg) is also failing deteministically, but the first build with the failure doesn't include the cl.
https://uberchromegw.corp.google.com/i/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%28dbg%29%281%29/builds/19457

and Linux Chromium OS ASan LSan Tests
https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/builds/17323

Removing calamity as
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/54189
includes the failure which is earlier than calamity@'s CL.
Adding avi@ as 

Kottackal.George@gmail.com, could you please take a look to check whether https://chromium.googlesource.com/chromium/src/+/72ea5805c1685a57d78aea392331df0b6854976f can be the cause or not?

Disabling the test
https://codereview.chromium.org/2464753002/

This test was stable for a long time and has suddenly started today: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal%20

The last sets of failures in AppBanner / InstallableManager related tests have been due to issues in SW code. Anything interesting happen there lately?

So, r428657 might be related?

+avi

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f6fc8420b146f9b159e4c9ad3f3bc7b78097ff82

commit f6fc8420b146f9b159e4c9ad3f3bc7b78097ff82
Author: tyoshino <tyoshino@chromium.org>
Date: Mon Oct 31 09:50:24 2016

Disable failing test AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal

TBR=benwells@chromium.org
BUG= 660798 

Review-Url: https://codereview.chromium.org/2464753002
Cr-Commit-Position: refs/heads/master@{#428671}

[modify] https://crrev.com/f6fc8420b146f9b159e4c9ad3f3bc7b78097ff82/chrome/browser/banners/app_banner_manager_browsertest.cc

Hi, 

  The change I landed involved modifications to the <embed> tag only. I did not see any use of the embed tag in the code referred to in the failure, namely , the javascript files and the actual source of 
chrome/test/data/banners/manifest_test_page.html. 

 I notice the test has now been disabled. If it is related to the <embed> tag, please do show me the code where the embed element is created and how it is used , I should be able to help further then.

Thanks,
George.

Thanks for looking into, kottackal.george! It seems your change is not related. We'd like avi@ to check it.

 Issue 660815  has been merged into this issue.

My change was switching out the use of obsolete functions with newer ones. I reviewed my code and it shouldn't have changed anything.

Detected 5 new flakes for test/step "AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyVQsSBUZsYWtlIkpBcHBCYW5uZXJNYW5hZ2VyQnJvd3NlclRlc3QuV2ViQXBwQmFubmVyQ3JlYXRlZEluZGlyZWN0TXVsdGlwbGVMYXJnZXJUb3RhbAw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

The 5 new flakes are happening on Windows so I'm going to add Windows to the disabled platforms (https://codereview.chromium.org/2464753002 only disabled for Linux and Mac).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ca0193ead3519c178b5b849041f9004dab73f01a

commit ca0193ead3519c178b5b849041f9004dab73f01a
Author: kjellander <kjellander@chromium.org>
Date: Tue Nov 01 11:44:47 2016

Disable failing test AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal

Disables the test on Windows. https://codereview.chromium.org/2464753002
already disabled it on Mac and Linux.

TBR=benwells@chromium.org
BUG= 660798 

Review-Url: https://codereview.chromium.org/2466313002
Cr-Commit-Position: refs/heads/master@{#428981}

[modify] https://crrev.com/ca0193ead3519c178b5b849041f9004dab73f01a/chrome/browser/banners/app_banner_manager_browsertest.cc

I've just synced to tip of tree, re-enabled this test on Linux, and run it repeatedly. It didn't fail once. I think crrev.com/2452833004 fixes this, because the app banner tests all rely on reloading the page to ensure that it is controlled by the SW.

I'm re-enabling this test now.

+johnme FYI

Update: further inspection of crrev.com/2452833004 shows it doesn't actually change anything that should affect the app banner tests. However, I'm still unable to reproduce this failure.

I'm going to run the same repeated testing on Mac, and if I can't repro the failure I'm going to re-enable the test and see how it goes.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/59a0f926eee47b951754598ac77dda4a5bbafcde

commit 59a0f926eee47b951754598ac77dda4a5bbafcde
Author: dominickn <dominickn@chromium.org>
Date: Wed Nov 02 02:49:57 2016

Re-enable AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal.

The failures don't reproduce locally - re-enabling to see if that
sticks.

BUG= 660798 
TBR=benwells@chromium.org

Review-Url: https://codereview.chromium.org/2465163003
Cr-Commit-Position: refs/heads/master@{#429191}

[modify] https://crrev.com/59a0f926eee47b951754598ac77dda4a5bbafcde/chrome/browser/banners/app_banner_manager_browsertest.cc

So far, so good, no flaking yet: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal. I'll check back in on this tomorrow and close if it's still okay then.

This failure is observed on precise64 for official.desktop as well.

Link to the build:
-----------------
https://uberchromegw.corp.google.com/i/official.desktop/builders/precise64/builds/1220

Link to the log file:
---------------------
https://uberchromegw.corp.google.com/i/official.desktop/builders/precise64/builds/1220/steps/browser_tests/logs/stdio

Detected 4 new flakes for test/step "AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyVQsSBUZsYWtlIkpBcHBCYW5uZXJNYW5hZ2VyQnJvd3NlclRlc3QuV2ViQXBwQmFubmVyQ3JlYXRlZEluZGlyZWN0TXVsdGlwbGVMYXJnZXJUb3RhbAw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

 Issue 661511  has been merged into this issue.

AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectLargerTotal is also flaky, see 
http://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyTQsSBUZsYWtlIkJBcHBCYW5uZXJNYW5hZ2VyQnJvd3NlclRlc3QuV2ViQXBwQmFubmVyQ3JlYXRlZEluZGlyZWN0TGFyZ2VyVG90YWwM

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e0169db530967046e639b3746dabb68741c893b8

commit e0169db530967046e639b3746dabb68741c893b8
Author: jam <jam@chromium.org>
Date: Thu Nov 03 15:01:15 2016

Revert of Re-enable AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal. (patchset #1 id:1 of https://codereview.chromium.org/2465163003/ )

Reason for revert:
This is causing a lot of try bot flakes

Please don't renable without fixing.

See
http://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyVQsSBUZsYWtlIkpBcHBCYW5uZXJNYW5hZ2VyQnJvd3NlclRlc3QuV2ViQXBwQmFubmVyQ3JlYXRlZEluZGlyZWN0TXVsdGlwbGVMYXJnZXJUb3RhbAw

Original issue's description:
> Re-enable AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal.
>
> The failures don't reproduce locally - re-enabling to see if that
> sticks.
>
> BUG= 660798 
> TBR=benwells@chromium.org
>
> Committed: https://crrev.com/59a0f926eee47b951754598ac77dda4a5bbafcde
> Cr-Commit-Position: refs/heads/master@{#429191}

TBR=benwells@chromium.org,dominickn@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 660798 

Review-Url: https://codereview.chromium.org/2471843004
Cr-Commit-Position: refs/heads/master@{#429585}

[modify] https://crrev.com/e0169db530967046e639b3746dabb68741c893b8/chrome/browser/banners/app_banner_manager_browsertest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5f6db7c6d0baa373cf4e054c8932fd1efee809ce

commit 5f6db7c6d0baa373cf4e054c8932fd1efee809ce
Author: jam <jam@chromium.org>
Date: Thu Nov 03 15:06:07 2016

Disable flaky AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectLargerTotal.

BUG= 660798 
TBR=dominickn@chromium.org

Review-Url: https://codereview.chromium.org/2474063002
Cr-Commit-Position: refs/heads/master@{#429587}

[modify] https://crrev.com/5f6db7c6d0baa373cf4e054c8932fd1efee809ce/chrome/browser/banners/app_banner_manager_browsertest.cc

 Issue 662123  has been merged into this issue.

removing from immediate sheriff queue since they're both disabled now

These are failing everywhere now. They're probably not flaky any more.

On Mac:
https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.11%20Tests/builds/3227
https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.10%20Tests/builds/8681
https://uberchromegw.corp.google.com/i/chromium.memory/builders/Mac%20ASan%2064%20Tests%20%281%29/builds/23837
https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.9%20Tests/builds/31229
https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.9%20Tests%20%28dbg%29/builds/32544
AppBannerManagerBrowserTest.CancelBannerIndirect [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectLargerTotal [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultipleLargerTotal [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultiple [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedVarious [logdog]

On Linux:
https://uberchromegw.corp.google.com/i/chromium.linux/builders/Linux%20Tests%20%28dbg%29%281%29/builds/58550
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple [logdog]

https://uberchromegw.corp.google.com/i/chromium.linux/builders/Linux%20Tests%20%28dbg%29%281%29%2832%29/builds/35193
https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/builds/17422
https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ASan%20LSan%20Tests%20%281%29/builds/31013
AppBannerManagerBrowserTest.CancelBannerIndirect [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectLargerTotal [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultiple [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedVarious [logdog]

On Windows
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win10%20Tests%20x64/builds/5699
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win%207%20Tests%20x64%20%281%29/builds/18145
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20%2832%29%20Tests/builds/12221
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%281%29/builds/59514
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/54348
AppBannerManagerBrowserTest.CancelBannerIndirect [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectLargerTotal [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultiple [logdog]
AppBannerManagerBrowserTest.WebAppBannerCreatedVarious [logdog]

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c230c76e6fc27b06c1da541fa50ca6c3eb624c71

commit c230c76e6fc27b06c1da541fa50ca6c3eb624c71
Author: tapted <tapted@chromium.org>
Date: Fri Nov 04 08:06:49 2016

Disable app_banner_manager_browsertest.cc. Systemic flakiness.

BUG= 660798 
TBR=dominickn@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2474783006
Cr-Commit-Position: refs/heads/master@{#429828}

[modify] https://crrev.com/c230c76e6fc27b06c1da541fa50ca6c3eb624c71/chrome/test/BUILD.gn

Errors like

[ RUN      ] AppBannerManagerBrowserTest.CancelBannerIndirect
AppBannerManagerBrowserTest.CancelBannerIndirect (run #1):
../../base/test/histogram_tester.cc:151: Failure
Value of: actual_count
  Actual: 0
Expected: expected_count
Which is: 1
Histogram "AppBanners.InstallableStatusCode" does not have the right number of samples (1) in the expected bucket (2). It has (0).
[  FAILED  ] AppBannerManagerBrowserTest.CancelBannerIndirect, where TypeParam =  and GetParam() =  (3381 ms)

AppBannerManagerBrowserTest.WebAppBannerCreatedDirectLargerTotal (run #1):
[ RUN      ] AppBannerManagerBrowserTest.WebAppBannerCreatedDirectLargerTotal
../../chrome/browser/banners/app_banner_manager_browsertest.cc:115: Failure
Value of: manager->will_show()
  Actual: false
Expected: expected_to_show
Which is: true
../../base/test/histogram_tester.cc:151: Failure
Value of: actual_count
  Actual: 0
Expected: expected_count
Which is: 1
Histogram "AppBanners.InstallableStatusCode" does not have the right number of samples (1) in the expected bucket (25). It has (0).
[  FAILED  ] AppBannerManagerBrowserTest.WebAppBannerCreatedDirectLargerTotal, where TypeParam =  and GetParam() =  (3588 ms)

AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple (run #1):
[ RUN      ] AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple
../../chrome/browser/banners/app_banner_manager_browsertest.cc:115: Failure
Value of: manager->will_show()
  Actual: false
Expected: expected_to_show
Which is: true
../../base/test/histogram_tester.cc:151: Failure
Value of: actual_count
  Actual: 0
Expected: expected_count
Which is: 1
Histogram "AppBanners.InstallableStatusCode" does not have the right number of samples (1) in the expected bucket (25). It has (0).
[  FAILED  ] AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultiple, where TypeParam =  and GetParam() =  (4163 ms)


AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultipleLargerTotal (run #1):
[ RUN      ] AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultipleLargerTotal
../../chrome/browser/banners/app_banner_manager_browsertest.cc:115: Failure
Value of: manager->will_show()
  Actual: false
Expected: expected_to_show
Which is: true
../../base/test/histogram_tester.cc:151: Failure
Value of: actual_count
  Actual: 0
Expected: expected_count
Which is: 1
Histogram "AppBanners.InstallableStatusCode" does not have the right number of samples (1) in the expected bucket (25). It has (0).
[  FAILED  ] AppBannerManagerBrowserTest.WebAppBannerCreatedDirectMultipleLargerTotal, where TypeParam =  and GetParam() =  (3753 ms)

AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultiple (run #1):
[ RUN      ] AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultiple
../../chrome/browser/banners/app_banner_manager_browsertest.cc:115: Failure
Value of: manager->will_show()
  Actual: false
Expected: expected_to_show
Which is: true
../../base/test/histogram_tester.cc:151: Failure
Value of: actual_count
  Actual: 0
Expected: expected_count
Which is: 1
Histogram "AppBanners.InstallableStatusCode" does not have the right number of samples (1) in the expected bucket (25). It has (0).
[  FAILED  ] AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultiple, where TypeParam =  and GetParam() =  (3323 ms)

AppBannerManagerBrowserTest.WebAppBannerCreatedVarious (run #1):
[ RUN      ] AppBannerManagerBrowserTest.WebAppBannerCreatedVarious
../../chrome/browser/banners/app_banner_manager_browsertest.cc:115: Failure
Value of: manager->will_show()
  Actual: false
Expected: expected_to_show
Which is: true
../../base/test/histogram_tester.cc:151: Failure
Value of: actual_count
  Actual: 0
Expected: expected_count
Which is: 1
Histogram "AppBanners.InstallableStatusCode" does not have the right number of samples (1) in the expected bucket (25). It has (0).
[  FAILED  ] AppBannerManagerBrowserTest.WebAppBannerCreatedVarious, where TypeParam =  and GetParam() =  (3234 ms)

There is no suspect CL. I suspect the test sharding just changed so that some conflict now occurs reliably that was only flaky previously. The links in #c30 are all the first failures on the bots. One has only a single CL in it:

https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.9%20Tests/builds/31229

r429813 -> "Fix some flaky event tests in ImeTest"
Changed files
content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java
content/test/data/android/input/input_forms.html

Obviously not related to a failure starting on a Mac bot. Also this CL does not appear in all of the other first-failing ranges.

https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.9%20Tests/builds/31228 is a green cycle.

r429828 was a pretty big hammer. This should be addressed before next branch.

I'm away/travelling from now until branch point, so I won't have time to look into this.

I'll reiterate that nothing in app banner code has landed in the range of this flakiness. The tests were completely fine and then suddenly started flaking (https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=appbannermanagerbrowsertest)

My suspicion is that something has changed in service worker related code, since app banners rely on being able to detect a service worker registration on first landing on a page. Tests have previously been made flaky by changes there.

+falken - is there anything you can think of that might have caused this?

I can't think of a recent change to service worker that could cause this. The only somewhat suspicious patch in this range is avi@'s r428657 since it touches SW registration code but I've read it over and agree it shouldn't have changed anything. If we can find no clues or repros, I'd recommend reverting that and reenabling the test although it could cause breakage on tryjobs again as a speculative fix. Is there some way to enable the test to see the result but not cause the CQ to fail?

It's also odd if only AppBanner is failing and not other users of service worker.

I'll note the flakiness occurred with a weird pattern: looking at AppBannerManagerBrowserTest.WebAppBannerCreatedIndirectMultipleLargerTotal on the flakiness dashboard, the test was passing all day on Nov 2, then failed on all bots for 2-3 runs, then passed again until it failed on Nov 3 at about the same time (5 PM GMT), failing on all bots 2-3 runs. The 24 hour failure is suspicious since SW have a 24 hour update behavior, but I'd expect browser tests to clear all stored state between runs.

 Issue 662351  has been merged into this issue.

 Issue 662518  has been merged into this issue.

I've ran browser_tests --gtest_repeat=1 --gtest_filter="AppBannerManagerBrowserTest" for over 24 hours and could not repro any flake. I also had edited service worker's 24 hour update mechanism to 1 hour in hopes of speeding up a repro if that really was the cause.

It could be the repro needs all browser tests running, and not --gtest_repeat, or something.

I've also been hammering browser_tests starting with A for some hours with shuffle and repeat and I can't reproduce any flakes.

One idea could be to periodically send try jobs (once every few hours) on a patch that re-enables the tests.

We're seeing some new service worker crashes since 56.0.2906.0, which is around the time of these failures:
issue 664014, issue 661339, issue 661890

We still don't understand what triggered these crashes but it could certainly be related to the failures here. We have a speculative fix in progress.

Curiously avi@'s https://crrev.com/428657 landed in 56.0.2906.0. I wonder if it could have caused some kind of slight timing change that made these failures more likely.

Very curious. I think there's enough evidence to try reverting that CL and re-enabling the tests, since locally reproducing the flake seems very difficult. Otherwise the try bot experiment you suggested is probably the way to go. 

If no one is opposed I will look into this next week.

Thanks. We also have a potential fix in the CQ: https://codereview.chromium.org/2498573002/

I have a CL re-enabling all the tests at https://codereview.chromium.org/2500123002/.

So far I've run linux_chromium_rel_ng 50 times, with no flakes of AppBannerManagerBrowserTests. I'm going to do the same with win_chromium_rel_ng and mac_chromium_rel_ng. If I can't get it to flake, I will land the CL unless there are any other objections.

The 50 trybot runs on Mac, Windows, and Linux have finished with no flakes of any AppBannerManagerBrowserTests. I'm going to land the CL which re-enables the tests now, and watch the flakiness dashboard after it lands.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a4d73beb118fed32cc6c53c78ec236e51a0ff32

commit 1a4d73beb118fed32cc6c53c78ec236e51a0ff32
Author: dominickn <dominickn@chromium.org>
Date: Tue Nov 15 20:40:37 2016

Re-enable AppBannerManagerBrowserTests.

This CL re-enables the disabled AppBannerManagerBrowserTests. Prior to
landing, this has been run through 50 iterations on each of the Linux,
Windows, and Mac trybots with no flakes observed.

BUG= 660798 
TBR=benwells@chromium.org

Review-Url: https://codereview.chromium.org/2500123002
Cr-Commit-Position: refs/heads/master@{#432244}

[modify] https://crrev.com/1a4d73beb118fed32cc6c53c78ec236e51a0ff32/chrome/browser/banners/app_banner_manager_browsertest.cc
[modify] https://crrev.com/1a4d73beb118fed32cc6c53c78ec236e51a0ff32/chrome/test/BUILD.gn

No flakes so far: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=appbannermanagerbrowsertest

I've watched these all day with no flakes. I'm going to mark this as fixed and put the cause down to those SW issues.