Have CL to disable the test:
https://chromium-review.googlesource.com/c/catapult/+/1094981

However I cannot TBR it because it's in catapult. Waiting for review by eakuefner@.

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/51593b4f24ae8096bfbbd8bb09ec17d44e52b7b4

commit 51593b4f24ae8096bfbbd8bb09ec17d44e52b7b4
Author: Samuel Huang <huangs@chromium.org>
Date: Mon Jun 11 15:47:25 2018

[Sheriff] Disable telemetry.internal.browser.browser_unittest.BrowserTest.testForegroundTab

The test was failing uner MacOS bots, so disabling it only for "mac".

Bug: chromium:851463
Change-Id: I8d555eadf33087ecfafbf07faa37389d53fc8d2e
Reviewed-on: https://chromium-review.googlesource.com/1094981
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/51593b4f24ae8096bfbbd8bb09ec17d44e52b7b4/telemetry/telemetry/internal/browser/browser_unittest.py

Adding another test to this bug, as these failures are related.  (The bot started flaking at the same time on the same bots.)

Shutao: have FindIt for telemetry_unittests launch yet? It would be a lot easier if we know which CL introduce these new flakes for Telemetry :-(

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/9f7abcb46e2fcb69143b8581f8c09ebe5ad414bd

commit 9f7abcb46e2fcb69143b8581f8c09ebe5ad414bd
Author: nednguyen <nednguyen@google.com>
Date: Mon Jun 11 19:00:17 2018

Disable BrowserTest.testActivateTab on Mac

Bug: chromium:851463
Change-Id: Id573a87b09f076f326f8698951bb2210135cd4ba
TBR=kbr@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/1095856
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/9f7abcb46e2fcb69143b8581f8c09ebe5ad414bd/telemetry/telemetry/internal/browser/tab_unittest.py

Optimistically removing from sheriff queue.

To be clear, these two tests almost always *fail*.  Look at the sea of red here:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.12%20Tests?limit=100

Maybe they're succeeding one time in forty?

I have see them fail independently (i.e., it's not like one only fails when the other does).  I've seen A fail and B fail, A succeed and B fail, A fail and B succeed, and both succeed.

The first build on this bot that contains any of these failures is
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.12%20Tests/13622

Before that it's all green for telemetry_unittests.

Looking at the blamelist, I can easily imagine that
https://chromium-review.googlesource.com/c/chromium/src/+/1017460
is related (a Mac-only change that enables "variations: add client config for ViewsBrowserWindows").  CC ellyjones@

<<< Note to future sheriffs >>>

If the catapult roll that includes the two telemetry changes above to disable the test doesn't land soon
( check https://catapult-roll.skia.org/ )
then simply try reverting the changelist mentioned in comment #10.

Actually FindIt confirm that CL also cause flakiness in MemoryTracingBrowserTest.TestMemoryInfra, I am making a revert CL 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/400bca39520a151766a16dbf35e53dcd97a93222

commit 400bca39520a151766a16dbf35e53dcd97a93222
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Tue Jun 12 04:31:57 2018

Roll src/third_party/catapult 411c93b1c918..0f4cd854b5c3 (5 commits)

https://chromium.googlesource.com/catapult.git/+log/411c93b1c918..0f4cd854b5c3


git log 411c93b1c918..0f4cd854b5c3 --date=short --no-merges --format='%ad %ae %s'
2018-06-11 benjhayden@chromium.org Remove stale hack in dashboard/graph_json.py.
2018-06-11 nednguyen@google.com Disable BrowserTest.testActivateTab on Mac
2018-06-11 nednguyen@google.com Add .vpython for catapult
2018-06-11 perezju@chromium.org [soundwave] Skip over test_paths with missing data
2018-06-11 huangs@chromium.org [Sheriff] Disable telemetry.internal.browser.browser_unittest.BrowserTest.testForegroundTab


Created with:
  gclient setdep -r src/third_party/catapult@0f4cd854b5c3

The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:851463,chromium:851498,chromium:851463
TBR=sullivan@chromium.org

Change-Id: I1cf2bc71b4ff32ebbb8360a484320bad68e55dde
Reviewed-on: https://chromium-review.googlesource.com/1096218
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#566312}
[modify] https://crrev.com/400bca39520a151766a16dbf35e53dcd97a93222/DEPS

Revert cannot land since it makes the following tests fail on interactive_ui_tests step on Mac bot:
GlobalKeyboardShortcutsTest.SwitchTabsMac
GlobalKeyboardShortcutsTest.CopyPasteOmnibox

Elly: can you either revert your change or investigate & fixed the tests that were disabled? Those tests are integration tests, so their breakage may mean actual defect in the Chrome product

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/44a90a0763a7c69af1c7b1def9a04200ddb6cfac

commit 44a90a0763a7c69af1c7b1def9a04200ddb6cfac
Author: Ned Nguyen <nednguyen@google.com>
Date: Tue Jun 12 16:30:50 2018

Revert "variations: add client config for ViewsBrowserWindows"

This reverts commit f824b3a3926819d6398b7a7b3da1e251d33417da.

Reason for revert: making Telemetry test flaky (https://bugs.chromium.org/p/chromium/issues/detail?id=851463#c10).

This is also confirmed by FindIt https://findit-for-me.appspot.com/waterfall/flake/flake-culprit?key=ag9zfmZpbmRpdC1mb3ItbWVyQwsSDEZsYWtlQ3VscHJpdCIxY2hyb21pdW0vZjgyNGIzYTM5MjY4MTlkNjM5OGI3YTdiM2RhMWUyNTFkMzM0MTdkYQw

BUG:851463

Original change's description:
> variations: add client config for ViewsBrowserWindows
> 
> Bug: 827247
> Change-Id: Ic09c38f7b658b1f6d79ecec8a6d0895aca44d797
> Reviewed-on: https://chromium-review.googlesource.com/1017460
> Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
> Reviewed-by: Ilya Sherman <isherman@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#565782}

TBR=ellyjones@chromium.org,isherman@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 827247
Change-Id: If0ec527c1a7572a06b0bdcce28fb7e5ac196d1e1
Reviewed-on: https://chromium-review.googlesource.com/1096376
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Elly Fong-Jones <ellyjones@chromium.org>
Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566454}
[modify] https://crrev.com/44a90a0763a7c69af1c7b1def9a04200ddb6cfac/testing/variations/fieldtrial_testing_config.json

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/f43f943c93d0967ee88fb638ac15cebd7f3cc956

commit f43f943c93d0967ee88fb638ac15cebd7f3cc956
Author: Ned Nguyen <nednguyen@google.com>
Date: Tue Jun 12 17:10:52 2018

Revert "Disable BrowserTest.testActivateTab on Mac"

This reverts commit 9f7abcb46e2fcb69143b8581f8c09ebe5ad414bd.

Reason for revert: culprit CL was reverted

Original change's description:
> Disable BrowserTest.testActivateTab on Mac
> 
> Bug: chromium:851463
> Change-Id: Id573a87b09f076f326f8698951bb2210135cd4ba
> TBR=kbr@chromium.org
> Reviewed-on: https://chromium-review.googlesource.com/1095856
> Reviewed-by: Ned Nguyen <nednguyen@google.com>
> Commit-Queue: Ned Nguyen <nednguyen@google.com>

TBR=kbr@chromium.org,nednguyen@google.com

Change-Id: Ib49100cba60d1f59833489b60817d518479141b6
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:851463
Reviewed-on: https://chromium-review.googlesource.com/1097516
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/f43f943c93d0967ee88fb638ac15cebd7f3cc956/telemetry/telemetry/internal/browser/tab_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/162d75e72414ca3ddc2871085a054f704fc49496

commit 162d75e72414ca3ddc2871085a054f704fc49496
Author: Ned Nguyen <nednguyen@google.com>
Date: Tue Jun 12 17:40:51 2018

Revert "[Sheriff] Disable telemetry.internal.browser.browser_unittest.BrowserTest.testForegroundTab"

This reverts commit 51593b4f24ae8096bfbbd8bb09ec17d44e52b7b4.

Reason for revert: culprit CL was reverted

Original change's description:
> [Sheriff] Disable telemetry.internal.browser.browser_unittest.BrowserTest.testForegroundTab
> 
> The test was failing uner MacOS bots, so disabling it only for "mac".
> 
> Bug: chromium:851463
> Change-Id: I8d555eadf33087ecfafbf07faa37389d53fc8d2e
> Reviewed-on: https://chromium-review.googlesource.com/1094981
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Reviewed-by: Ned Nguyen <nednguyen@google.com>

TBR=huangs@chromium.org,aberent@chromium.org,eakuefner@chromium.org,nednguyen@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: chromium:851463
Change-Id: I9f3c66b23b830b51daa259837652e731881a0128
Reviewed-on: https://chromium-review.googlesource.com/1097515
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/162d75e72414ca3ddc2871085a054f704fc49496/telemetry/telemetry/internal/browser/browser_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e49065b632a0e741e7ad6bd19a9728f46ac42a1c

commit e49065b632a0e741e7ad6bd19a9728f46ac42a1c
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed Jun 13 04:14:23 2018

Roll src/third_party/catapult fdacd1639e2a..459c4d0417df (5 commits)

https://chromium.googlesource.com/catapult.git/+log/fdacd1639e2a..459c4d0417df


git log fdacd1639e2a..459c4d0417df --date=short --no-merges --format='%ad %ae %s'
2018-06-12 benjhayden@chromium.org Add an index for GetAlertsAroundRevision for non-internal users.
2018-06-12 achuith@chromium.org oobe: Also support gaiastaging.
2018-06-12 nednguyen@google.com Revert "[Sheriff] Disable telemetry.internal.browser.browser_unittest.BrowserTest.testForegroundTab"
2018-06-12 nednguyen@google.com Revert "Disable BrowserTest.testActivateTab on Mac"
2018-06-12 perezju@chromium.org [Telemetry] Remove measurement property from TimelineBasedPageTest


Created with:
  gclient setdep -r src/third_party/catapult@459c4d0417df

The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:851463,chromium:851463, chromium:851948 
TBR=sullivan@chromium.org

Change-Id: Ib82446d7c3fe1aa9f915886dfdee5059fecd4bf1
Reviewed-on: https://chromium-review.googlesource.com/1097817
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#566716}
[modify] https://crrev.com/e49065b632a0e741e7ad6bd19a9728f46ac42a1c/DEPS

re #6: I don't think so. (sorry for late reply, was in summit)

chanli@: how far are we to extend to support telemetry-based tests?

How do I reproduce this failure locally? I believe I am properly running the telemetry browser_unittest suite, but the test doesn't fail for me.

I believe this test is inherently racy and MacViews just happens to tickle it:

1. Telemetry tests run with `--ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I=` - https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_startup_args.py?type=cs&q=%22ignore-certificate-errors-spki-list%22&sq=package:chromium&g=0&l=110 (confirmed from bot logs: https://chromium-swarm.appspot.com/task?id=3e0b2242e1fa7310&refresh=10&show_raw=1&wide_logs=true)

2. This switch is listed in kBadFlags, so it causes the "stability and security" infobar to show.

3. That infobar animates in asynchronously at browser startup, so whether it is present or not at a particular point in the test is random.

4. If we try to activate the browser window while an infobar is present and animating, we can enter NativeViewHostMac::ShowWidget() with fast_resize() == true. Note: I don't actually see how this could happen, since the *only* place this is set to true is via WebView::SetFastResize(), which in turn is only called in BrowserView::ToolbarSizeChanged(), which seems to always call it with false, so we can't ever be in fast_resize() == true when we return to the message loop.

Anyway, I think the real issue is that:

5. The test does this:

    # Make sure that activating the background tab makes it the foreground tab
    original_tab.Activate()
    self.assertEqual(self._browser.foreground_tab, original_tab)

But the documentation for tab.Activate() says that:

    Please note: this is asynchronous. There is a delay between this call
    and the page's documentVisibilityState becoming 'visible', and yet more
    delay until the actual tab is visible to the user. None of these delays
    are included in this call.

> So I think this test is just inherently racy, and something about MacViews is taking long enough to break it. Perhaps the animation on the infobar.

Re item 4 in above list: ToolbarSizeChanged is called via  BrowserView::InfoBarContainerStateChanged - 
which is passed true when a new infobar is animating in.

Thanks for the analysis Elly! I will add "wait for visible" to see if the test is less racy

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/31767de5cf0a4baa96c28c90663276e165d86473

commit 31767de5cf0a4baa96c28c90663276e165d86473
Author: nednguyen <nednguyen@google.com>
Date: Fri Jun 15 12:38:52 2018

Add wait to make sure that activate tab call wait for the tab to be foregrounded

This helps making sure that telemetry.internal.browser.browser_unittest.BrowserTest.testActivateTab is not racy

Bug: chromium:851463
Change-Id: I021c0ecb70643a8c5b7254a8ffc300535e89129b
Reviewed-on: https://chromium-review.googlesource.com/1101929
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>

[modify] https://crrev.com/31767de5cf0a4baa96c28c90663276e165d86473/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py
[modify] https://crrev.com/31767de5cf0a4baa96c28c90663276e165d86473/telemetry/telemetry/internal/browser/tab.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/baa137fe501cde24a0faf599e1d275650097f8a7

commit baa137fe501cde24a0faf599e1d275650097f8a7
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 15 13:47:41 2018

Roll src/third_party/catapult d02fd6682709..31767de5cf0a (1 commits)

https://chromium.googlesource.com/catapult.git/+log/d02fd6682709..31767de5cf0a


git log d02fd6682709..31767de5cf0a --date=short --no-merges --format='%ad %ae %s'
2018-06-15 nednguyen@google.com Add wait to make sure that activate tab call wait for the tab to be foregrounded


Created with:
  gclient setdep -r src/third_party/catapult@31767de5cf0a

The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:851463
TBR=sullivan@chromium.org

Change-Id: I41847b83c6fbe4d7c0052e9a49dc38ef2f46c464
Reviewed-on: https://chromium-review.googlesource.com/1102399
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#567626}
[modify] https://crrev.com/baa137fe501cde24a0faf599e1d275650097f8a7/DEPS

Removing from sheriff queue since people are actively working on it and there's no action for the sheriff to take.

The Telemetry-based GPU tests have been passing the --test-type flag to the browser for a while in order to suppress some infobars. Maybe that would help here as well. See:
https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/screenshot_sync_integration_test.py?type=cs&q=%22--test-type%3Dgpu%22&sq=package:chromium&g=0&l=66

Issue 853994 has been merged into this issue.

Issue 854222 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1822fbdb820d3c518eea72910de72ceba4652483

commit 1822fbdb820d3c518eea72910de72ceba4652483
Author: Elly Fong-Jones <ellyjones@chromium.org>
Date: Thu Jun 28 16:22:20 2018

Reland "variations: add client config for ViewsBrowserWindows"

This is a reland of f824b3a3926819d6398b7a7b3da1e251d33417da

Original change's description:
> variations: add client config for ViewsBrowserWindows
>
> Bug: 827247
> Change-Id: Ic09c38f7b658b1f6d79ecec8a6d0895aca44d797
> Reviewed-on: https://chromium-review.googlesource.com/1017460
> Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
> Reviewed-by: Ilya Sherman <isherman@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#565782}

Relanding the CL.

Bug: 827247,851463
TBR: isherman@chromium.org
Change-Id: Ic3b23594fa74b24023924ad4107bf262e641bfb7
Reviewed-on: https://chromium-review.googlesource.com/1106498
Reviewed-by: Robert Liao <robliao@chromium.org>
Commit-Queue: Robert Liao <robliao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#571154}
[modify] https://crrev.com/1822fbdb820d3c518eea72910de72ceba4652483/testing/variations/fieldtrial_testing_config.json