I ran the above browser test on Linux and the test ran successfully. Is there a way to reproduce the above test failure?

added dxie@.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc45269c31186c63cfa2b99921593806b9b54c36

commit fc45269c31186c63cfa2b99921593806b9b54c36
Author: Jun Cai <juncai@chromium.org>
Date: Wed Nov 07 19:48:16 2018

Revert "Reland: Network Service: Migrate NetworkMetricsProvider to NetworkConnectionTracker"

This reverts commit 0df82fcde5a33197e31ad4f981197b279e7327ec.

Reason for revert: The Builder linux64 trunk keeps failing due to ChromeBrowserMainBrowserTest.VariationsServiceStartsRequestOnNetworkChange browser test.
https://bugs.chromium.org/p/chromium/issues/detail?id=902784

Bug:  902784 ,  883121 

Original change's description:
> Reland: Network Service: Migrate NetworkMetricsProvider to NetworkConnectionTracker
> 
> The "Initial upload" patch of this CL is the same as:
> https://chromium-review.googlesource.com/c/chromium/src/+/1274585
> 
> The above original CL was reverted at:
> https://chromium-review.googlesource.com/c/chromium/src/+/1319614
> 
> The major changes since the "Initial upload" patch set 1:
> Removed some unnecessary include files and removed the "//services/network/public/cpp"
> dep from "components/metrics/BUILD.gn". These include files and dep caused the
> build error Cronet for Android.
> 
> TBR=olivierrobin@chromium.org, michaelbai@chromium.org, jam@chromium.org, fdoray@chromium.org, asvitkine@chromium.org
> 
> Bug:  883121 
> Change-Id: Ifcaf40ea1b72ddc3a6a72be16e4d52190740e5a4
> Reviewed-on: https://chromium-review.googlesource.com/c/1320710
> Reviewed-by: Jun Cai <juncai@chromium.org>
> Commit-Queue: Jun Cai <juncai@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#605899}

TBR=michaelbai@chromium.org,jam@chromium.org,fdoray@chromium.org,asvitkine@chromium.org,olivierrobin@chromium.org,juncai@chromium.org

Change-Id: I37832c49552e6d163439d719d5820e1811ccadcb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  883121 
Reviewed-on: https://chromium-review.googlesource.com/c/1324104
Reviewed-by: Jun Cai <juncai@chromium.org>
Commit-Queue: Jun Cai <juncai@chromium.org>
Cr-Commit-Position: refs/heads/master@{#606136}
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/android_webview/browser/aw_metrics_service_client.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/chrome/browser/metrics/chrome_metrics_service_client.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/chromecast/browser/metrics/cast_metrics_service_client.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/components/metrics/BUILD.gn
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/components/metrics/metrics_service.h
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/components/metrics/net/DEPS
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/components/metrics/net/network_metrics_provider.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/components/metrics/net/network_metrics_provider.h
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/components/metrics/net/network_metrics_provider_unittest.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/content/browser/network_service_instance.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/content/public/browser/network_service_instance.h
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/ios/chrome/browser/metrics/ios_chrome_metrics_service_client.mm
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/services/network/public/cpp/network_connection_tracker.h
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/services/network/test/test_network_connection_tracker.cc
[modify] https://crrev.com/fc45269c31186c63cfa2b99921593806b9b54c36/services/network/test/test_network_connection_tracker.h

I couldn't reproduce this browser test failure locally on my Linux machine, but it shows up on the Builder linux64, not sure why. Any suggestion? Thanks!

Some update:
I built the Official Chrome build following the instructions at:
https://www.chromium.org/developers/gn-build-configuration
since the above bot builds an official build. And I also appended "testing/xvfb.py" before the command when running the test to simulate what the bot does, the above test still succeeded on my local linux machine.

Given that this browser test failure couldn't be reproduced locally and only happens on this particular bot, do you think it makes sense to mark this test disabled?

sky@ and cduvall@, since you made changes to this tests before, what do you think?

This test is related to //components/variations, asvitkine@, isherman@, rkaplow@, what do you think?

Thanks a lot for the suggestions!

"Given that this browser test failure couldn't be reproduced locally and only happens on this particular bot, do you think it makes sense to mark this test disabled?"

I don't think we should disable tests if it's clearly related to a CL. We should fix it instead. Or revert the CL.

Have you checked if there's a corresponding try bot with the same configuration? If so, you could potentially debug with print statements & try jobs. Alternatively, you could talk to a trooper and see if you can be granted SSH access to the bot to try to repro there, if that's possible.

Note that over the past couple of months network service related changes caused 3 separate breakages related to variations and metrics which took a lot of time for us to investigate, get fixed and write postmortems for[1][2][3].

So at this point I'm going to be pretty insistent that we don't just keep pushing through and breaking things. Let's take the time to understand the cause and fix the issue.

[1] https://docs.google.com/document/d/16vupsPw1SobMfuScN7F1Z6M-JI4W3ldrOxInyQ7F1A4/edit
[2] https://docs.google.com/document/d/1E0Tzp5nUuaJfJ7Et9p0zEApQoUm21YeyePvZxU4owlU/edit
[3]  http://crbug.com/896696 

+1 to what asvitkine says. Please revert until the problem is understood.

Makes sure you build with the same args.gn as the bot, as well as running with the same command line flags as the bot. If that still doesn't trigger the failure, someone in infra could likely help you find a machine in the lab that should be similar to what is used for running (try dpranke, he could point you in the right direction).

(btw, the cl has already been reverted).

cc dpranke@.

Tried the same args.gn as the bot:

concurrent_links = 6
is_chrome_branded = true
is_debug = false
is_official_build = true

https://logs.chromium.org/logs/chrome/bb/official.desktop.continuous/linux64_trunk/13609/+/recipes/steps/generate_build_files/0/stdout

and running with the same command flags (--test-launcher-bot-mode --test-launcher-summary-output=/tmp/tmp2oy9Yd.json) as:
https://logs.chromium.org/logs/chrome/bb/official.desktop.continuous/linux64_trunk/13553/+/recipes/steps/browser_tests/0/stdout

Couldn't reproduce the browser test failure locally.

I also tried running a subset of the tests using: --gtest_filter=Chrome*, and tried a couple of times, still couldn't reproduce the test failure locally.

I guess the next step is to see if I can ssh access to that bot, or borrow a machine which has similar configuration as the bot for debugging. dpranke@, what do you think? Thanks a lot!

This issue is marked as a release blocker with no OS labels associated. Please add an appropriate OS label.

All release blocking issues should have OS labels associated to it, so that the issue can tracked and promptly verified, once it gets fixed.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Are we sure the CL in question is the cause?

Because we're also seeing it here:
https://bugs.chromium.org/p/chromium/issues/detail?id=904108

Which found a different culprit.

+carlosil

Perhaps the test is merely flaky and both CLs make it more flakier?

Yeah, there's an issue with the test itself, this CL just made it worse. juncai@ is fixing the test in crrev.com/c/1323658.

A separate CL is created for fixing this browser test:
https://chromium-review.googlesource.com/c/chromium/src/+/1336433

Re #17: It seems  crbug.com/904108  is a different type of flakiness, since this one is causing the final request count to be == to the initial one, while on the other one the final count is initial+2 (instead of the expected initial+1).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/be06a8d6e480874d37a35d8bcf02c06a9b17dba6

commit be06a8d6e480874d37a35d8bcf02c06a9b17dba6
Author: Jun Cai <juncai@chromium.org>
Date: Thu Nov 15 19:01:41 2018

Fix ChromeBrowserMainBrowserTest.VariationsServiceStartsRequestOnNetworkChange

The browsertest changes in the following CL is moved to this CL since it
is not related to the following CL:
https://chromium-review.googlesource.com/c/chromium/src/+/1323658/7/chrome/browser/chrome_browser_main_browsertest.cc

This CL fixes the ChromeBrowserMainBrowserTest.VariationsServiceStartsRequestOnNetworkChange
browser test which failed on the release Builder linux64 machine. For
more details, please refer to the following issue link.

In the release Builder linux64 machine, the intermediate connection state
is CONNECTION_ETHERNET, so in the test it needs to make sure the
NetworkConnectionTracker's underlying network connection state is
CONNECTION_NONE before it starts to run the actual test code.

Bug:  902784 
Change-Id: I16866f1fb0e8834dfa621930efa83aba26aa52d0
Reviewed-on: https://chromium-review.googlesource.com/c/1336433
Commit-Queue: Jun Cai <juncai@chromium.org>
Reviewed-by: Scott Violet <sky@chromium.org>
Reviewed-by: Ilya Sherman <isherman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#608457}
[modify] https://crrev.com/be06a8d6e480874d37a35d8bcf02c06a9b17dba6/chrome/browser/chrome_browser_main_browsertest.cc