Issue metadata
Sign in to add a comment
|
network_service_browser_tests very flaky in component builds (e.g. on ToTLinux) |
||||||||||||||||||||||||
Issue descriptionToTLinux builds everything with trunk clang and does component builds. We've rolled clang several times over the window mentioned below, so the issues here must be due to the component build. 102 of the last 200 builds have network_service_browser_tests failing: https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/?limit=200 Things were consistently green from https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/3119 to https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/3128 . Then compile was broken for a bit for unrelated reasons, and since then network_service_browser_tests has been failing (first, a few other suites were failing too, but after https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/3200 it's been almost exclusively network_service_browser_tests The list of failing tests varies a bit each time. Here's the union of failures from the last 3 builds I looked at: BackgroundXhrTest.HttpAuth BackgroundXhrTest.TlsClientAuth DeclarativeApiTest.NoTracesAfterUninstalling DeclarativeContentApiTest.ReusedActionInstance DeclarativeContentApiTest.UninstallWhileActivePageAction NTPInterceptionWebRequestAPITest.NTPRendererRequestsHidden SetIconAPITest.Overview ShowPageActionWithoutPageActionTest/ShowPageActionWithoutPageActionTest.Test/0 Most stacks look like so: BrowserTestBase received signal: Terminated. Backtrace: #0 0x7fec1aae466c base::debug::StackTrace::StackTrace() #1 0x000002dc88c5 content::(anonymous namespace)::DumpStackTraceSignalHandler() #2 0x7fec0eeb1cb0 <unknown> #3 0x7fec0ef6bc9d __poll #4 0x7fec10aaffe4 <unknown> #5 0x7fec10ab00ec g_main_context_iteration #6 0x7fec1aa506e2 base::MessagePumpGlib::Run() #7 0x7fec1aa75625 base::RunLoop::Run() #8 0x000002dcdc21 content::DOMMessageQueue::WaitForMessage() #9 0x000002dcb56e content::(anonymous namespace)::ExecuteScriptHelper() #10 0x000002dcb7cb content::ExecuteScriptAndExtractBool() #11 0x000001f143e8 extensions::ExtensionBrowserTest::NavigateInRenderer() #12 0x000001c5b067 extensions::(anonymous namespace)::DeclarativeContentApiTest_ReusedActionInstance_Test::RunTestOnMainThread() #13 0x000002dc7e48 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #14 0x0000029cd39e ChromeBrowserMainParts::PreMainMessageLoopRunImpl() #15 0x0000029cc041 ChromeBrowserMainParts::PreMainMessageLoopRun() #16 0x7fec172bd681 content::BrowserMainLoop::PreMainMessageLoopRun() #17 0x7fec17647907 content::StartupTaskRunner::RunAllTasksNow() #18 0x7fec172bc094 content::BrowserMainLoop::CreateStartupTasks() #19 0x7fec172bfe33 content::BrowserMainRunnerImpl::Initialize() #20 0x7fec172b9f24 content::BrowserMain() #21 0x7fec17aac049 content::ContentMainRunnerImpl::Run() #22 0x7fec159076fb service_manager::Main() One looks like so: ../../chrome/browser/extensions/api/web_request/web_request_apitest.cc:1582: Failure Value of: was_ntp_script_loaded(web_contents) Actual: false Expected: true Stack trace: #0 0x0000024e07ec testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop() #1 0x0000024e01c9 testing::internal::AssertHelper::operator=() #2 0x000001e34eea extensions::NTPInterceptionWebRequestAPITest_NTPRendererRequestsHidden_Test::RunTestOnMainThread() #3 0x000002dc4d48 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #4 0x0000029ca29e ChromeBrowserMainParts::PreMainMessageLoopRunImpl() #5 0x0000029c8f41 ChromeBrowserMainParts::PreMainMessageLoopRun() #6 0x7fc65921a5c1 content::BrowserMainLoop::PreMainMessageLoopRun() #7 0x7fc6595a4a17 content::StartupTaskRunner::RunAllTasksNow() #8 0x7fc659218fd4 content::BrowserMainLoop::CreateStartupTasks() #9 0x7fc65921cf43 content::BrowserMainRunnerImpl::Initialize() #10 0x7fc659216e64 content::BrowserMain() #11 0x7fc659a09169 content::ContentMainRunnerImpl::Run() #12 0x7fc6578646fb service_manager::Main() #13 0x7fc659a074d1 content::ContentMain() #14 0x000002dc493c content::BrowserTestBase::SetUp() #15 0x000002974aeb InProcessBrowserTest::SetUp()
,
Aug 6
,
Aug 6
ncarter and lukasza touched browser_test_utils.cc around the regression date
,
Aug 6
,
Aug 6
,
Aug 6
These tests are still flaking a lot, upping priority and marking it as `assigned`.
,
Aug 6
engedy: can you link to bots were you saw that?
,
Aug 6
r580702 was indeed related to DOMMessageQueue / ExecuteScriptAndExtractBool. It is not clear to me how the presence of NetworkService can impact things here, but maybe we want to try a speculative revert? WDYT nick@?
,
Aug 6
+jam@ since this is related to the NetworkService
,
Aug 6
Blamelist for 576588...577071 is: https://chromium.googlesource.com/chromium/src/+log/8a51776b..9960a1b2?pretty=fuller&n=10000 Although it's definitely on the stack, it's hard to imagine how this change could be due to r580702 -- the changes to ExecuteScriptHelper were just plumbing in a world_id parameter that this codepath doesn't use. https://chromium-review.googlesource.com/c/chromium/src/+/1125121/30/content/public/test/browser_test_utils.cc#169 Adding +cduvall because r577036 touched the network service, and some of these failures look to be webrequest api related.
,
Aug 6
It looks like these failures are probably due to r577036, since the tests all seem to be related to webRequest/declarativeWebRequest/declarativeNetRequest. For some reason these are behaving differently with component builds. I can look into fixing these.
,
Aug 6
,
Aug 6
Put up http://crrev.com/c/1163987 which should fix these flakes.
,
Aug 6
,
Aug 6
Thanks! Note that I just looked though the top 3 builds to find that test list. Opening 3 random other ones finds these additional tests: DeclarativeApiTest.ExtensionLifetimeRulesHandling (on https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/3278) So consider writing a small script going through the last 200 builds and collecting all the flaking tests.
,
Aug 6
Wrote a script looking at the last 200 builds to collect flakes from network_service_browser_tests, here are the flakes: BackgroundXhrTest.HttpAuth BackgroundXhrTest.TlsClientAuth CaptivePortalBrowserTest.SSLCertErrorLogin CaptivePortalBrowserTest.ShowCaptivePortalInterstitialOnCertError DeclarativeApiTest.ExtensionLifetimeRulesHandling DeclarativeApiTest.NoTracesAfterUninstalling DeclarativeApiTest.PRE_PersistRules DeclarativeContentApiTest.ReusedActionInstance DeclarativeContentApiTest.UninstallWhileActivePageAction DevToolsFrontendInWebRequestApiTest.HiddenRequests DownloadTest.FeedbackServiceKeepDownload InterstitialAccessibilityBrowserTest.TestSSLInterstitialAccessibility LoginPromptBrowserTest.CancelLoginInterstitialOnRedirect LoginPromptBrowserTest.ShouldReplaceExistingInterstitialWhenNavigated LoginPromptBrowserTest.ShowCorrectUrlForCrossOriginMainFrameRedirects NTPInterceptionWebRequestAPITest.NTPRendererRequestsHidden SSLUIDynamicInterstitialTest.MismatchCommonNameRegex SSLUIDynamicInterstitialTest.MismatchHash SSLUIDynamicInterstitialTest.MismatchOrganizationRegex SetIconAPITest.Overview I believe my change should fix all the webrequest related tests, the others do not seem related to my change. They also may have been fixed, since the last few builds seem to only be failing the webrequest tests.
,
Aug 6
Any idea why the component build would be behaving differently? Is it just timing? (We don't wind up with two copies of some global, do we?)
,
Aug 6
As far as I can tell it's just timing. I can look around a bit more to see if there could be an issue with copies of a global.
,
Aug 6
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e commit 30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e Author: Clark DuVall <cduvall@chromium.org> Date: Mon Aug 06 20:50:41 2018 Fix flaky webrequest related network service tests in component builds These tests needed some extra synchronization due to the changes made in http://crrev.com/c/1139048. These were flaking in component builds. Bug: 871224 Change-Id: Ibe370f9b9c8ebb94ee1dd21ba204c06e91518057 Reviewed-on: https://chromium-review.googlesource.com/1163987 Commit-Queue: Clark DuVall <cduvall@chromium.org> Reviewed-by: John Abd-El-Malek <jam@chromium.org> Cr-Commit-Position: refs/heads/master@{#580980} [modify] https://crrev.com/30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e/chrome/browser/extensions/api/declarative/declarative_apitest.cc [modify] https://crrev.com/30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e/chrome/browser/extensions/api/declarative_content/declarative_content_apitest.cc [modify] https://crrev.com/30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e/chrome/browser/extensions/api/declarative_content/set_icon_apitest.cc [modify] https://crrev.com/30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e/chrome/browser/extensions/api/web_request/web_request_apitest.cc [modify] https://crrev.com/30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e/chrome/browser/extensions/background_xhr_browsertest.cc
,
Aug 7
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1c472ffdfc0d6ba628f08e08c4558246f98c6bd3 commit 1c472ffdfc0d6ba628f08e08c4558246f98c6bd3 Author: Rune Lillesveen <futhark@chromium.org> Date: Tue Aug 07 11:26:06 2018 Revert "Fix flaky webrequest related network service tests in component builds" This reverts commit 30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e. Reason for revert: Made SetIconAPITest.Overview test flaky (871720) Original change's description: > Fix flaky webrequest related network service tests in component builds > > These tests needed some extra synchronization due to the changes made in > http://crrev.com/c/1139048. These were flaking in component builds. > > Bug: 871224 > Change-Id: Ibe370f9b9c8ebb94ee1dd21ba204c06e91518057 > Reviewed-on: https://chromium-review.googlesource.com/1163987 > Commit-Queue: Clark DuVall <cduvall@chromium.org> > Reviewed-by: John Abd-El-Malek <jam@chromium.org> > Cr-Commit-Position: refs/heads/master@{#580980} TBR=jam@chromium.org,cduvall@chromium.org Change-Id: If7bb80ef219f5aa5107ffd012981839dfaceac76 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 871224 , 871720 Reviewed-on: https://chromium-review.googlesource.com/1165122 Reviewed-by: Rune Lillesveen <futhark@chromium.org> Commit-Queue: Rune Lillesveen <futhark@chromium.org> Cr-Commit-Position: refs/heads/master@{#581195} [modify] https://crrev.com/1c472ffdfc0d6ba628f08e08c4558246f98c6bd3/chrome/browser/extensions/api/declarative/declarative_apitest.cc [modify] https://crrev.com/1c472ffdfc0d6ba628f08e08c4558246f98c6bd3/chrome/browser/extensions/api/declarative_content/declarative_content_apitest.cc [modify] https://crrev.com/1c472ffdfc0d6ba628f08e08c4558246f98c6bd3/chrome/browser/extensions/api/declarative_content/set_icon_apitest.cc [modify] https://crrev.com/1c472ffdfc0d6ba628f08e08c4558246f98c6bd3/chrome/browser/extensions/api/web_request/web_request_apitest.cc [modify] https://crrev.com/1c472ffdfc0d6ba628f08e08c4558246f98c6bd3/chrome/browser/extensions/background_xhr_browsertest.cc
,
Aug 7
Good news: While that CL was in, the component bot has been consistently green: https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/?limit=200
,
Aug 7
Issue 871601 has been merged into this issue.
,
Aug 7
,
Aug 8
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/444729a60f6700e82a9385392d584f1814cd9129 commit 444729a60f6700e82a9385392d584f1814cd9129 Author: Clark DuVall <cduvall@chromium.org> Date: Wed Aug 08 23:09:54 2018 Reland "Fix flaky webrequest related network service tests in component builds" This is a reland of 30d8a3b9d66558f01b026c11ffe7db2bbda6fc1e This fixes an issue where FlushForTesting would timeout if the InterfacePtr was destroyed in the middle of a flush. Also fixes a webview test that was flaky on component builds. Original change's description: > Fix flaky webrequest related network service tests in component builds > > These tests needed some extra synchronization due to the changes made in > http://crrev.com/c/1139048. These were flaking in component builds. > > Bug: 871224 > Change-Id: Ibe370f9b9c8ebb94ee1dd21ba204c06e91518057 > Reviewed-on: https://chromium-review.googlesource.com/1163987 > Commit-Queue: Clark DuVall <cduvall@chromium.org> > Reviewed-by: John Abd-El-Malek <jam@chromium.org> > Cr-Commit-Position: refs/heads/master@{#580980} Bug: 871224 , 871720 Change-Id: Ib66c759ef422124f166583c57f89f0a56d92f841 Reviewed-on: https://chromium-review.googlesource.com/1165504 Reviewed-by: John Abd-El-Malek <jam@chromium.org> Reviewed-by: Ken Rockot <rockot@chromium.org> Commit-Queue: Clark DuVall <cduvall@chromium.org> Cr-Commit-Position: refs/heads/master@{#581721} [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/chrome/browser/extensions/api/declarative/declarative_apitest.cc [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/chrome/browser/extensions/api/declarative_content/declarative_content_apitest.cc [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/chrome/browser/extensions/api/declarative_content/set_icon_apitest.cc [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/chrome/browser/extensions/api/web_request/web_request_apitest.cc [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/chrome/browser/extensions/background_xhr_browsertest.cc [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/extensions/test/data/web_view/apitest/main.js [modify] https://crrev.com/444729a60f6700e82a9385392d584f1814cd9129/mojo/public/cpp/bindings/lib/control_message_proxy.cc
,
Aug 9
Looks like the bot is green now, closing this. |
|||||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||||
Comment 1 by thakis@chromium.org
, Aug 6