Issue 822397  has been merged into this issue.

content_browsertests content_browsertests
Run on OS: 'Ubuntu-14.04'
Max shard duration: 0:12:36.867820 (shard #4)
failures:
SavableResourcesTest.GetSavableResourceLinksWithPageHasValidLinks
SitePerProcessMouseWheelHitTestBrowserTest.MainframeWheelEventsOnMainThread/0
SitePerProcessMouseWheelHitTestBrowserTest.MainframeWheelEventsOnMainThread/1
SavableResourcesTest.GetSavableResourceLinksWithPageHasInvalidLinks
RenderThreadImplDiscardableMemoryBrowserTest.ReleaseFreeDiscardableMemory
RenderThreadImplDiscardableMemoryBrowserTest.LockDiscardableMemory
DomSerializerTests.SerializeHTMLDOMWithEntitiesInText
SitePerProcessMouseWheelHitTestBrowserTest.SubframeWheelEventsOnMainThread/0
DomSerializerTests.SerializeHTMLDOMWithoutDocType
SitePerProcessMouseWheelHitTestBrowserTest.SubframeWheelEventsOnMainThread/1

Suspect https://chromium-review.googlesource.com/961264

crouleau@ has reverted the suspected cl (https://chromium-review.googlesource.com/961264) in #19156 and #19157 has still failed after the revert.

Okay. You can feel free to reland if my revert of your change didn't fix it.

We will need to disable the tests and find owners for them.

My previous comment wasn't clear enough, I apologize:
https://chromium-review.googlesource.com/961264 is reverted because I changed a flaky layout test and removed its test expectations hoping that my change will fix the flake and it didn't, so the reverting is valid:) and I'll reland the patch without updating the test expectations.

But https://chromium-review.googlesource.com/961264 is not cause of the content_browsertests failures on TSAN since the bot is still failing after the revert.

Looking at the TSAN output:

  Read of size 8 at 0x7b1400039b70 by thread T2:
    #4 base::FeatureList::IsEnabled

  Previous write of size 8 at 0x7b1400039b70 by main thread:
    #10 base::FeatureList::RegisterOverride

This appears to be a test calling RegisterOverride *after* another thread has been spawned which then checks if a feature is enabled.

FeatureList is *not* thread-safe.  It must be fully initialized before other threads start.

Still looking for the change that is triggering this...

Here's the issue:

1) During startup for browsertests, the IO thread is created.

2) A test utilizes test::ScopedFeatureList to change features.
https://cs.chromium.org/chromium/src/content/browser/site_per_process_hit_test_browsertest.cc?rcl=67d30672a641a6c065f92d494243f66ff5db21ef&l=1890

3) The IO thread tests a feature.
https://cs.chromium.org/chromium/src/content/shell/browser/shell_url_request_context_getter.cc?rcl=67d30672a641a6c065f92d494243f66ff5db21ef&l=213

4) TSAN error.


Both of the linked changes appeared in late January but didn't cause a problem until today.  The test on the IO thread is wrapped by
#if BUILDFLAG(ENABLE_REPORTING)

so my working theory is that this has just become enabled.

I think ScopedFeatureList should be set up in the browser test ctor, rather than the test itself.

That seems reasonable but it doesn't explain how that change, submitted on Jan 31st, just started crashing yesterday.

Found another...

1) IO thread is initializing ChromeAppCacheService and does some SSL stuff.
2) Main thread has changed the SSL stuff after the IO thread was launched.

There's no #if here.  New working theory:  BrowserTest startup order has changed so that the IO thread is launched earlier than it used to be.

 Issue 822468  has been merged into this issue.

I tried to reproduce locally and found when it started earlier, it fails even for older commits. 
I went to the last commit of 
https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20TSan%20Tests/builds/19153 (i.e. efs/heads/master@{#543382})
TSAN are still complaining (I tried tests DomSerializerTests.SerializeHTMLDOMWithEntitiesInText and SitePerProcessMouseWheelHitTestBrowserTest.MainframeWheelEventsOnMainThread/0), but complains are flaky, that's probably the reason why they were not seen earlier.

I tried building it on my workstation.  None of the tests would pass, let alone have TSAN errors.  But if the problems go back even further, perhaps I'll have better luck just looking around at HEAD.

Looking at the latest builds, there are many failures but none of them occur when I build and run them locally.  The fix for the one I can reproduce ( issue 822764 ) was reverted because the revert broke other tests.

I'm going to leave this for now until that issue can be resolved.

Are there any examples of TSan reports?

There is TSAN output in every failed build.

This appears resolved, in that I see passing builds now.

Close?

Yup.  I didn't check exactly when they returned to that status.