Issue 834220  has been merged into this issue.

 Issue 833124  provides some data:

List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNySwsSBUZsYWtlIkBDaHJvbWVUcmFjaW5nRGVsZWdhdGVCcm93c2VyVGVzdE9uU3RhcnR1cC5TdGFydHVwVHJhY2luZ1Rocm90dGxlDA.

Seems to have started at try run at 2018-04-09 20:51:33 UTC a few days ago, and has been a persistent flake since that time.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests%20(with%20patch)&showAllRuns=true&tests=ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle

ie., with "show all runs" suggests this test has been flaking on linux-chomeos-rel for quite a while.

... well at least as far back as 2018-04-17. #2 still has the earliest report, namely, 2018-04-09.

+rune (who is maybe gardening) as an FYI.

The test failure is persistent on ChromeOS only, see:

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests%20(with%20patch)&tests=ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle

Trace log:

[ RUN      ] ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle
[10085:10085:0419/055017.718965:WARNING:chrome_browser_main_chromeos.cc(641)] Running as stub user with profile dir: test-user
[10085:10104:0419/055017.784060:ERROR:account_manager.cc(37)] Failed to read tokens file
[10085:10085:0419/055017.794887:WARNING:user_session_manager.cc(1084)] Attempting to save user password for non enterprise user.
[10085:10085:0419/055017.803048:WARNING:extension.cc(761)] Failed to load extension.  Manifest JSON: {"default_locale":"en","description":"Chrome OS Mobile Activation Resources","incognito":"split","key":"MIGdMA0GCSqGSIb3DQEBAQUAA4GLADCBhwKBgQDITTAJ8zmgRjcmCRIAUbzlrSSzoWPBRY/kaBP3mxZ8nlCMljhQLDDKClnyGK6+krYq8/LayF4r0PR7g0Q27lwvw+asg1BFN+AL+suJqe4jJzxBK3GaMBRsBCCcwpAQw4F82zI9Hk0e6EqfrDq4ePNgv+PsI2MDbOuk3b1vS5oBjQIBIw==","name":"Mobile Activation","permissions":[],"version":"1.0"}
[10085:10085:0419/055017.803075:ERROR:component_loader.cc(256)] The 'manifest_version' key must be present and set to 2 (without quotes). See developer.chrome.com/extensions/manifestVersion.html for details.
[10085:10085:0419/055017.816197:WARNING:extension.cc(761)] Failed to load extension.  Manifest JSON: {"default_locale":"en","description":"Chrome OS Mobile Activation Resources","incognito":"split","key":"MIGdMA0GCSqGSIb3DQEBAQUAA4GLADCBhwKBgQDITTAJ8zmgRjcmCRIAUbzlrSSzoWPBRY/kaBP3mxZ8nlCMljhQLDDKClnyGK6+krYq8/LayF4r0PR7g0Q27lwvw+asg1BFN+AL+suJqe4jJzxBK3GaMBRsBCCcwpAQw4F82zI9Hk0e6EqfrDq4ePNgv+PsI2MDbOuk3b1vS5oBjQIBIw==","name":"Mobile Activation","permissions":[],"version":"1.0"}
[10085:10085:0419/055017.816233:ERROR:component_loader.cc(256)] The 'manifest_version' key must be present and set to 2 (without quotes). See developer.chrome.com/extensions/manifestVersion.html for details.
[10085:10085:0419/055017.856117:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.
[10085:10085:0419/055017.856332:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.
[10085:10085:0419/055017.856672:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.
[10085:10085:0419/055017.856738:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.
[10085:10096:0419/055018.003671:WARNING:discardable_shared_memory_manager.cc(436)] Some MojoDiscardableSharedMemoryManagerImpls are still alive. They will be leaked.
[10085:10152:0419/055018.006733:WARNING:internal_linux.cc(64)] Failed to read /proc/10114/stat
[10085:10100:0419/055018.002608:FATAL:handle.h(98)] Check failed: handle_.is_valid().
#0 0x00000442068c base::debug::StackTrace::StackTrace()
#1 0x00000443b5ab logging::LogMessage::~LogMessage()
#2 0x000004add8fb mojo::BlockingCopyFromString()
#3 0x00000321c55b tracing::Coordinator::TraceStreamer::StreamEventsForCurrentLabel()
#4 0x00000321bc91 tracing::Coordinator::TraceStreamer::OnRecorderDataChange()
#5 0x0000019dfa70 _ZN4base8internal7InvokerINS0_9BindStateIMN8chromeos22ManagerPasswordServiceEFvRKNS3_11UserContextEEJNS_7WeakPtrIS4_EES5_EEEFvvEE3RunEPNS0_13BindStateBaseE
#6 0x00000321d8b8 tracing::Recorder::AddChunk()
#7 0x00000695fac6 tracing::mojom::RecorderStubDispatch::Accept()
#8 0x0000054d8be6 mojo::InterfaceEndpointClient::HandleValidatedMessage()
#9 0x0000054d87d6 mojo::FilterChain::Accept()
#10 0x0000054d9f25 mojo::InterfaceEndpointClient::HandleIncomingMessage()
#11 0x0000054dfc1c mojo::internal::MultiplexRouter::ProcessIncomingMessage()
#12 0x0000054df020 mojo::internal::MultiplexRouter::Accept()
#13 0x0000054d87d6 mojo::FilterChain::Accept()
#14 0x0000054d69a3 mojo::Connector::ReadSingleMessage()
#15 0x0000054d72c1 mojo::Connector::ReadAllAvailableMessages()
#16 0x0000054d7169 mojo::Connector::OnHandleReadyInternal()
#17 0x00000294e897 mojo::SimpleWatcher::DiscardReadyState()
#18 0x000004ae03d6 mojo::SimpleWatcher::OnHandleReady()
#19 0x000004ae08e1 _ZN4base8internal7InvokerINS0_9BindStateIMN4mojo13SimpleWatcherEFvijRKNS3_18HandleSignalsStateEEJNS_7WeakPtrIS4_EEijS5_EEEFvvEE7RunImplIRKS9_RKNSt3__15tupleIJSB_ijS5_EEEJLm0ELm1ELm2ELm3EEEEvOT_OT0_NSI_16integer_sequenceImJXspT1_EEEE
#20 0x000004420f25 base::debug::TaskAnnotator::RunTask()
#21 0x00000448efa5 base::internal::TaskTracker::RunOrSkipTask()
#22 0x000004490248 base::internal::TaskTrackerPosix::RunOrSkipTask()
#23 0x00000448e43f base::internal::TaskTracker::RunAndPopNextTask()
#24 0x0000044e1fab base::internal::SchedulerWorker::Thread::ThreadMain()
#25 0x00000449833f base::(anonymous namespace)::ThreadFunc()
#26 0x7f4803a06184 start_thread
#27 0x7f480073bffd clone

Looks like mojo and tracing::Coordinator::TraceStreamer::OnRecorderDataChange() don't agree about something or other.

Can you fix this today or disable the test, please?

 Issue 833124  has been merged into this issue.

Detected 17 new flakes for test/step "ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNySwsSBUZsYWtlIkBDaHJvbWVUcmFjaW5nRGVsZWdhdGVCcm93c2VyVGVzdE9uU3RhcnR1cC5TdGFydHVwVHJhY2luZ1Rocm90dGxlDA. This message was posted automatically by the chromium-try-flakes app.

I suspect there's a race going on.

Coordinator::TraceStreamer::CloseStream() closes |stream_| on |main_task_runner_|.

Coordinator::TraceStreamer::OnRecorderDataChange() uses |stream_| on |background_task_runner_|.

The use of |stream_| should be via a non-owning DataPipeProducerHandle so it remains valid, albeit referencing a closed handle, when the owning handle is reset.

Thanks Sam, seems right to me and suggests a plausible fix.  Assigning this issue to a tracing owner for triage and fix.

#8 > Can you fix this today or disable the test, please?

Yes, I will disable.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5e2385bbaeb05a4a9fdcd0675c3b0634eca4fc0f

commit 5e2385bbaeb05a4a9fdcd0675c3b0634eca4fc0f
Author: Noel Gordon <noel@chromium.org>
Date: Fri Apr 20 13:07:15 2018

[ChromeOS] Disable ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle

StartupTracingThrottle flakes on a Mojo CHECK stop on ChromeOS. Seems different
threads access or close a shared Mojo handle on shutdown. The loser thread hits
a Mojo "handle invalid" CHECK stop, causing a browser test failure.

Tbr: mstensho
Bug: 832981
Change-Id: I434c36cb54b1dfa90b8518210684c0516a9be467
Reviewed-on: https://chromium-review.googlesource.com/1021751
Reviewed-by: Noel Gordon <noel@chromium.org>
Commit-Queue: Noel Gordon <noel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552312}
[modify] https://crrev.com/5e2385bbaeb05a4a9fdcd0675c3b0634eca4fc0f/chrome/browser/tracing/chrome_tracing_delegate_browsertest.cc

Detected 3 new flakes for test/step "ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNySwsSBUZsYWtlIkBDaHJvbWVUcmFjaW5nRGVsZWdhdGVCcm93c2VyVGVzdE9uU3RhcnR1cC5TdGFydHVwVHJhY2luZ1Rocm90dGxlDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Even after #14 disabled the test on ChromeOS, I see it flake on mash browser tests (with patch) linux-chromeos-rel [1].

[1] https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/linux-chromeos-rel/106432

Sending a patch to disable also via the mash browser test config.

... with the same problem cause, for the record.: snippet from the build 106432 mentioned in #19.

[25518:25533:0420/065857.229064:FATAL:handle.h(98)] Check failed: handle_.is_valid().
#0 0x00000443a85c base::debug::StackTrace::StackTrace()
#1 0x00000445577b logging::LogMessage::~LogMessage()
#2 0x000004af81db mojo::BlockingCopyFromString()
#3 0x00000322f59b tracing::Coordinator::TraceStreamer::StreamEventsForCurrentLabel()
#4 0x00000322ecd1 tracing::Coordinator::TraceStreamer::OnRecorderDataChange()
#5 0x0000019ed4c0 _ZN4base8internal7InvokerINS0_9BindStateIMN8chromeos22ManagerPasswordServiceEFvRKNS3_11UserContextEEJNS_7WeakPtrIS4_EES5_EEEFvvEE3RunEPNS0_13BindStateBaseE
#6 0x0000032308f8 tracing::Recorder::AddChunk()
#7 0x00000697a256 tracing::mojom::RecorderStubDispatch::Accept()
#8 0x0000054ef7c6 mojo::InterfaceEndpointClient::HandleValidatedMessage()
#9 0x0000054ef3b6 mojo::FilterChain::Accept()
...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f4a30da72d5bab1f316ea8561b6112968300c771

commit f4a30da72d5bab1f316ea8561b6112968300c771
Author: Noel Gordon <noel@chromium.org>
Date: Mon Apr 23 03:48:37 2018

[Mash] Disable ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle

Flakes in mash browser test (with patch) on linux-chromeos-rel.
See crbug.com/832981#c19

Tbr: yoichi
Bug: 832981
Change-Id: Icb9361b457f2843e9a181f0421064ee10a65bb5a
Reviewed-on: https://chromium-review.googlesource.com/1023492
Reviewed-by: Noel Gordon <noel@chromium.org>
Commit-Queue: Noel Gordon <noel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552628}
[modify] https://crrev.com/f4a30da72d5bab1f316ea8561b6112968300c771/testing/buildbot/filters/mash.browser_tests.filter

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c54cf063f7c499a6c6ea18a4b7739125f45eefde

commit c54cf063f7c499a6c6ea18a4b7739125f45eefde
Author: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Date: Wed Apr 25 16:10:20 2018

Tracing: fix a race if streaming at shutdown

Bug: 832981
Change-Id: I3711f0f03306a74b9a062f388beb2f3915eecb91
Reviewed-on: https://chromium-review.googlesource.com/1024969
Commit-Queue: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Reviewed-by: oysteine <oysteine@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553589}
[modify] https://crrev.com/c54cf063f7c499a6c6ea18a4b7739125f45eefde/chrome/browser/tracing/chrome_tracing_delegate_browsertest.cc
[modify] https://crrev.com/c54cf063f7c499a6c6ea18a4b7739125f45eefde/services/tracing/coordinator.cc
[modify] https://crrev.com/c54cf063f7c499a6c6ea18a4b7739125f45eefde/testing/buildbot/filters/mash.browser_tests.filter

Detected 3 new flakes for test/step "ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNySwsSBUZsYWtlIkBDaHJvbWVUcmFjaW5nRGVsZWdhdGVCcm93c2VyVGVzdE9uU3RhcnR1cC5TdGFydHVwVHJhY2luZ1Rocm90dGxlDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

I'm going to remove this from the sheriff queue since a fix landed after the most recent flakes. If it flakes again post-fix, it should get added back to the queue. 

The test is still flaky, with a different error (timeout simetimes). I'll disable it again, in crbug.com/c/1028730, until I find a fix.

crbug.com/c/1028730 =>
  https://chromium-review.googlesource.com/c/chromium/src/+/1028730 LGTM

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4c9b4be38aa735966e706bf44dc3b6d95231e286

commit 4c9b4be38aa735966e706bf44dc3b6d95231e286
Author: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Date: Wed Apr 25 22:38:19 2018

Tracing: disable StartupTracingThrottle, again

It is still flaky:

https://logs.chromium.org/v/?s=chromium%2Fbb%2Ftryserver.chromium.chromiumos%2Flinux-chromeos-rel%2F109818%2F%2B%2Frecipes%2Fsteps%2Fmash_browser_tests__with_patch_%2F0%2Fstdout

Bug: 832981
Change-Id: Idd9646c0bfe7ebec3c1ff9068c39d11ba02faf67
Reviewed-on: https://chromium-review.googlesource.com/1028730
Commit-Queue: Noel Gordon <noel@chromium.org>
Reviewed-by: oysteine <oysteine@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553798}
[modify] https://crrev.com/4c9b4be38aa735966e706bf44dc3b6d95231e286/chrome/browser/tracing/chrome_tracing_delegate_browsertest.cc
[modify] https://crrev.com/4c9b4be38aa735966e706bf44dc3b6d95231e286/testing/buildbot/filters/mash.browser_tests.filter

The issue has failed on win 7 Tests as well:

https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%281%29/builds/79553

Detected 4 new flakes for test/step "ChromeTracingDelegateBrowserTestOnStartup.StartupTracingThrottle". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNySwsSBUZsYWtlIkBDaHJvbWVUcmFjaW5nRGVsZWdhdGVCcm93c2VyVGVzdE9uU3RhcnR1cC5TdGFydHVwVHJhY2luZ1Rocm90dGxlDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

I'll disable the test on all platforms, for now: crrev.com/c/1033343.

removing sheriff label given #31

Issue 837604 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f7819ae0915e9e9f331d151c770e362c98488956

commit f7819ae0915e9e9f331d151c770e362c98488956
Author: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Date: Fri Apr 27 21:08:11 2018

Tracing: disable a flaky test on all platforms

Bug: 832981
Change-Id: I230384753592e4397d164460a9cc953c17b10daf
Reviewed-on: https://chromium-review.googlesource.com/1033343
Reviewed-by: oysteine <oysteine@chromium.org>
Commit-Queue: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554508}
[modify] https://crrev.com/f7819ae0915e9e9f331d151c770e362c98488956/chrome/browser/tracing/chrome_tracing_delegate_browsertest.cc

Reporting to stale-flakes-reports@google.com to investigate why this issue has been in the appropriate queue 5 times or more.