New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 642991 link

Starred by 2 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

"TracingControllerTest.NotWhitelistedMetadataStripped" is flaky

Project Member Reported by chromium...@appspot.gserviceaccount.com, Sep 1 2016

Issue description

"TracingControllerTest.NotWhitelistedMetadataStripped" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 7 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyPwsSBUZsYWtlIjRUcmFjaW5nQ29udHJvbGxlclRlc3QuTm90V2hpdGVsaXN0ZWRNZXRhZGF0YVN0cmlwcGVkDA.

Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
 
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/285399/steps/content_browsertests%20%28with%20patch%29%20on%20Windows-7-SP1/logs/TracingControllerTest.NotWhitelistedMetadataStripped

TracingControllerTest.NotWhitelistedMetadataStripped (run #1):
[ RUN      ] TracingControllerTest.NotWhitelistedMetadataStripped
[1440:4688:0831/171410:23328982:FATAL:trace_log.cc(1091)] Check failed: !argument_filter_predicate_.is_null().
Backtrace:
	base::debug::StackTrace::StackTrace [0x01C24067+23]
	logging::LogMessage::~LogMessage [0x01BCBCD1+49]
	base::trace_event::TraceLog::FinishFlush [0x01BF290F+255]
	base::internal::Invoker<base::internal::BindState<void (__thiscall base::trace_event::TraceLog::*)(int,bool),base::internal::UnretainedWrapper<base::trace_event::TraceLog>,int,bool>,void __cdecl(void)>::Run [0x01BF43A6+22]
	base::debug::TaskAnnotator::RunTask [0x01C40600+256]
	base::MessageLoop::RunTask [0x01BE24FB+1211]
	base::MessageLoop::DoWork [0x01BE172E+606]
	base::MessagePumpForIO::DoRunLoop [0x01C3574C+188]
	base::MessagePumpWin::Run [0x01C3655A+74]
	base::MessageLoop::RunHandler [0x01BE2037+103]
	base::RunLoop::Run [0x01BDE269+41]
	base::Thread::Run [0x01C14DDC+92]
	base::Thread::ThreadMain [0x01C15785+469]
	base::PlatformThread::Sleep [0x01BDFD02+290]
	BaseThreadInitThunk [0x7691337A+18]
	RtlInitializeExceptionChain [0x777D9882+99]
	RtlInitializeExceptionChain [0x777D9855+54]


https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/134304/steps/content_browsertests%20%28with%20patch%29%20on%20Android/logs/TracingControllerTest.NotWhitelistedMetadataStripped

TracingControllerTest.NotWhitelistedMetadataStripped (run #1):
[ RUN      ] TracingControllerTest.NotWhitelistedMetadataStripped
[WARNING:dns_config_service_posix.cc(316)] Failed to read DnsConfig.
[ERROR:devtools_http_handler.cc(226)] Cannot start http server for devtools. Stop devtools.
[FATAL:context_provider_factory_impl_android.cc(181)] Check failed: !in_shutdown_. The factory is shutting down, can't handle new requests
#00 0x758b2e67 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x005c2e67
#01 0x76fd31c7 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ce31c7
#02 0x76fd3489 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ce3489
#03 0x76fd34e7 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ce34e7
#04 0x76fd2915 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ce2915
#05 0x76da4789 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ab4789
#06 0x76da6db9 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ab6db9
#07 0x76da2bc3 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01ab2bc3
#08 0x7706a5c3 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01d7a5c3
#09 0x7706513b /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01d7513b
#10 0x770d1c83 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01de1c83
#11 0x770d1de1 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01de1de1
#12 0x770d1e69 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01de1e69
#13 0x770d3769 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01de3769
#14 0x770ced0f /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01dded0f
#15 0x7707ed71 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01d8ed71
#16 0x77067a7b /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01d77a7b
#17 0x77076e8d /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x01d86e8d
#18 0x755caa9b /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x002daa9b
#19 0x75896125 /data/app-lib/org.chromium.content_browsertests_apk-1/lib_content_browsertests__library.so+0x005a6125
#20 0x41531bcf /system/lib/libdvm.so+0x0001dbcf
#21 0x41562125 /system/lib/libdvm.so+0x0004e125
#22 0x4153afe3 /system/lib/libdvm.so+0x00026fe3
#23 0x41541fa3 /system/lib/libdvm.so+0x0002dfa3
#24 0x4153f63b /system/lib/libdvm.so+0x0002b63b
#25 0x41574863 /system/lib/libdvm.so+0x00060863
#26 0x4157c7c5 /system/lib/libdvm.so+0x000687c5
#27 0x4153afe3 /system/lib/libdvm.so+0x00026fe3
#28 0x41541fa3 /system/lib/libdvm.so+0x0002dfa3
#29 0x4153f63b /system/lib/libdvm.so+0x0002b63b
#30 0x4157457f /system/lib/libdvm.so+0x0006057f
#31 0x4155dd0d /system/lib/libdvm.so+0x00049d0d
#32 0x401d62b3 /system/lib/libandroid_runtime.so+0x0004d2b3
#33 0x401d6fd9 /system/lib/libandroid_runtime.so+0x0004dfd9
#34 0x4009905d /system/bin/app_process+0x0000105d
#35 0x400d034b /system/lib/libc.so+0x0000e34b
#36 0x40098d7f /system/bin/app_process+0x00000d7f

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/290319/steps/content_browsertests%20%28with%20patch%29%20on%20Ubuntu-12.04/logs/TracingControllerTest.NotWhitelistedMetadataStripped

TracingControllerTest.NotWhitelistedMetadataStripped (run #1):
[ RUN      ] TracingControllerTest.NotWhitelistedMetadataStripped
[32710:32710:0831/173039:9281809800:WARNING:audio_manager.cc(317)] Multiple instances of AudioManager detected
[32710:32710:0831/173039:9281809851:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
Xlib:  extension "RANDR" missing on display ":9".
[301:328:0831/173039:9281946701:FATAL:trace_log.cc(1091)] Check failed: !argument_filter_predicate_.is_null().
#0 0x00000200cc6e base::debug::StackTrace::StackTrace()
#1 0x000002021e5b logging::LogMessage::~LogMessage()
#2 0x000002085c12 base::trace_event::TraceLog::FinishFlush()
#3 0x000002094af6 base::debug::TaskAnnotator::RunTask()
#4 0x000002029085 base::MessageLoop::RunTask()
#5 0x0000020293c8 base::MessageLoop::DeferOrRunPendingTask()
#6 0x00000202970b base::MessageLoop::DoWork()
#7 0x00000202c259 base::MessagePumpLibevent::Run()
#8 0x000002028b81 base::MessageLoop::RunHandler()
#9 0x000002045cd0 base::RunLoop::Run()
#10 0x00000206676d base::Thread::Run()
#11 0x000002066b12 base::Thread::ThreadMain()
#12 0x000002060b75 base::(anonymous namespace)::ThreadFunc()
#13 0x7f88c74e2e9a start_thread
#14 0x7f88c261436d clone
Owner: primiano@chromium.org
Status: Assigned (was: Untriaged)
Primiano could you please find an owner for this?

Thanks. If I don't hear from you before, I'll disable the test when I get a chance.
Cc: dsinclair@chromium.org
Components: Tests>Disabled
Labels: -Sheriff-Chromium
CL to disable test is in CQ. Moving bug out of Sheriff queue.
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 1 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/30f869ca29ea81de0464779dec96d4453534c21f

commit 30f869ca29ea81de0464779dec96d4453534c21f
Author: battre <battre@chromium.org>
Date: Thu Sep 01 10:12:25 2016

Disable TracingControllerTest.NotWhitelistedMetadataStripped for flakiness

BUG=642991
TBR=primiano@chromium.org, dsinclair@chromium.org

Review-Url: https://codereview.chromium.org/2296863006
Cr-Commit-Position: refs/heads/master@{#415922}

[modify] https://crrev.com/30f869ca29ea81de0464779dec96d4453534c21f/content/browser/tracing/tracing_controller_browsertest.cc

Cc: -dsinclair@chromium.org primiano@chromium.org kraynov@chromium.org picksi@chromium.org
Labels: Hotlist-MemoryInfra
Owner: ssid@chromium.org
ssid can you see what's going on here?
Don't want to leave that test uncovered for too long. 

-dsinclair@chromium.org as this is clearly on me/ssid.

Comment 6 by ssid@chromium.org, Nov 29 2016

Cc: ssid@chromium.org oysteine@chromium.org
Owner: zh...@chromium.org
Assigning to Zhen who wrote the test.
Running the test locally fails all the time (not flaky).
TracingControllerTest.NotWhitelistedMetadataStripped starts a content_shell with multiple processes, but SetArgumentFilterPredicate is called only on browser process.
TraceLog::FinishFlush is called on all processes and argument filter is needed for ConvertTraceEventsToTraceFormat. So, I am not sure how thie test was running at all?
Hm as far as I can remember the filtering predicate should always be set by  ChromeMainDelegate::BasicStartupComplete for all Chrome processes but probably not for content_shell child processes, so that makes sense. Presumably what changed here is that no child process was started in the past in this test? Not sure why one would be started here now, we're not doing any navigations. Some non-renderer utility process?

Comment 8 by zh...@chromium.org, Nov 29 2016

I tried to print the process type in trace_log (https://codereview.chromium.org/2533333002/).

It seems the GPU process and renderer process does not have predicate set:

[22665:22673:1129/093516:3356961746456:WARNING:trace_log.cc(1033)] ============================= gpu-process ==== 0
[1:2:1129/093516:3356961752064:WARNING:trace_log.cc(1033)] ============================= renderer ==== 0

[22630:22630:1129/093516:3356961848033:WARNING:trace_log.cc(1033)] =============================  ==== 1

The last one I think is the test process.

But why do we have multiple processes here? I even used --single_process flag. Here is the command I used:

ninja -C out/Debug content_browsertests && ./out/Debug/content_browsertests --gtest_filter=TracingControllerTest.NotWhitelistedMetadataStripped --single_process

Comment 9 by ssid@chromium.org, Nov 29 2016

The predicate is being set by the test at TestStartAndStopTracingStringWithFilter().
Note that this is a content_browsertest and the ChromeMainDelegate::BasicStartupComplete is not called because it is in chrome.

Comment 10 by zh...@chromium.org, Dec 16 2017

Owner: ----
Status: Available (was: Assigned)
Components: Tests>Disabled
Labels: Test-Disabled
Owner: etienneb@chromium.org
Status: Started (was: Available)
For #c7, the GPUMain doesn't seems to call:
  ChromeMainDelegate::BasicStartupComplete(int* exit_code)

And the test was failing in this stackframe:

7923:197923:1121/151153.522324:FATAL:trace_log.cc(997)] Check failed: !argument_filter_predicate_.is_null().
#0 0x5566dc64db5f base::debug::StackTrace::StackTrace()
#1 0x5566dc5ca8d0 logging::LogMessage::~LogMessage()
#2 0x5566dc63daa0 base::trace_event::TraceLog::FinishFlush()
#3 0x5566dc5d10b8 base::debug::TaskAnnotator::RunTask()
#4 0x5566dc5d06bf base::MessageLoopImpl::RunTask()
#5 0x5566dc5d0be2 base::MessageLoopImpl::DoWork()
#6 0x5566dc5d2a39 base::MessagePumpGlib::Run()
#7 0x5566dc5eb575 base::RunLoop::Run()
#8 0x5566dee25f62 content::GpuMain()
#9 0x5566dbbe7fed content::ContentMainRunnerImpl::Run()
#10 0x5566dd887246 service_manager::Main()
#11 0x5566db211361 content::ContentMain()
#12 0x5566dc2d0c84 content::LaunchTests()
#13 0x5566dc2ad930 main

I'm putting a fix together. And I'll re-activate the tests.
Project Member

Comment 15 by bugdroid1@chromium.org, Nov 27

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9dcc6824cdbc7a65ad15a4b6d16298edc22ec50a

commit 9dcc6824cdbc7a65ad15a4b6d16298edc22ec50a
Author: Etienne Bergeron <etienneb@chromium.org>
Date: Tue Nov 27 21:41:02 2018

Fix TracingControllerTest.NotWhitelistedMetadataStripped flakiness

The |argument_filter_predicate_| is not set in the GPU process.
The CHECK is triggered in the GPU process when flushing the buffer.

void TraceLog::FinishFlush(int generation, bool discard_events) {
  [...]
    if (trace_options() & kInternalEnableArgumentFilter) {
      CHECK(!argument_filter_predicate_.is_null());
      argument_filter_predicate = argument_filter_predicate_;
    }

7923:197923:1121/151153.522324:FATAL:trace_log.cc(997)] Check failed: !argument_filter_predicate_.is_null().


We tried with a CHECK to catch this case early in
base::trace_event::TraceLog::SetEnabled(...).
Which make the test fail here:

[202280:202280:1121/152013.218605:FATAL:trace_log.cc(587)] Check failed: !trace_config.IsArgumentFilterEnabled() || !argument_filter_predicate_.is_null().

The current solution is to provide a default safe predicate filter in this case.

R=oysteine@chromium.org

Bug: 642991
Change-Id: Id52357299d170e66596081b8a020a42f9589754e
Reviewed-on: https://chromium-review.googlesource.com/c/1348633
Commit-Queue: Etienne Bergeron <etienneb@chromium.org>
Reviewed-by: oysteine <oysteine@chromium.org>
Cr-Commit-Position: refs/heads/master@{#611343}
[modify] https://crrev.com/9dcc6824cdbc7a65ad15a4b6d16298edc22ec50a/base/trace_event/trace_log.cc
[modify] https://crrev.com/9dcc6824cdbc7a65ad15a4b6d16298edc22ec50a/content/browser/tracing/tracing_controller_browsertest.cc
[modify] https://crrev.com/9dcc6824cdbc7a65ad15a4b6d16298edc22ec50a/content/browser/tracing/tracing_controller_impl.cc
[modify] https://crrev.com/9dcc6824cdbc7a65ad15a4b6d16298edc22ec50a/content/browser/tracing/tracing_controller_impl.h

Sign in to add a comment