SingleProcessMemoryTracingTest.ManyInterleavedDumps fails |
|||||||
Issue descriptionSingleProcessMemoryTracingTest.ManyInterleavedDumps failed on "Android Tests" builder: https://uberchromegw.corp.google.com/i/chromium.linux/builders/Android%20Tests/builds/33567/steps/content_browsertests%20on%20Android The issue: Note: Google Test filter = SingleProcessMemoryTracingTest.ManyInterleavedDumps [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from SingleProcessMemoryTracingTest, where TypeParam = [ RUN ] SingleProcessMemoryTracingTest.ManyInterleavedDumps [WARNING:dns_config_service_posix.cc(316)] Failed to read DnsConfig. [ERROR:devtools_http_handler.cc(221)] Cannot start http server for devtools. Stop devtools. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. ../../content/browser/tracing/memory_tracing_browsertest.cc:120: Failure Value of: success Actual: false Expected: true [ERROR:memory_dump_manager.cc(573)] Disabling MemoryDumpProvider "ContextProviderCommandBuffer". Dump failed multiple times consecutively. [ERROR:memory_dump_manager.cc(573)] Disabling MemoryDumpProvider "ContextProviderCommandBuffer". Dump failed multiple times consecutively. [ERROR:memory_dump_manager.cc(573)] Disabling MemoryDumpProvider "ContextProviderCommandBuffer". Dump failed multiple times consecutively. ../../content/browser/tracing/memory_tracing_browsertest.cc:207: Failure Mock function called more times than expected - returning default value. Function call: OnMemoryDump(@0x7b83e980 4-byte object <01-00 00-00>, 0x7be6d3e0) Returns: false Expected: to be called 4 times Actual: called 5 times - over-saturated and active ../../content/browser/tracing/memory_tracing_browsertest.cc:207: Failure Mock function called more times than expected - returning default value. Function call: OnMemoryDump(@0x7b83e980 4-byte object <01-00 00-00>, 0x7be6d3e0) Returns: false Expected: to be called 4 times Actual: called 6 times - over-saturated and active ../../content/browser/tracing/memory_tracing_browsertest.cc:207: Failure Mock function called more times than expected - returning default value. Function call: OnMemoryDump(@0x7b83e980 4-byte object <01-00 00-00>, 0x7be6d3e0) Returns: false Expected: to be called 4 times Actual: called 7 times - over-saturated and active [ERROR:memory_dump_manager.cc(573)] Disabling MemoryDumpProvider "MockDumpProvider". Dump failed multiple times consecutively. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [WARNING:graphics_memory_dump_provider_android.cc(67)] Could not connect to the memtrack_helper daemon. Please build memtrack_helper, adb push to the device and run it before starting the trace to get graphics memory data. [INFO:test_support_android.cc(185)] MessagePumpForUIFactory already set, unable to override. >>ScopedMainEntryLogger
,
Oct 20 2016
Hmm the warnings and graphics failures are WAI, some providers will not work in a test environment, but we want to check they don't crash. What is NOT WAI insted is the actual 4 vs 5 times. It smells to me like tracing is doing the periodic dumps as well, but that should be disabled. It might be some of the recent changes to the traceconfig. kraynov can you take a look to this as well? Essentially I expect MemoryDumpManager::PeriodicGlobalDumpTimer::Start to bail out immediately after the first if. If that doesn't happen something is going wrong.
,
Oct 20 2016
Ok I strongly suspect that the root cause of both this and Issue 656729 is this recent change: https://codereview.chromium.org/2323483005/diff/380001/base/trace_event/trace_config.cc I think what's happening here is that the trace config gets merged and persists between each test. So depending on the order the tests are executed we keep the periodic dump config even when we shouldn't.
,
Oct 20 2016
I actually checked that to some extent - there was a successful run which had similar previous test. But I didn't check deeper than that, so maybe it broke earlier.
,
Oct 20 2016
The run can be successful depending on: - the order in which the tests run (which IIRC is random). - timing, you might be lucky and never hit the periodic dump.
,
Oct 24 2016
,
Oct 31 2016
PING, any update on this?
,
Nov 1 2016
I think guess about tests ordering is not correct, since I've managed to print PIDs from tests and each tests came up with distinct PID. Does gtest run in (default) spawn-on-test mode or in single process mode?
,
Nov 1 2016
Re #8: Right for BROWSERTEST_F each test fixture should get its own process. Not sure about what is the behavior for the in-process ones. Where did you print the pids? in which function? Since this is a SingleProcess test I wonder if somehow we manage to oddly merge the traceconfig between what are supposed to be multiple processes. One thing is not clear to me. Did you manage to repro this? how frequently it repros? Can you print the name of the MDP that gets invoked when it repros?
,
Nov 2 2016
It's really hard to reproduce. I think I need to rollback to older revision of codebase. Now it runs hundreds of times and not failing.
I tried to catch invoker (stack trace)
base::subtle::Atomic32* n = new base::subtle::Atomic32[1];
*n = 0;
EXPECT_CALL(*mock_dump_provider_, OnMemoryDump(_,_))
.Times(4)
.WillRepeatedly(testing::Invoke([n](const MemoryDumpArgs&,
ProcessMemoryDump* pmd) -> bool {
auto new_n = base::subtle::Barrier_AtomicIncrement(n, 1);
CHECK(new_n < 5);
return true;
}));
,
Nov 2 2016
If I read the logs correctly that bot is running on a "hammerhead" (old Nexus 5, NOT 5x) @ KTU84P. Are you testing on the same configuration?
,
Nov 2 2016
Nope, Nexus 7
,
Nov 2 2016
well that might explain. There are definitely lot of gpu-related dumpers there which stricly depend on the device. Please try on a Nexus 5 flashing that exact android version. I should have a N5 in my deck on my desk.
,
Nov 3 2016
I strongly suspect that this is the same root cause of Issue 656729 . That is: previous crashed test executions leave around the /data/local/chrome-trace-config.json startup trace file and that interferes with the test.
,
Nov 3 2016
+Dave, Zhen, Emily. See #14: Should a recipe step clean that up? A cleanup step in telemetry?
,
Nov 3 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ea26a4f05c216cd7dd7d76ea19fc46665c300baa commit ea26a4f05c216cd7dd7d76ea19fc46665c300baa Author: primiano <primiano@chromium.org> Date: Thu Nov 03 15:22:03 2016 tracing: Bail out early in tracing tests when tracing is already enabled A bunch of tracing browsertests have recently become flaky. I strongly suspect this is because some other test leaves the startup tracing file (/data/local/chrome-trace-config.json) around. On Android the presence of that file is enough to initialize the tracing machinery and interfere with tests. Bail out early explicitly in that case. BUG= 657628 , 656729 Review-Url: https://codereview.chromium.org/2479563002 Cr-Commit-Position: refs/heads/master@{#429593} [modify] https://crrev.com/ea26a4f05c216cd7dd7d76ea19fc46665c300baa/content/browser/tracing/memory_tracing_browsertest.cc
,
Nov 3 2016
Chrome tracing agent in Telemetry should be able to clean up the file. And it always override the config file before starting tracing. See the code pointers in Issue 656729 .
,
Nov 7 2016
Let's continue the discussion on Issue 656729 .
,
Nov 10 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f98f6a4df243fd2bff7f2c0dfb919a0717f9a511 commit f98f6a4df243fd2bff7f2c0dfb919a0717f9a511 Author: kraynov <kraynov@chromium.org> Date: Thu Nov 10 18:14:00 2016 Tracing: Bail out early in tracing tests when tracing is already enabled. Following up https://crrev.com/2479563002 . BUG= 657628 , 656729 Review-Url: https://codereview.chromium.org/2479593002 Cr-Commit-Position: refs/heads/master@{#431291} [modify] https://crrev.com/f98f6a4df243fd2bff7f2c0dfb919a0717f9a511/components/tracing/child/child_trace_message_filter_browsertest.cc [modify] https://crrev.com/f98f6a4df243fd2bff7f2c0dfb919a0717f9a511/content/browser/tracing/memory_tracing_browsertest.cc
,
Nov 11 2016
Seems to be fixed, thanks perezju@ Do re-open if flakyness appears again. More context: https://crbug.com/656729
,
Jan 4 2017
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by dskiba@chromium.org
, Oct 19 2016