"Memlog/MemlogBrowserTest.EndToEnd/13" is flaky |
|||||||
Issue description"Memlog/MemlogBrowserTest.EndToEnd/13" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label. We have detected 4 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiRNZW1sb2cvTWVtbG9nQnJvd3NlclRlc3QuRW5kVG9FbmQvMTMM. Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
,
Feb 13 2018
There are currently 4 failures from today, but also one from 2018-02-09 which looks identical. CCing owners.
,
Feb 13 2018
According to the flakiness dashboard [1], many of the MemlogBrowserTest.EndToEnd tests are flaky to varying degrees on multiple platforms, but /13 got much worse on Windows recently, around 2018-02-13 at 7:30 AM GMT. The timing matches CL [2] which looks plausibly related. I'll try reproing locally. [1] https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=Memlog%2FMemlogBrowserTest.EndToEnd%2F [2] https://chromium-review.googlesource.com/c/chromium/src/+/900222
,
Feb 13 2018
I've locally verified that https://crrev.com/c/900222 is indeed the culprit, will revert. Note to anyone else looking at this: The test in question is only compiled with "is_component_build = false".
,
Feb 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1c849e92aef656ed495be99af74442c75243789c commit 1c849e92aef656ed495be99af74442c75243789c Author: Marc Treib <treib@chromium.org> Date: Tue Feb 13 14:30:42 2018 Revert "Clean up of memory instrumentation interface." This reverts commit 99b4a583f634ba382fd597cda1148cc4299dc2e0. Reason for revert: Breaks Memlog/MemlogBrowserTest.EndToEnd/13 on Windows; see crbug.com/811711 Original change's description: > Clean up of memory instrumentation interface. > > This CL cleans up both the public interface and private implementation of memory > instrumentation. It is in preparation for allowing memory instrumentation to > asynchronously add heap dumps to traces. > > memory_instrumentation supports two interfaces: HeapProfilerHelper and > Coordinator [specifically RequestGlobalMemoryDump]. They are used by different > clients, with different privileges, for different purposes. The internal > implementation used a single code path to support both, which further results in > leakage of implementation details into the mojom interface, in the form of > LevelOfDetail::VM_REGIONS_ONLY_FOR_HEAP_PROFILER. > > The previous implementation only allowed 1 global dump to be issued at a time. > This causes problems for my upcoming refactor, which makes > memory_instrumentation, during a global dump, request a heap dump from the > profiling service. The profiling service in turn requests heaps from > memory_instrumentation, resulting in a deadlock. > > This CL updates the implementation of CoordinatorImpl to use a separate queue of > |in_progress_vm_region_requests_| to track heap dump requests. These are > executed as soon as they are received. This CL also removes > LevelOfDetail::VM_REGIONS_ONLY_FOR_HEAP_PROFILER, and updates > GetVmRegionsForHeapProfiler to explicitly specify PIDs to dump. > > Bug: 758739 > Change-Id: I1ee41e272ac4379f791abe469e0d8efda42636de > Reviewed-on: https://chromium-review.googlesource.com/900222 > Commit-Queue: Erik Chen <erikchen@chromium.org> > Reviewed-by: Daniel Cheng <dcheng@chromium.org> > Reviewed-by: Ilya Sherman <isherman@chromium.org> > Reviewed-by: Primiano Tucci <primiano@chromium.org> > Cr-Commit-Position: refs/heads/master@{#536263} TBR=dcheng@chromium.org,isherman@chromium.org,primiano@chromium.org,erikchen@chromium.org,lalitm@chromium.org Change-Id: I6ba2c5548b4fcd2950a3e0a1734a10140c6c152a No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 758739, 811711 Reviewed-on: https://chromium-review.googlesource.com/916063 Reviewed-by: Marc Treib <treib@chromium.org> Commit-Queue: Marc Treib <treib@chromium.org> Cr-Commit-Position: refs/heads/master@{#536353} [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/browser/metrics/process_memory_metrics_emitter_unittest.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/browser/profiling_host/background_profiling_triggers_unittest.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/browser/profiling_host/profiling_test_driver.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/profiling/memlog_connection_manager.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/profiling/memlog_connection_manager.h [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/profiling/profiling_service.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/chrome/profiling/profiling_service.h [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/coordinator_impl.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/coordinator_impl.h [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/coordinator_impl_unittest.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/queued_request.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/queued_request.h [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/queued_request_dispatcher.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/memory_instrumentation/queued_request_dispatcher.h [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/public/cpp/memory_instrumentation/memory_instrumentation_mojom_traits.cc [modify] https://crrev.com/1c849e92aef656ed495be99af74442c75243789c/services/resource_coordinator/public/mojom/memory_instrumentation/memory_instrumentation.mojom
,
Feb 13 2018
,
Feb 13 2018
,
Feb 13 2018
Re c#3 - I checked out the link in (1), and didn't see any flakiness [ I was curious about the flakiness before the CL in question ]. Maybe I'm using the dashboard incorrectly? Can you send a link to a flaky failure [prior to 2-13?]
,
Feb 13 2018
All the gray/black boxes correspond to flakes. In the link, you'll have to scroll down a bit, e.g. /15 and /16 are quite flaky. One example for /13 is here: https://ci.chromium.org/buildbot/chromium.win/Win%207%20Tests%20x64%20(1)/34648 Note that this run still passed, because the test got retried and passed on the second run. But you'll see the failure in the actual logs: https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.win%2FWin_7_Tests_x64__1_%2F34648%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Fstdout
,
Feb 13 2018
Thanks. Almost all of the cases I looked at were timeouts, and the root cause is the same as the one you locally - the factor that affects whether the test times out or crashes is whether DCHECKs are enabled.
,
Feb 13 2018
,
Feb 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a6f3feb77a659dee2dd09707203fa43e0d800894 commit a6f3feb77a659dee2dd09707203fa43e0d800894 Author: erikchen <erikchen@chromium.org> Date: Wed Feb 14 16:21:56 2018 Fix 2 bugs in out of process heap profiling implementation. Bug 1: When an AllocationTracker is destroyed from a client disconnecting, queued barrier callbacks would not be run. This would cause the heap dump to never be emitted. Bug 2 : The implementation of MemlogConnectionManager::OnNewConnection called mojo::SharedBufferHandle::Create, which uses synchronous IPC with the browser process. This can cause deadlock/timeouts, since the browser process synchronously waits for the memlog pipe to free up and eventually gives up after 10 seconds, tearing down the allocator shim. The solution is to create a dedicated thread for calling mojo::SharedBufferHandle::Create, thus avoiding the deadlock. Change-Id: Ib3f4143381ce03c353ffe93d45794070e3b007d5 Bug: 811711 Reviewed-on: https://chromium-review.googlesource.com/917243 Commit-Queue: Erik Chen <erikchen@chromium.org> Reviewed-by: Primiano Tucci <primiano@chromium.org> Cr-Commit-Position: refs/heads/master@{#536729} [modify] https://crrev.com/a6f3feb77a659dee2dd09707203fa43e0d800894/chrome/profiling/allocation_tracker.cc [modify] https://crrev.com/a6f3feb77a659dee2dd09707203fa43e0d800894/chrome/profiling/memlog_connection_manager.cc [modify] https://crrev.com/a6f3feb77a659dee2dd09707203fa43e0d800894/chrome/profiling/memlog_connection_manager.h
,
Feb 15 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by treib@chromium.org
, Feb 13 2018