Apparent race condition in Telemetry's minidump processing on Perf bots |
||||
Issue descriptionIn https://bugs.chromium.org/p/chromium/issues/detail?id=530197#c22 it was pointed out that there may be a race condition in Telemetry's minidump processing where if the minidump hasn't been fully generated yet, it might be skipped. This assertion needs to be verified and any race condition fixed.
,
May 27 2016
As far as I can tell, at least one bug in the Perf bots' setup is that they build the chromium_builder_perf target (see any recent build on https://build.chromium.org/p/chromium.perf/builders/Win%20x64%20Builder ) , and this doesn't include a dependency on crashpad_database_util.exe, or any of the other important dependencies like the copy_cdb_to_output step in src/tools/perf/chrome_telemetry_build/BUILD.gn. I'm going to add logging to Telemetry's desktop_browser_backend to narrow down why it didn't find the minidump. On my machine, adding a DCHECK early on in renderer startup, if crashpad_database_util was missing then the symptom on the Perf bots happened. Once that target was built, the minidumps started getting found and symbolized.
,
May 27 2016
,
May 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d078c69020beaab39b72bba3d6a1a45a2386cd12 commit d078c69020beaab39b72bba3d6a1a45a2386cd12 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Fri May 27 06:37:56 2016 Roll src/third_party/catapult/ baa2999e8..aa8636016 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/baa2999e8462..aa8636016c8a $ git log baa2999e8..aa8636016 --date=short --no-merges --format='%ad %ae %s' BUG= 615180 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2015143003 Cr-Commit-Position: refs/heads/master@{#396408} [modify] https://crrev.com/d078c69020beaab39b72bba3d6a1a45a2386cd12/DEPS
,
May 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1e58e78ecce28f5891912c000c5d2520e4f5b09d commit 1e58e78ecce28f5891912c000c5d2520e4f5b09d Author: kbr <kbr@chromium.org> Date: Fri May 27 17:21:19 2016 Refactor dependencies for chromium_builder_perf target. The perf bots build this target, but don't use isolates and therefore aren't on the same overall path as the bots which run telemetry_perf_unittests, where minidump processing is well tested. Make this target depend on telemetry_chrome_test, and remove redundant dependencies on chrome and bitmaptools. This should cause the tools which generate and process minidumps to be properly built. BUG= 615180 Review-Url: https://codereview.chromium.org/2019553002 Cr-Commit-Position: refs/heads/master@{#396497} [modify] https://crrev.com/1e58e78ecce28f5891912c000c5d2520e4f5b09d/BUILD.gn
,
May 27 2016
I think this is fixed. Unfortunately the only way to really test it is to induce a crash on the Perf bots, which obviously isn't desirable. Maybe we could try re-enabling the test from Issue 615178 briefly and see if a reasonable crash stack is printed. |
||||
►
Sign in to add a comment |
||||
Comment 1 by scottmg@chromium.org
, May 26 2016