New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 601839 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
OOO until 2019-01-24
Closed: Apr 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug

Blocked on:
issue 601762



Sign in to add a comment

Windows GPU bots failing to find crash_service.exe

Project Member Reported by ajuma@chromium.org, Apr 8 2016

Issue description

Comment 1 by jo...@chromium.org, Apr 8 2016

Labels: Infra-Troopers
+troopers

crash_service.exe appears to be present in the checkout on a couple of machines that I checked.
Cc: serg...@chromium.org
Oh god no. We went through so much to make sure this worked after it landed, and now it has been reverted?

Context: https://bugs.chromium.org/p/chromium/issues/detail?id=601640
And context for why the CL was reverted: https://bugs.chromium.org/p/chromium/issues/detail?id=601762
Status: Available (was: Unconfirmed)
Story:
Original crash_service removal CL: https://codereview.chromium.org/1862773003
This caused the Win GPU bots to start failing, because they were still running the crash_service (because it doesn't get killed at the end of a build) but weren't using it: https://bugs.chromium.org/p/chromium/issues/detail?id=601640
Fix for that was to auto_reboot: https://codereview.chromium.org/1866403003

However at the same time the GPU bots on the perf waterfall started failing to compile/link, due to being unable to find crash_service: https://bugs.chromium.org/p/chromium/issues/detail?id=601762
This was fixed by reverting the crash_service recipe CL: https://codereview.chromium.org/1871583004

So it looks to me like the crash_service recipe CL maybe removed *too many* references to it? Or maybe the problem is simply that the perf gpu bots and the main waterfall gpu bots are building slightly different targets that don't have their dependencies in sync, so the latter still need the crash service?
In particular, note that a successful build[1] on the perf waterfall explicitly builds the crash_service target (very end of line at the top of the log) while a failing build[2] doesn't specify that target on the command line at all.

[1]: https://build.chromium.org/p/chromium.perf/builders/Win%20Builder/builds/6882/steps/compile/logs/stdio
[2]: https://build.chromium.org/p/chromium.perf/builders/Win%20Builder/builds/6881/steps/compile/logs/stdio
And now sergiyb has reverted the revert: https://codereview.chromium.org/1867293003

The perf bots have broken again: https://build.chromium.org/p/chromium.perf/builders/Win%20Builder/builds/6887

The main waterfall bots haven't cycled again yet.
Owner: dpranke@chromium.org
Status: Assigned (was: Available)
dpranke, I need your help to figure out why the crash_service recipe CL would cause compile/link to fail on the perf bots. I don't have enough context there to know why that is happening, or if any src-side changes need to land to fix it.
Strangely, despite the revert of the revert, we now also have a green build on the perf bots: https://build.chromium.org/p/chromium.perf/builders/Win%20Builder/builds/6886

Comment 9 by kbr@chromium.org, Apr 8 2016

Cc: nedngu...@google.com sullivan@chromium.org
Owner: kbr@chromium.org
Dirk's crash_service CL should remain in. Reverting it was a mistake.

We need to understand why the Perf builders are still trying to build crash_service. I've found a couple of references but they're in the GN build.

Comment 10 by kbr@chromium.org, Apr 8 2016

To clarify: Dirk's CL https://codereview.chromium.org/1862773003 will stop manually adding crash_service to the list of built targets. The question is why the Perf bots are trying to run the manifest tool against the nonexistent crash_service binary.

If you search the compile logs on the perf bots, they're still trying to STAMP the crash_service: https://build.chromium.org/p/chromium.perf/builders/Win%20Builder/builds/6892/steps/compile/logs/stdio
Which means my naive guess would be that the 'chrome_builder_perf' compile target depends on the crash service?

Comment 13 by kbr@chromium.org, Apr 8 2016

Status: Started (was: Assigned)

Comment 14 by kbr@chromium.org, Apr 8 2016

Blockedon: 601762
Current theory is that prior to my CL, we were running crash_service on all of the bots, including the builders. When we landed (or re-landed) my CL, we stopped killing any existing crash_service processes, and so the next compile on the builder failed because the file was still open.

I've landed a follow-up CL that will still try to kill any crash_service processes, and hopefully that'll fix things, or at least reveal the next issue ;).
Project Member

Comment 16 by bugdroid1@chromium.org, Apr 8 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4cc1754d4ddb33f39dd9fe9cc9518f78bd840091

commit 4cc1754d4ddb33f39dd9fe9cc9518f78bd840091
Author: kbr <kbr@chromium.org>
Date: Fri Apr 08 23:14:35 2016

Remove stray references to crash_service.

Stop bundling it in the installer (FILES.cfg). Remove the references
to the crash_service target from the perf builders' targets. (Issue
601762)

It's necessary to proceed with the removal of the start_crash_service
step on the bots, because putting it back now breaks the GPU bots
( Issue 601839 ).

The BUILD.gn removals are proactive, and not actually used yet.

BUG= 601762 ,  601839 

Review URL: https://codereview.chromium.org/1875613004

Cr-Commit-Position: refs/heads/master@{#386239}

[modify] https://crrev.com/4cc1754d4ddb33f39dd9fe9cc9518f78bd840091/BUILD.gn
[modify] https://crrev.com/4cc1754d4ddb33f39dd9fe9cc9518f78bd840091/build/all.gyp
[modify] https://crrev.com/4cc1754d4ddb33f39dd9fe9cc9518f78bd840091/chrome/tools/build/win/FILES.cfg
[modify] https://crrev.com/4cc1754d4ddb33f39dd9fe9cc9518f78bd840091/tools/perf/chrome_telemetry_build/BUILD.gn

Comment 17 by kbr@chromium.org, Apr 9 2016

Status: Fixed (was: Started)
dpranke's CL fixed the Perf builders. Mine helps ensure his CL won't be reverted again, which would break the GPU bots. Closing as fixed.

Components: Infra>Labs
Labels: -Infra-Labs

Sign in to add a comment