Looking at https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%20tester/builds/4420/steps/extract%20build/logs/stdio

It extracts these files:

Extracting  full-build-win32\crash_service.exe.assert.manifest
Extracting  full-build-win32\crash_service.exe.manifest
Extracting  full-build-win32\crash_service64.exe.assert.manifest
Extracting  full-build-win32\crash_service64.exe.manifest

But no crash_service.exe, but in the previous build, it did.

Looks like it was never compiled: https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin/builds/7620/steps/compile/logs/stdio

But in the previous build, it was:
https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin/builds/7617/steps/compile/logs/stdio

Probably related to scottmg's https://codereview.chromium.org/1858883004

I don't really know what crash_service is, but was removing it intentional, and the tests should not be using it?

https://codereview.chromium.org/1862773003/ looks like it's removing references to crash_service on the bots, including ours

+dpranke for that patch
looks like it got reverted, but maybe it re-landed later? do we need a master restart for chromium.fyi to pick it up?

I think the update_scripts step on that bot is updating the src/ checkout, which suggests that maybe we have a bad .gclient somewhere.

@stip, can you take a look?

We shouldn't be starting up crash_service any more, and I don't see any sign of it in the recipes at tip of tree.

chromium.fyi was restarted last night (before that build), so that's not the problem.

https://build.chromium.org/p/chromium.fyi/builders/CrWinAsan%28dll%29/builds/4001/steps/compile/logs/stdio still passes `-w dupbuild=err` which we removed months ago. It looks like that bot is never updating slave scripts maybe?

stip, did you have a chance to check what's up here?

Upping priority as this means we currently don't have test coverage for Win Clang release builds.

stiiiiiiiip haaaaaaaaalllp

I logged into vm310-m1 (ClangToTWin tester) and moved a suspicious-looking pair of DEPS and .DEPS.git files, and forced a build.

If that goes through ok, I can look at the other failing bots, since stip@'s awol.

That build failed with another exception:

@@@STEP_LOG_LINE@exception@KeyError: 'parent_buildername'@@@

but that might just be because it's confused about how the build was triggered..

Yup, that was me failing to remember that you can't force builds on triggered builders. 

So, we just have to wait.

hrm. My changes don't appear to have helped:

https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%20tester/builds/4454/steps/update_scripts/logs/stdio

I'm not sure what's going on, so I'm going to back away slowly and try to get a trooper to look at it.

*moves forward slowly to replace dpranke*

so update_scripts is *definitely* checking out src/, which is 

(somehow hit send accidentally).

update_scripts checking our src is definitely not correct. labs, can you reimage vm310-m1?

Taking a look.

Re-imaged and migrated to new hardware in the process. Waiting on a build for verification.

Build is now green: 

https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%20tester/builds/4456

Thanks.

As per comment #9, there's two more bots with the same issue.

@johnw, mind reimaging those as well?

https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%20tester (vm312-m1)
https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64 (build129-m1)

Or should we file a new ticket for them?

Sorry, I missed that there were multiple bots involved. We'll get these all fixed up.

Thanks.

Builder just went green: https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64/builds/7779

And the final tester just kicked off here: https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%20tester/builds/4480

Thanks.

Thanks, John!

It looks like build27-m1 needs this too:

https://build.chromium.org/p/chromium.fyi/builders/CrWinAsan%28dll%29/builds/4008

(https://build.chromium.org/p/chromium.fyi/builders/CrWinClangLLD64 on the same slave is happy though...)

Waiting for build27-m1 to finish it's current build before working on it.

Thanks.

@thakis - it looks like `update_scripts` is broken in the same way on both builders, but some how we're seeing crash service get added on one builder, and not the other, which is really weird. I don't even know how that's possible, since both builders should be seeing the same recipes.

vm311-m1 needs this too:
https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin(dll)%20tester
https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin(dbg)%20tester

vm978-m1 too:
https://build.chromium.org/p/chromium.fyi/builders/CrWinAsan%20tester

It's pretty new on these two slaves; things worked well there until like yesterday :-/

I expect that `update_scripts` has been failing on these bots for some time, but you've been getting away with this because you've been doing incremental builds and have an old crash_service lying around.

However, when MB is enabled, it starts off by doing a clobber build so it can be sure it's in a clean directory and not confused over whether it was previously GYP or GN (it also does clobbers when switching between the two). As a result, your old crash_service is deleted, and things start to break.

update_scripts should really be a fatal error, but currently isn't (see bug 421769).

Ah, that's a good explanation, thanks.

CrWinAsan(dll) is now passing:

https://build.chromium.org/p/chromium.fyi/builders/CrWinAsan%28dll%29/builds/4019

I can look at vm311-m1 vm978-m1 today.

any luck?

Yeah, I redeployed both this morning. Neither has picked up any jobs for me to confirm greenness yet, however. 

Looks like all our fyi bots are now happy. Thanks, johnw :-)

https://build.chromium.org/p/chromium.fyi/builders/ClangToTMacASan%20tester/builds/2095 -- vm30-m1 also prints

Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.


Traceback (most recent call last):
  File "../build/mac/find_sdk.py", line 97, in <module>
    print main()
  File "../build/mac/find_sdk.py", line 89, in main
    'macosx' + best_sdk, 'Path']).strip()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['xcodebuild', '-version', '-sdk', 'macosx10.10', 'Path']' returned non-zero exit status 69
CalledProcessError: Command '['/usr/bin/python', '../build/mac/find_sdk.py', '--print_sdk_path', '10.10']' returned non-zero exit status 1:
  File "/b/build/slave/ClangToTMacASan_tester/build/src/native_client/SConstruct", line 2437:
    (mac_debug_env, mac_optimized_env) = GenerateOptimizationLevels(MakeMacEnv())


justincohen, can you heal that bot too please?

sorry, wrong bug

Here's an exciting new bot where this is broken: https://build.chromium.org/p/chromium.fyi/builders/CrWinClang(shared)%20tester -- vm962-m1

Removing troopers label, as labs is handling this.

Ping, https://build.chromium.org/p/chromium.fyi/builders/CrWinClang(shared)%20tester / vm962-m1 is still having this problem.

https://build.chromium.org/p/chromium.fyi/builders/CrWinClang%28shared%29%20tester/builds/4756

Thanks again :-)