Issue 628050 has been merged into this issue.

Issue 666182 has been merged into this issue.

We just hit another one of these on the PGO FYI builder, build slave build15-m1. However the frequency seems to be fairly low - we've had dozens of builds without this issue.

c:\b\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\vc\include\xlocnum(1261): fatal error C1090: PDB API call failed, error code '23': '(
c:\b\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\vc\include\xlocnum(1255): note: while compiling class template member function 'std::ostreambuf_iterator<_Elem,_Traits> std::num_put<_Elem,std::ostreambuf_iterator<_Elem,_Traits>>::do_put(_OutIt,std::ios_base &,_Elem,bool) const'
        with
        [
            _Elem=char,
            _Traits=std::char_traits<char>,
            _OutIt=std::ostreambuf_iterator<char,std::char_traits<char>>
        ]
c:\b\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\vc\include\ostream(405): note: see reference to class template instantiation 'std::num_put<_Elem,std::ostreambuf_iterator<_Elem,_Traits>>' being compiled
        with
        [
            _Elem=char,
            _Traits=std::char_traits<char>
        ]
c:\b\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\vc\include\ostream(396): note: while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)'
c:\b\build\slave\chromium-win-pgo-builder\build\src\third_party\protobuf\src\google\protobuf\stubs\int128.cc(174): note: see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled
c:\b\build\slave\chromium-win-pgo-builder\build\src\third_party\protobuf\src\google\protobuf\stubs\int128.cc(141): note: see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled

libyuv is hitting this consistently after rolling chromium_revision 941118827f..316b880c55 (it's using the same build toolchain). See
https://build.chromium.org/p/client.libyuv/waterfall where I recently reverted the roll in https://codereview.chromium.org/2648573002/). Example build: https://build.chromium.org/p/client.libyuv/builders/Win32%20Debug/builds/24

It fails reliably so it should be possible to bisect down that range to a much smaller one by firing tryjobs to the libyuv trybots: https://build.chromium.org/p/tryserver.libyuv/waterfall

I'll look at this asap. 

It seems to be caused by https://codereview.chromium.org/2296953002 (the libyuv bots uses Goma).

The problem doesn't repro locally with Goma disabled, I'll see if enabling it allows me to repro the issue.

The bug repro if I enable Goma locally.

Note that we've observed the "PDB API call failed, error code '23'" error message on some bots that don't use Goma, so the overall problem isn't specific to Goma but in this case it seems to be what's causing the build failures.

Is there anything unusual about the symbol settings for libyuv, or the size of the .lib being produced? goma on Windows generally only works with symbol_level == 0 or 1 so if libyuv is trying to use symbol_level == 2 then this is somewhat expected.

Yes, libyuv is using symbol_level == 2.

That is unsupported and undesirable when using goma. When you enable goma it defaults to symbol_level == 1. Most builders also explicitly set symbol_level == 1.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/da16a7170b068e037c93d00a78ef602895977ab2

commit da16a7170b068e037c93d00a78ef602895977ab2
Author: Henrik Kjellander <kjellander@chromium.org>
Date: Fri Feb 03 11:20:03 2017

libyuv: Don't set symbol_level=2 for Windows.

Windows compilation is failing for libyuv due to  crbug.com/659439 .

This has the side-effect that the --ninja-log-compiler flag is set to
"clang" instead of "goma-clang" on Mac, but I don't think that affects
anything.

BUG= libyuv:677 , chromium:659439 
TBR=fbarchard@google.com

Change-Id: I95920e38aff12749e8b01042794fdb5ee0e969b4
Reviewed-on: https://chromium-review.googlesource.com/437124
Reviewed-by: Henrik Kjellander <kjellander@chromium.org>
Commit-Queue: Henrik Kjellander <kjellander@chromium.org>

[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/tryserver_libyuv_win_clang.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/client_libyuv_win32_release.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipe_modules/libyuv/api.py
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/tryserver_libyuv_win_x64_rel.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/client_libyuv_win64_debug.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/tryserver_libyuv_win_clang_rel.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/tryserver_libyuv_win_x64_clang_rel.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/client_libyuv_win32_debug.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/tryserver_libyuv_win.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/client_libyuv_win64_release.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipes/libyuv/libyuv.expected/tryserver_libyuv_win_rel.json
[modify] https://crrev.com/da16a7170b068e037c93d00a78ef602895977ab2/scripts/slave/recipe_modules/libyuv/chromium_config.py

Issue 714504 has been merged into this issue.

Issue 715275 has been merged into this issue.

Currently win trunk and win beta builders are failing. If this fails constantly, it will be blocking our planned releases. Can we take a action ASAP.

https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win%20trunk/builds/62235

https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win%20beta

Thank you Ligi.

It looks like the failures are still intermittent, so annoying but non-blocking.

Unfortunately I have no suggestions other than reboot/clobber/try-again.

Should we reach out to MS to see if there's a way to fix this? I don't think that generating a link repro will be useful here but there's maybe something else that we can do to help them investigate on this?

I filed a bug last January:

http://connect.microsoft.com/VisualStudio/feedback/details/2299595/fatal-error-c1090-pdb-api-call-failed)

but Microsoft's brilliant bug database seems to have lost the report, or accidentally blocked access to it. So, I guess I should file a new report in their new system. Sigh...

I filed a new bug:

https://developercommunity.visualstudio.com/content/problem/48897/c1090-pdb-api-call-failed-error-code-23.html

I included a link to the old bug, but I've learned vital lessons from filing previous bugs. The new bug reporting system will strip out URLs if it detects them, leaving nothing but a bare https:// or http://. The trick, of course, is to strip out the leading https:// so that it doesn't recognize the link as a URL. I do so enjoy hacking around bug reporting quirks. Anyway, don't hold your breath. This bug has been around for a *long* time.

Some searching around pops up a few old references which suggest this error might be due to mspdbsrv.exe crashing[1]. Has anyone looked into that? If that is the problem, could it maybe be worked around by trying to detect the error, and then restarting mspdbsrv and rerunning the failed commands?


[1] https://issues.jenkins-ci.org/browse/JENKINS-9104 and
https://connect.microsoft.com/VisualStudio/feedback/details/494122/random-c1090-pdb-api-call-failed-when-compiling-2-builds-concurrently

Note that there's a way to use several instances of mspdbsrv.exe instances (by setting the undocummented _MSPDBSRV_ENDPOINT_, see [1]) so if one instance can't handle Chrome anymore then we could look at sharding it.

[1] https://cs.chromium.org/chromium/src/build/toolchain/win/tool_wrapper.py?l=36&rcl=43fcd3bea1e19775dde52a3180e23a7ef3cf2363

If we restart the build then mspdbsrv will automatically restart. So, we would just need to detect that the build error was a "restartable" error and re-run ninja. That seems like a worthwhile experiment - no idea whether it will work.

We could even decide that we should just routinely re-run ninja. Although, that is ugly.

Issue 735636 has been merged into this issue.

Issue 736450 has been merged into this issue.

Issue 746127 has been merged into this issue.

Issue 749311 has been merged into this issue.

Issue 756307 has been merged into this issue.

Seeing similar failure on 63.0.3238.0 on Windows(Non PGO) on official desktop builder:

Link to the builder:
===================
https://uberchromegw.corp.google.com/i/official.desktop/builders/win/builds/2256

Link to the log:
================
https://uberchromegw.corp.google.com/i/official.desktop/builders/win/builds/2256/steps/compile/logs/stdio

Error log:
==========
FAILED: obj/third_party/boringssl/boringssl/ssl_transcript.obj 
ninja -t msvc -e environment.x86 -- "c:\b\c\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\vc\tools\msvc\14.11.25503\bin\hostx64\x86/cl.exe" /nologo /showIncludes  @obj/third_party/boringssl/boringssl/ssl_transcript.obj.rsp /c ../../third_party/boringssl/src/ssl/ssl_transcript.cc /Foobj/third_party/boringssl/boringssl/ssl_transcript.obj /Fd"obj/third_party/boringssl/boringssl_cc.pdb"
c:\b\c\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\include\10.0.15063.0\ucrt\stdlib.h(360): fatal error C1090: PDB API call failed, error code '23': '(
[9255/51519] CXX obj/third_party/WebKit/common/mojo_bindings_blink/message_port.mojom-blink.obj
[9256/51519] ACTION //components/policy:policy_templates(//build/toolchain/win:x86)
ninja: build stopped: subcommand failed.
step returned non-zero exit code: 1

Suspecting the recent failure on Windows Non PGO build(63.0.3238.0) to be the same as this one.

sebmarchand@: Could you please confirm the same.

Thank you!

Yep, it's the same one! There's not so much we can do here unfortunately... 

I just got this update from the (now inaccessible) Microsoft connect bug:

"We have added /fastlink support to mspdbsrv.exe so that whenever it crashes during linking we would have a crash dump saved on local machine assuming all the registry settings are in place to allow OS to do so.  I'll close this issue at our end for now.  Feel free to reactivate this if you have any other concern or question on this issue.  Thanks again for the bug report."

I assume that by /fastlink they mean /fastfail.

So, the next time this happens it might be worth seeing if there is a crash dump, presumably in %localappdata%\crashdumps.

In the past I've found that many of our builders refuse to save crash dumps, so this may not work. Maybe we'll get lucky and it will crash on a developer's workstation.

It's not clear what version of mspdbsrv.exe has the /fastfail support but we are on the latest released version of VS (15.4). However, we may not be on this version for long since a switch to clang is probably not far away and it doesn't use mspdbsrv.exe.

Thanks for the update. I don't have any data to back this up, but it feels like this problem is less frequent than it used to be? Which probably also means that getting a crash dump for this will be harder :)

Issue 777580 has been merged into this issue.

I hit this failure today on a local LTCG build with VC++ 2017. However I was using the 15.3 toolchain which I don't think yet has the fastfail support mentioned in comment #35. So, no crash dump was recorded and we're no closer to understanding this issue.

I restarted my build and it's working fine now.

Issue 788612 has been merged into this issue.

Lowering the priority of this as this doesn't seem like a P1 for me, and there's not so much we can do for now...

Issue 799831 has been merged into this issue.

I hope that lld will resolve this. The main disadvantage with lld (use_lld=true) is that it doesn't support incremental linking, but for many builders this is mostly or entirely irrelevant so if this issue happens frequently enough we should consider switching.

Just to update the latest behavior still this failure is observed for Win-asan build #65.0.3315.1.

Build Path:
===========
https://uberchromegw.corp.google.com/i/official.desktop/builders/win-asan/builds/869

Log Path: 
==========
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fofficial.desktop%2Fwin-asan%2F869%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

Thanks!

Seems to be #65.0.3316.1 compiled successfully and build is available to download.

Build Path:
===========
https://uberchromegw.corp.google.com/i/official.desktop/builders/win-asan/builds/870

Thanks!

We don't use link.exe anymore.