gtest is special in that it uses SEH, so that's a good place to start.

It's not always gtest:

remoting_host.lib(remoting_host.rdp_client_window.obj) : fatal error LNK1223: invalid or corrupt file: file contains invalid .pdata contributions

That TU doesn't have SEH as far as I can tell.

I haven't been able to reproduce this locally. I nuked the out/ dir on the builder, maybe that helps.

The first build after nuking the out/ dir is green: https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%28dbg%29/builds/5692

I'll check back tomorrow and see how it's developed.

Nope, the errors are back :-/
I'll look some more.

The latest build failed with:
FAILED: C:\b\depot_tools\python276_bin\python.exe gyp-win-tool link-with-manifests environment.x64 True mojo_common_unittests.exe "C:\b\depot_tools\python276_bin\python.exe gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:mojo_common_unittests.exe @mojo_common_unittests.exe.rsp" 1 mt.exe rc.exe "obj\mojo\mojo_common_unittests.mojo_common_unittests.exe.intermediate.manifest" obj\mojo\mojo_common_unittests.mojo_common_unittests.exe.generated.manifest ..\..\build\win\compatibility.manifest
gtest.lib(gtest.gtest.obj) : fatal error LNK1223: invalid or corrupt file: file contains invalid .pdata contributions

(https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%28dbg%29/builds/5722/steps/compile/logs/stdio)

Attaching gtest.lib from the bot.

Deleted: gtest.lib 5.0 MB

gtest.lib 5.0 MB Download

There's nothing obviously wrong with the .obj file. I tried replacing my local gtest.lib with this one, and everything went find. =/

This hasn't happened on the pinned 64 dbg bot on https://build.chromium.org/p/chromium.fyi/console?category=win%20clang after rolling clang. (That one uses gn while the tot bot doesn't, which might be why?)

I've been able to reproduce this a few times now. It looks like an incremental linking issue. If you hit the failure once, it will not reproduce because the next link will not be incremental.

Here are some full repro steps. First, clean your output directory and link a binary. This will succeed because it is not incremental. Now edit build.ninja to modify cl_x64 so that all .objs are rebuilt, and rebuild. Linking should fail with LNK1223. Building again should succeed.

Did we roll in any interesting /Brepro incremental linking changes in the last roll window?

Not as far I know.

Some revisions from the tot bot:

Bad:
https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%28dbg%29/builds/5593
clang r263469
chromium #381033

Before that we have a pretty long window of builds failing while building LLVM (due to upstream being broken), and before that things looked fine.

The VS2015 switch went in in #380711, so it's well possible this is due to that too -- we already have known crashes in the incremental linker ( bug 482671 ).

Bruce, have you received a fix for that linker crash? That's not fixed yet in your update 2 toolchain, right?

Reid, there's a new VS2015 toolchain in the CQ corresponding to update 2 (https://codereview.chromium.org/1851573002/) -- maybe it's better with that?

Ah right, the pinned bots didn't get this, so it's probably not due to 2015 but due to clang changes. I keep forgetting that.

I guess that means we can bisect?

> Bruce, have you received a fix for that linker crash? That's
> not fixed yet in your update 2 toolchain, right?

I have not received a fix for the linker crash. That was figured out too close to Update 2 release for them to roll it in (even though it was basically a one line fix). We can get a fixed linker if we need one, but if we don't then that is probably simpler.

This is now also happening on our shiny new trybots and blocking turning those on by default :-(

https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang/builds/903/steps/compile%20%28without%20patch%29/logs/stdio

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/053fe8c6b14cb81880e37f9af3ea4fbde99e45fb

commit 053fe8c6b14cb81880e37f9af3ea4fbde99e45fb
Author: kjellander@webrtc.org <kjellander@webrtc.org>
Date: Fri Apr 01 07:03:10 2016

CQ: Disable win_x64_clang_dbg trybot

Needed to unblock rolling chromium_revision in WebRTC DEPS.

BUG= chromium:595702 
TBR=phoglund@webrtc.org

Review URL: https://codereview.webrtc.org/1850113002 .

Cr-Commit-Position: refs/heads/master@{#12189}

[modify] https://crrev.com/053fe8c6b14cb81880e37f9af3ea4fbde99e45fb/infra/config/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/ef67e7f63f8a312ee0f5590f0f17a5bfbacb7c3d

commit ef67e7f63f8a312ee0f5590f0f17a5bfbacb7c3d
Author: kjellander@chromium.org <kjellander@chromium.org>
Date: Fri Apr 01 07:05:10 2016

WebRTC: Disable Win64 Debug (Clang) bot

Needed to unblock rolling chromium_revision in WebRTC DEPS.

BUG= 595702 
TBR=phoglund@chromium.org

Review URL: https://codereview.chromium.org/1850103002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299624 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/ef67e7f63f8a312ee0f5590f0f17a5bfbacb7c3d/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win64_debug__clang_.json
[modify] https://crrev.com/ef67e7f63f8a312ee0f5590f0f17a5bfbacb7c3d/scripts/slave/recipes/webrtc/standalone.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/a312c3d88e868d57f33e5ed6c8fd81515159a46e

commit a312c3d88e868d57f33e5ed6c8fd81515159a46e
Author: kjellander <kjellander@webrtc.org>
Date: Fri Apr 01 07:10:15 2016

Roll chromium_revision 355fabfffa..08daa0770a (383228:384471)

Change log: https://chromium.googlesource.com/chromium/src/+log/355fabfffa..08daa0770a
Full diff: https://chromium.googlesource.com/chromium/src/+/355fabfffa..08daa0770a

Changed dependencies:
* src/third_party/boringssl/src: https://boringssl.googlesource.com/boringssl.git/+log/4339552fbb..feaa57d13d
* src/tools/gyp: https://chromium.googlesource.com/external/gyp.git/+log/8e8c3178d9..28340b46df
DEPS diff: https://chromium.googlesource.com/chromium/src/+/355fabfffa..08daa0770a/DEPS

Clang version changed 263324:264334
Details: https://chromium.googlesource.com/chromium/src/+/355fabfffa..08daa0770a/tools/clang/scripts/update.py

BUG= chromium:595702 
TBR=pbos@webrtc.org
NOTRY=True

Review URL: https://codereview.webrtc.org/1847963004

Cr-Commit-Position: refs/heads/master@{#12190}

[modify] https://crrev.com/a312c3d88e868d57f33e5ed6c8fd81515159a46e/DEPS

kjellander's update only has our last clang roll but not the vs2015 cl in it (https://chromium.googlesource.com/chromium/src/+log/355fabfffa..08daa0770a/?s=56b0cbb91b8ae09e2304eb8e6503ebd729332c35; note "next" and "prev" links at the bottom), which also hints at this not being due to 2015.

One difference between our object files and MSVC's object files is that they always have a public symbol at the start of each .xdata and .pdata section. I didn't think they were necessary, as they are unreferenced, but maybe the incremental linker cares about them...

Did we use to have one there and then got rid of it? This used to work and broke somewhere in llvm 263324:263469 as far as we can tell, right?

…hey, that's not all that many commits. `svn log -r263325:263469 https://llvm.org/svn/llvm-project`...

The ones that jump out to me the most:


------------------------------------------------------------------------
r263377 | majnemer | 2016-03-13 04:01:03 -0400 (Sun, 13 Mar 2016) | 5 lines

[Bitcode] Make writeComdats less strange

It had a weird artificial limitation on the write side: the comdat name
couldn't be bigger than 2**16.  However, the reader had no such
limitation.  Make the reader and the writer agree.
------------------------------------------------------------------------
r263378 | majnemer | 2016-03-13 06:53:30 -0400 (Sun, 13 Mar 2016) | 16 lines

[CodeView] Truncate display names

Fundamentally, the length of a variable or function name is bound by the
maximum size of a record: 0xffff.  However, the name doesn't live in a
vacuum; other data is associated with the name, lowering the bound
further.

We would naively attempt to emit the name, causing us to assert because
the record would no-longer fit in 16-bits.  Instead, truncate the name
but preserve as much as we can.

While I have tested this locally, I've decided to not commit it due to
the test's size.

N.B.  While this behavior is undesirable, it is better than MSVC's
behavior.  They seem to truncate to ~4000 characters.

------------------------------------------------------------------------
r263408 | majnemer | 2016-03-14 01:15:09 -0400 (Mon, 14 Mar 2016) | 4 lines

[CodeView] Consistently handle overly large symbol names

Overly large symbol names weren't correctly handled for leaf function
records.
------------------------------------------------------------------------
r263464 | pcc | 2016-03-14 14:41:59 -0400 (Mon, 14 Mar 2016) | 8 lines

CodeGen: Mark functions used in vtables as unnamed_addr.

This marks virtual function declarations, as well as runtime library functions
__cxa_pure_virtual, __cxa_deleted_virtual and _purecall, as unnamed_addr. This
will allow us to correctly form relative references to them from vtables in
the relative vtable ABI.

Differential Revision: http://reviews.llvm.org/D18071
r263469 | pcc | 2016-03-14 15:07:10 -0400 (Mon, 14 Mar 2016) | 14 lines

CodeGen: Use 32-bit gep offsets to address vtable address points.

The relative vtable ABI will use a struct rather than an array as the type
of a vtable. LLVM only allows 32-bit integers as struct indices, so we need
to use 32-bit integers to get addresses of address points. In order to keep
the code simple, we might as well do that unconditionally.

It's probably a reasonable implementation limit to support no more than 2
billion virtual functions per class.

This change causes quite a bit of churn in the test suite, so I'm making
it separately.

Differential Revision: http://reviews.llvm.org/D18113



I think the [CodeView] commits should all not matter since the bots all build with fastbuild=1, so those are probably not it. pcc's changes are probably a noop on Windows (but maybe not, didn't look at the actual patch).


http://permalink.gmane.org/gmane.comp.compilers.llvm.cvs/310279 looks unlikely too though. Hm.

Yesterday I tried emitting $pdata$foo and $unwind$foo symbols into .pdata and .xdata like MSVC does, but it did not fix the problem.

I went back to r263275 and I still see these LNK1223 errors. This might not be a regression, maybe we just never saw it before.

Oh noes :-( So this is due to vs2015 after all and we were just "lucky" on some of the bots?

And this is still believe to be triggered by incremental linking? That seems to be a problematic feature in VS 2015.

Yes, it only reproduces when linking incrementally. Retrying the link a second time always fixes the link. We could hack gyp-win-tool to retry the link at most once when it sees LNK1223 in the output, but that's pretty horrible.

I think I could zip up these object files and send it to Microsoft with instructions on how to reproduce. I just don't know if they'd support us. They'll probably say that incremental linking is only supported when MSVC is the compiler, but it might be worth a try.

Bruce, could you pass those files along? Maybe they can tell us what's wrong with the input files at least, and then we can fix that.

I'm happy to help report the bug. The preferred way for reporting a linker bug is with a link repro which is Microsoft's format for packaging up all of the inputs (files, settings, etc.). Creating link repros is discussed here:

https://support.microsoft.com/en-us/kb/134650

Doing this in the context of ninja requires a bit of hacking. I think I modified ninja itself last time but there are lots of other options such as manually running the link command with the LINK_REPRO environment variable set.

It's hard to predict whether Microsoft will be interested in fixing this, but I think if it is an easy fix then they would probably be interested, or they could at least give us some hints on how to avoid the problem.

I think it'd be preferable if they could tell us what to change, since this isn't a problem with cl.

Other options, if that avenue doesn't work out:
* Use the 2013 link.exe on the clang bots
* Turn on lld on at least the 64bit dbg bots (the only config where we've seen this problem so far – I guess it only happens in debug builds, and only with 64-bit seh?)

Switching to LLD will probably work for these bots. Chromium only uses incremental linking for debug builds, and .pdata is 64-bit only, so yeah, only 64-bit debug has this problem.

I just want to verify that this works with the 2013 linker and then I'll try to make a linker reproduction. It's great that they have a tool for making a reproduction.

For the record, I went back to r260890 from Feb 15 in LLVM and I can still reproduce this problem.

I followed the LINK_REPRO steps, but the link fails without copying enough data to reproduce the problem. My reproduction directory only contains a copy of the PDB prior to incremental linking, and a short response file with no object files.

I'm going to try to make a zip file and see if I can unpack it elsewhere and still reproduce the issue.

Here is a ~130MB zip file that you can unpack anywhere to reproduce the problem:
https://drive.google.com/file/d/0B--mUSPjF09RcE1nX2lvbFQ4SEU/view?usp=sharing

The archive is mostly a copy of a Chromium out/Debug_x64 directory that's been pruned down. The .obj files in there were produced by clang-cl.

It also has two executables: ninja and "touchcoff.exe", which is something I wrote to update the timestamp in the header of a COFF object file. This timestamp, not the filesystem mtime, is what MSVC checks to decide if an object file has changed since the last link. I have hacked the build.ninja file to call my touchcoff program instead of a real compiler.

The repro_pdata.py script deletes base_unittests.exe to start from scratch, builds base_unittests.exe, and then forces a full incremental rebuild by tweaking the compiler command line.

This is all designed to simulate the scenario where somebody in Chromium touches a header file or -D flag, forcing a full recompile, but all the object files are functionally identical. In these kinds of situations we see this LNK1223 .pdata error.

Steps to reproduce:
- Unpack the zip
- Get a shell that has Python 2.7 and the VS2015 x64 tools on PATH
- In the unpacked linker-repro directory, run 'python repro_pdata.py'
- Observe the following output:

C:\Users\rnk\Downloads\linker-repro>python repro_pdata.py
Touching all object files and linking initial exe
ninja: Entering directory `.'
[808/808] LINK_EMBED base_unittests.exe
Touching all object files and linking incrementally, expecting LNK1223
ninja: Entering directory `.'
[808/808] LINK_EMBED base_unittests.exe
FAILED: python gyp-win-tool link-with-manifests environment.x64 True base_unittests.exe "python gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:base_unittests.exe @base_unittests.exe.rsp" 1 mt.exe rc.exe "obj\base\base_unittests.base_unittests.exe.intermediate.manifest" obj\base\base_unittests.base_unittests.exe.generated.manifest obj\base\compatibility.manifest
gtest.lib(gtest.gtest.obj) : fatal error LNK1223: invalid or corrupt file: file contains invalid .pdata contributions
Traceback (most recent call last):
  File "gyp-win-tool", line 317, in <module>
    sys.exit(main(sys.argv[1:]))
  File "gyp-win-tool", line 29, in main
    exit_code = executor.Dispatch(args)
  File "gyp-win-tool", line 71, in Dispatch
    return getattr(self, method)(*args[1:])
  File "gyp-win-tool", line 173, in ExecLinkWithManifests
    subprocess.check_call(ldcmd + add_to_ld)
  File "C:\python_27_amd64\files\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'python gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:base_unittests.exe @base_unittests.exe.rsp base_unittests.exe.manifest.res' returned non-zero exit status 1223
ninja: build stopped: subcommand failed.

If you don't see LNK1223, then you haven't reproduced the problem.

If I manually put the VS2013 linker first on PATH, then the problem goes away. You have to keep the 2015 libs on LIB and LIBPATH, or it won't link.

Hans was able to reproduce this on his machine.

So, what should we do with this information?

Bruce, can you pass this on to your link.exe contact at Microsoft, please? If they think the .obj files are incorrect we can fix clang-cl to make them emit what they expect.

Excellent repro. I filed a bug and have asked the VC++ team for comment.

Note that the simplest way to get the VS 2015 x64 tools on PATH is to run:

"%VS140COMNTOOLS%..\..\VC\vcvarsall.bat" amd64

I made a CL to switch the bot to lld (https://codereview.chromium.org/1881313002/) but that doesn't currently work: https://llvm.org/bugs/show_bug.cgi?id=27335 (I'm surprised our existing lld bots don't run into this, though -- probably this only happens in gn builds, and all lld bots are still on gyp.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9f230704fde72e9ae4a7a023b6e6c3d8666e920f

commit 9f230704fde72e9ae4a7a023b6e6c3d8666e920f
Author: Nico Weber <thakis@chromium.org>
Date: Wed Apr 13 13:36:52 2016

clang/win: Disable incremental linking.

MSVC 2015's linker regressed incremental linking with clang, so turn it off
for now.

Going forward, hopefully we can get that fixed, or we can switch to
linking with lld.

BUG= 595702 
TBR=dpranke

Review URL: https://codereview.chromium.org/1884973002 .

Cr-Commit-Position: refs/heads/master@{#386978}

[modify] https://crrev.com/9f230704fde72e9ae4a7a023b6e6c3d8666e920f/build/config/win/BUILD.gn

No longer blocking the default try bot.

We now know how to let clang-cl write .obj files that don't trigger this bug, so soon we'll be able to revert the change in comment 37 again.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/079ddd1507906bc030539fc33036dd5f6a9bbae0

commit 079ddd1507906bc030539fc33036dd5f6a9bbae0
Author: kjellander <kjellander@webrtc.org>
Date: Mon Apr 18 03:22:51 2016

Revert of CQ: Disable win_x64_clang_dbg trybot (patchset #1 id:1 of https://codereview.webrtc.org/1850113002/ )

Reason for revert:
I think the bug has been making enough progress for us to turn this back on (Chrome turned on their trybot).
I fired of builds and now have 6 green ones in a row.

Original issue's description:
> CQ: Disable win_x64_clang_dbg trybot
>
> Needed to unblock rolling chromium_revision in WebRTC DEPS.
>
> BUG= chromium:595702 
> TBR=phoglund@webrtc.org
>
> Committed: https://chromium.googlesource.com/external/webrtc/+/053fe8c6b14cb81880e37f9af3ea4fbde99e45fb

TBR=phoglund@webrtc.org
NOTRY=True
BUG= chromium:595702 

Review URL: https://codereview.webrtc.org/1897743002

Cr-Commit-Position: refs/heads/master@{#12399}

[modify] https://crrev.com/079ddd1507906bc030539fc33036dd5f6a9bbae0/infra/config/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/37286d141956d47c2b019dfee8bb48473809c1ef

commit 37286d141956d47c2b019dfee8bb48473809c1ef
Author: kjellander <kjellander@webrtc.org>
Date: Tue Apr 19 08:33:35 2016

Reland of CQ: Disable win_x64_clang_dbg trybot (patchset #1 id:1 of https://codereview.webrtc.org/1897743002/ )

Reason for revert:
We got a new error on this again, so I'm removing this trybot from the CQ again: https://build.chromium.org/p/tryserver.webrtc/builders/win_x64_clang_dbg/builds/2335/steps/compile/logs/stdio

Original issue's description:
> Revert of CQ: Disable win_x64_clang_dbg trybot (patchset #1 id:1 of https://codereview.webrtc.org/1850113002/ )
>
> Reason for revert:
> I think the bug has been making enough progress for us to turn this back on (Chrome turned on their trybot).
> I fired of builds and now have 6 green ones in a row.
>
> Original issue's description:
> > CQ: Disable win_x64_clang_dbg trybot
> >
> > Needed to unblock rolling chromium_revision in WebRTC DEPS.
> >
> > BUG= chromium:595702 
> > TBR=phoglund@webrtc.org
> >
> > Committed: https://chromium.googlesource.com/external/webrtc/+/053fe8c6b14cb81880e37f9af3ea4fbde99e45fb
>
> TBR=phoglund@webrtc.org
> NOTRY=True
> BUG= chromium:595702 
>
> Committed: https://crrev.com/079ddd1507906bc030539fc33036dd5f6a9bbae0
> Cr-Commit-Position: refs/heads/master@{#12399}

TBR=phoglund@webrtc.org
NOTRY=True
BUG= chromium:595702 

Review URL: https://codereview.webrtc.org/1896003004

Cr-Commit-Position: refs/heads/master@{#12421}

[modify] https://crrev.com/37286d141956d47c2b019dfee8bb48473809c1ef/infra/config/cq.cfg

Does webrtc also have the gyp bits that Nico added to disable incremental linking?

I may have time to fix the underlying cause this week.

Yes, we share the same //build with Chromium, but our bot is GYP, so it doesn't get what's in https://codereview.chromium.org/1884973002.
Will a similar CL be landed for GYP, as asked in the comment of that CL?

I re-enabled the bot after doing 6 green builds - I guess I should have taken a closer look at what actually was done in here first.

Fixed upstream in LLVM r268331, when we roll that into Chromium, we should try re-enabling incremental linking.

\o/

We rolled this in with https://codereview.chromium.org/1944713002

The roll bug: https://bugs.chromium.org/p/chromium/issues/detail?id=607312

Nico said that we have a 64-bit gyp bot somewhere that went green with this change, so I think this is fixed?

https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%28dbg%29?numbuilds=200 used to be red due to this fairly often. Here's a recent red build: https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin64%28dbg%29/builds/6023/steps/compile/logs/stdio

The bot's been green since your fixed went in, so chances are it's better now. (Since you had a reliable repro, verifying locally probably won't hurt either, but we'll turn incremental linking back on on win_clang in a day or two now, so if it's not fixed we'll see this there for sure.)

(Also, you fixed this, so changing owner back to you ;-) )

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5541c6da402958f5b3c77cf628f4e4bf45b7bda6

commit 5541c6da402958f5b3c77cf628f4e4bf45b7bda6
Author: thakis <thakis@chromium.org>
Date: Wed May 04 23:05:29 2016

clang/win: Reenable incremental linking.

llvm r268331 let clang-cl reate a unique .pdata section for every .text
section, which should unconfuse link.exe.

This reverts https://codereview.chromium.org/1884973002

BUG= 595702 

Review-Url: https://codereview.chromium.org/1945063003
Cr-Commit-Position: refs/heads/master@{#391677}

[modify] https://crrev.com/5541c6da402958f5b3c77cf628f4e4bf45b7bda6/build/config/win/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/c5614798414aa72f8232fd14947f3c0bc35fe69a

commit c5614798414aa72f8232fd14947f3c0bc35fe69a
Author: kjellander <kjellander@webrtc.org>
Date: Tue May 17 08:06:35 2016

Revert of CQ: Disable win_x64_clang_dbg trybot (patchset #1 id:1 of https://codereview.webrtc.org/1896003004/ )

Reason for revert:
With https://bugs.chromium.org/p/chromium/issues/detail?id=595702 being fixed 2 weeks ago and no new errors being reported. I'm attempting to re-enable this again.
I ran a couple of trial tryjobs for our win_x64_clang_dbg trybot and it seems to compile fine.

Original issue's description:
> Reland of CQ: Disable win_x64_clang_dbg trybot (patchset #1 id:1 of https://codereview.webrtc.org/1897743002/ )
>
> Reason for revert:
> We got a new error on this again, so I'm removing this trybot from the CQ again: https://build.chromium.org/p/tryserver.webrtc/builders/win_x64_clang_dbg/builds/2335/steps/compile/logs/stdio
>
> Original issue's description:
> > Revert of CQ: Disable win_x64_clang_dbg trybot (patchset #1 id:1 of https://codereview.webrtc.org/1850113002/ )
> >
> > Reason for revert:
> > I think the bug has been making enough progress for us to turn this back on (Chrome turned on their trybot).
> > I fired of builds and now have 6 green ones in a row.
> >
> > Original issue's description:
> > > CQ: Disable win_x64_clang_dbg trybot
> > >
> > > Needed to unblock rolling chromium_revision in WebRTC DEPS.
> > >
> > > BUG= chromium:595702 
> > > TBR=phoglund@webrtc.org
> > >
> > > Committed: https://chromium.googlesource.com/external/webrtc/+/053fe8c6b14cb81880e37f9af3ea4fbde99e45fb
> >
> > TBR=phoglund@webrtc.org
> > NOTRY=True
> > BUG= chromium:595702 
> >
> > Committed: https://crrev.com/079ddd1507906bc030539fc33036dd5f6a9bbae0
> > Cr-Commit-Position: refs/heads/master@{#12399}
>
> TBR=phoglund@webrtc.org
> NOTRY=True
> BUG= chromium:595702 
>
> Committed: https://crrev.com/37286d141956d47c2b019dfee8bb48473809c1ef
> Cr-Commit-Position: refs/heads/master@{#12421}

TBR=phoglund@webrtc.org
NOTRY=True
BUG= chromium:595702 

Review-Url: https://codereview.webrtc.org/1986753002
Cr-Commit-Position: refs/heads/master@{#12766}

[modify] https://crrev.com/c5614798414aa72f8232fd14947f3c0bc35fe69a/infra/config/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/b74488bbfee40805db67c625efd3fba0418d2181

commit b74488bbfee40805db67c625efd3fba0418d2181
Author: kjellander@chromium.org <kjellander@chromium.org>
Date: Wed May 18 11:58:14 2016

Revert of WebRTC: Disable Win64 Debug (Clang) bot (patchset #1 id:1 of https://codereview.chromium.org/1850103002/ )

Reason for revert:
The trybot is now working well, so this one should too.

Original issue's description:
> WebRTC: Disable Win64 Debug (Clang) bot
> 
> Needed to unblock rolling chromium_revision in WebRTC DEPS.
> 
> BUG= 595702 
> TBR=phoglund@chromium.org
> 
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299624

TBR=phoglund@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 595702 

Review-Url: https://codereview.chromium.org/1993693002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@300636 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/b74488bbfee40805db67c625efd3fba0418d2181/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win64_debug__clang_.json
[modify] https://crrev.com/b74488bbfee40805db67c625efd3fba0418d2181/scripts/slave/recipes/webrtc/standalone.py

The following revision refers to this bug:
  http://goto.ext.google.com/viewvc/chrome-internal?view=rev&revision=88016

------------------------------------------------------------------
r88016 | recipe-roller@chromium.org | 2016-05-18T12:03:18.434709Z

-----------------------------------------------------------------