I think the reasoning goes "there's no tot bot, so we do not support it". Let's revert https://chromium-review.googlesource.com/c/1299973 until there's a tot bot.

Ben, can you please help us here. The CQ for Win libfuzzer needs to be brought back and it is blocked on this.

 Issue 903305  has been merged into this issue.

Ben, just fyi, 
this needs to be similar to ToTLinuxASanLibfuzzer (linux tot libfuzzer bot)
https://cs.chromium.org/search/?q=%22ToTLinuxASanLibfuzzer%22&sq=package:chromium&type=cs

Doesn't the "Libfuzzer Upload Windows ASan" bot basically mirror this config?

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Windows%20ASan
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win-libfuzzer-asan-rel

Maybe we just need to make sure it is showing up in sheriff-o-matic? It looks like it might be considered an FYI bot at the moment.

> I think the reasoning goes "there's no tot bot, so we do not support it"

I certainly wasn't aware of this requirement. I guess we're all getting surprised today. Is every bot on the CQ supposed to have a clang ToT mirror? Or just some specific class of bots?

I'll start spinning up the new bot... that's 99% of my job these days anyway.

"Libfuzzer Upload Windows ASan" builds fuzz targets and uploads them to ClusterFuzz. As I understand, it doesn't solve either of the following problems:

1) Make sure that newer LLVM revisions do not break our workflow

2) Make sure Chromium developers do not land changes that breaks the build

Dirk, are you saying that there is a way to leverage this bot to solve any of these two issues?

re #c9 - let's have this discussion in bug 893614 instead.

https://chromium-review.googlesource.com/c/chromium/tools/build/+/1327321
https://chromium-review.googlesource.com/c/chromium/src/+/1327147
https://chrome-internal-review.googlesource.com/c/infradata/config/+/713729

> Is every bot on the CQ supposed to have a clang ToT mirror?

No, only bots that add new kinds of dependencies on the clang zip file. We update that zip every few weeks, and unless we have tot bots for everything that uses it, we only learn that things are broken when we try to roll. Rolling requires first pushing to goma and is generally somewhat involved, so we want to know about issues as soon as they appear.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1a4a9af0873b1f992f6ab20a632374e3ae39c2c6

commit 1a4a9af0873b1f992f6ab20a632374e3ae39c2c6
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Nov 08 21:51:20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a3e348a8e1f546affe7205cefed1764289d1196e

commit a3e348a8e1f546affe7205cefed1764289d1196e
Author: Ben Pastene <bpastene@chromium.org>
Date: Sat Nov 10 03:10:20 2018

Add src-side configs for clang ToTWinASanLibfuzzer builder.

Bug: 903078
Change-Id: I30666c534644b6ccf4909456b50a47d510abff4a
Reviewed-on: https://chromium-review.googlesource.com/c/1327147
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Jonathan Metzman <metzman@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607093}
[modify] https://crrev.com/a3e348a8e1f546affe7205cefed1764289d1196e/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/a3e348a8e1f546affe7205cefed1764289d1196e/infra/config/global/luci-milo.cfg
[modify] https://crrev.com/a3e348a8e1f546affe7205cefed1764289d1196e/infra/config/global/luci-scheduler.cfg
[modify] https://crrev.com/a3e348a8e1f546affe7205cefed1764289d1196e/tools/mb/mb_config.pyl

Thanks!

I can see the bot now: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ToTWinASanLibfuzzer

But it fails with "Incorrect or missing bot configuration". I guess it needs more patches?

Yes, I'm waiting for OWNERS approval (hans/thakis) on https://chromium-review.googlesource.com/c/chromium/tools/build/+/1327321

Sorry, I missed that. Stamped it now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/37c76f582c9486e18a4d348889130a50e51410bc

commit 37c76f582c9486e18a4d348889130a50e51410bc
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Nov 13 08:10:09 2018

Add recipe configs for clang ToTWinASanLibfuzzer builder.

Bug: 903078
Change-Id: I2d1cec71e352cfc7e0486f00f6fe75c591788727
Reviewed-on: https://chromium-review.googlesource.com/c/1327321
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>

[modify] https://crrev.com/37c76f582c9486e18a4d348889130a50e51410bc/scripts/slave/recipe_modules/chromium_tests/chromium_clang.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/133f5790775a57f71f1f889000c3e00b06497e6b

commit 133f5790775a57f71f1f889000c3e00b06497e6b
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Nov 13 22:18:19 2018

Don't use goma for WinLibfuzzer clang ToT bot.

Looks like no other tot bot uses goma, and the bot's currently failing
since the goma client isn't available.

Bug: 903078
Change-Id: I14e9a3772f3df464d5e6549fd785b20bc10d0509
Reviewed-on: https://chromium-review.googlesource.com/c/1334272
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607766}
[modify] https://crrev.com/133f5790775a57f71f1f889000c3e00b06497e6b/tools/mb/mb_config.pyl

 Issue 893256  has been merged into this issue.

Still there is some failure left here -

FAILED: test_child_process.exe test_child_process.exe.pdb 
ninja -t msvc -e environment.x64 -- ../../third_party/llvm-build/Release+Asserts/bin/lld-link.exe /nologo /OUT:./test_child_process.exe /PDB:./test_child_process.exe.pdb @./test_child_process.exe.rsp
lld-link: error: could not open clang_rt.fuzzer_no_main-x86_64.lib: no such file or directory
[107/61939] LINK(DLL) pe_image_test.dll pe_image_test.dll.lib pe_image_test.dll.pdb
FAILED: pe_image_test.dll pe_image_test.dll.lib pe_image_test.dll.pdb 
ninja -t msvc -e environment.x64 -- ../../third_party/llvm-build/Release+Asserts/bin/lld-link.exe /nologo /IMPLIB:./pe_image_test.dll.lib /DLL /OUT:./pe_image_test.dll /PDB:./pe_image_test.dll.pdb @./pe_image_test.dll.rsp
lld-link: error: could not open clang_rt.fuzzer_no_main-x86_64.lib: no such file or directory
[108/61939] LINK(DLL) base_profiler_test_support_library.dll base_profiler_test_support_library.dll.lib base_profiler_test_support_library.dll.pdb
FAILED: base_profiler_test_support_library.dll base_profiler_test_support_library.dll.lib base_profiler_test_support_library.dll.pdb 
ninja -t msvc -e environment.x64 -- ../../third_party/llvm-build/Release+Asserts/bin/lld-link.exe /nologo /IMPLIB:./base_profiler_test_support_library.dll.lib /DLL /OUT:./base_profiler_test_support_library.dll /PDB:./base_profiler_test_support_library.dll.pdb @./base_profiler_test_support_library.dll.rsp
lld-link: error: could not open clang_rt.fuzzer_no_main-x86_64.lib: no such file or directory

Jonathan, have you seen this before ? https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8929789000286684080/+/steps/compile/0/stdout

It looks like https://chromium-review.googlesource.com/c/chromium/src/+/1315361 stopped shipping the fuzzer runtime for Windows.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/66dabdf3786dda320bbdc3d0427709d1d7d26c42

commit 66dabdf3786dda320bbdc3d0427709d1d7d26c42
Author: Jonathan Metzman <metzman@chromium.org>
Date: Thu Nov 15 17:14:36 2018

[libFuzzer][Windows] Ship runtime again

LibFuzzer runtime on Windows was accidentally removed, add it back.

Bug:  905664 ,903078
Change-Id: Ie6343d02c6d13e18c71dee11b5219c0bb4e81dd0
Reviewed-on: https://chromium-review.googlesource.com/c/1337809
Commit-Queue: Jonathan Metzman <metzman@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#608413}
[modify] https://crrev.com/66dabdf3786dda320bbdc3d0427709d1d7d26c42/tools/clang/scripts/package.py

The bot's still failing with "lld-link: error: could not open clang_rt.fuzzer_no_main-x86_64.lib: no such file or directory"

Looks like that's getting added here:
https://codesearch.chromium.org/chromium/src/build/config/sanitizers/BUILD.gn?rcl=99162f9e29b36c069d244c6f5dc432674504085b&l=260

Does the bot need something that it's not getting?

Yeah, I've accidentally deleted it recently and since then we were waiting for a minor clang roll which was blocked due to some problem on the bots.

K. LMK when we expect it to start working. Hopefully then we can close this.

> Yeah, I've accidentally deleted it recently and since then we were waiting for a minor clang roll which was blocked due to some problem on the bots.

This is a ToT bot, i.e. it uses tip-of-tree Clang and isn't affected by Clang rolls. There must be some other problem, e.g. the library isn't getting built by update.py or it's not in the place where the build expects to find it.

Hans, I don't think that this bot is Clang ToT bot. It's on the main waterfall and (as I understand) its purpose is to simply mirror the config we're going to have in the CQ.

To be clear, I mean https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ToTWinASanLibfuzzer

"ToTWinASanLibfuzzer" has "ToT" right in the name, it better be a tot bot :-) If it isn't, that's a bug.

You're right though that we need two win-libfuzzer bots:

1. A clang tot bot doing win-fuzz builds to make sure tot clang works
2. A main waterfall win-fuzz bot to make sure the win-fuzz cq bot works

> 1. A clang tot bot doing win-fuzz builds to make sure tot clang works

That's https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/ToTWinASanLibfuzzer

> 2. A main waterfall win-fuzz bot to make sure the win-fuzz cq bot works

That's https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Windows%20ASan


Looks like the tot bot has gotten past the error in #25, but is still failing, this time with:

FAILED: obj/testing/libfuzzer/tests/libfuzzer_tests/fuzzer_launcher_test.obj 
../../testing/libfuzzer/tests/fuzzer_launcher_test.cc(23,24):  error: no matching member function for call to 'Append'
    exe_path.DirName().Append("check_fuzzer_config.py").value();
from https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/ToTWinASanLibfuzzer/496

This is a bit out of my wheelhouse. Can someone take this bug to fix/investigate the compile failures?

Thanks Ben, looks like Jonathan has uploaded a fix as https://chromium-review.googlesource.com/c/chromium/src/+/1342772

> "ToTWinASanLibfuzzer" has "ToT" right in the name, it better be a tot bot :-) If it isn't, that's a bug.


Ok, I got confused because I haven't seen any clang / LLVM related build step, but then I've found out that the bot does that in gclient_runhooks step. No bugs in there! :)

For some reason the issue in #25 still occurs intermittently.
It occurs in build 517 for example: https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/ToTWinASanLibfuzzer/517

I assume once the CL linked to in #33 lands it will occur all of the time.

It's still a mystery to me why this failure occurs since I thought I fixed the cause in #24. Will try to solve that issue next.

I think clang_rt.fuzzer_no_main-x86_64.lib is not getting built because the clang built in gclient runhooks (https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8929332448205975136/+/steps/gclient_runhooks/0/stdout) is built with cl.exe MSVC. libFuzzer on Windows only builds with clang.

I assumed this was fine since I thought chrome clang is built with clang?

The two options are:

1. Changing libFuzzer to build with MSVC (either directly or with the newly built clang through cmake magic)

2. Changing the bot to build clang with clang.

I was already planning on doing option 1, 

I don't think I will get a chance to fix this issue with MSVC before next week.

Pursuing 1 seems like it should be the most straightforward. I seem to recall that Hans wanted to try to bring order to the update.py vs package.py clang scripts (https://crbug.com/884608), so maybe 2 will happen later.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2656ecd2302c696fffc71f5b7d2f3edffce0ea99

commit 2656ecd2302c696fffc71f5b7d2f3edffce0ea99
Author: Jonathan Metzman <metzman@chromium.org>
Date: Wed Nov 21 08:22:41 2018

[libFuzzer][Windows] Fix for ToT Clang bot

Don't allow building of afl_driver since it isn't intended to be built
on Windows.

Change-Id: I8aeb89b5cf98f1998258847b60a8e12a465d3ca9
Bug: 903078
Reviewed-on: https://chromium-review.googlesource.com/c/1342772
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609949}
[modify] https://crrev.com/2656ecd2302c696fffc71f5b7d2f3edffce0ea99/BUILD.gn
[modify] https://crrev.com/2656ecd2302c696fffc71f5b7d2f3edffce0ea99/testing/libfuzzer/tests/BUILD.gn
[modify] https://crrev.com/2656ecd2302c696fffc71f5b7d2f3edffce0ea99/third_party/libFuzzer/BUILD.gn

Over to metzman for the follow-ups mentioned in #36/#37.

The NextAction date has arrived: 2018-11-28

metzman, this bot is still failing compile. How is your fix coming along?

To be clear, while this bot is red we can't know beforehand if clang rolls are going to break win/fuzzer builds, so until the bot is green win/fuzzer issues can't block clang rolls.

>metzman, this bot is still failing compile. How is your fix coming along?

Sorry it has taken so long, the fix is somewhat complicated and I got distracted by other things I needed to finish this quarter. I'll try to finish up after the holidays.

One fix I considered was making sure libFuzzer is built with the newly built clang (in the same way that integration tests use clang) instead of the compiler (MSVC) used to build clang. 
But I haven't had much luck with this. 
I'll try to investigate this more because the other solution seems like it will be ugly. 
If you have any ideas on how to do this I would really appreciate since I don't know CMake.

The other solution, which I've had more success with, is changing the libFuzzer source code to build with MSVC directly. 
This is turning out ugly because libFuzzer is full of non-portable code, and I need to use macros everywhere there is an __attribute__ or __builtin (https://cs.chromium.org/search/?q=+file:%5Esrc/third_party/libFuzzer/src/+package:%5Echromium$+(builtin+OR+__attribute__)&type=cs) and in other places as well. 
There are other issues beyond this, such as the MSVC linker behaving differently than the clang linker (which I've figured out how to work around, though I'm unsure upstream will want it), and weak symbols not existing in MSVC-world (I haven't figured this out and libFuzzer is very dependent on it). 
I'm inclined to look at building libFuzzer with clang during MSVC builds instead of completing this fix as it seems harder to maintain.

>To be clear, while this bot is red we can't know beforehand if clang rolls are going to break win/fuzzer builds, so until the bot is green win/fuzzer issues can't block clang rolls.

It looks like libFuzzer on Windows survived the last clang roll (https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Windows%20ASan/3182 succeeded). Phew!
But I'll try to finish this fix so as to not rely on luck again.

LLVM's code is supposed to build with MSVC, so making the libfuzzer code build with MSVC should definitely be done.

The weak symbols issues is also one that needed to be solved for the sanitizers, maybe that same approach could work here? The symbol visibility issues are a property of running on Windows though and are independent of building with cl.exe or clang-cl, right?

>LLVM's code is supposed to build with MSVC, so making the libfuzzer code build with MSVC should definitely be done.

I'll try it, but the libFuzzer devs told me they prefer the other solution. My experience is that they also dislike having system-dependent #ifdefs in their code. 

>The weak symbols issues is also one that needed to be solved for the sanitizers, maybe that same approach could work here?

I think so. I've started looking at ASAN for inspiration.

>The symbol visibility issues are a property of running on Windows though and are independent of building with cl.exe or clang-cl, right?

I think weak symbols work on Windows if clang-cl is used: that is why when I include "weak" as an attribute in https://cs.chromium.org/chromium/src/third_party/libFuzzer/src/FuzzerExtFunctionsWeakAlias.cpp libFuzzer targets successfully find functions such as __sanitizer_acquire_crash_state and LLVMFuzzerInitialize which aren't defined by libFuzzer. They don't get found if I remove "weak".

I'm happy to talk to libFuzzer devs if you get pushback. libFuzzer is an LLVM project and gets to play by LLVM's rules.

Re __attribute__((weak, alias...): Maybe 

#pragma comment(linker, "/alternatename:foo=FoODef")

could work? Looks like both link.exe and lld-link support this. There's also declspec(selectany) but it probably doesn't quite do the right thing.

>Re __attribute__((weak, alias...): Maybe 

>#pragma comment(linker, "/alternatename:foo=FoODef")

Thanks! I think I saw ASAN doing this. I'll try it out soon.

>There's also declspec(selectany) but it probably doesn't quite do the right thing.

Right, I tried it out, I'm not sure it works for functions.

Just wanted to ping this and say I'm working on it now. 
The solution in #48 works for MSVC but it seems to badly break libFuzzer when built with Clang by causing Function1 == Function2 to return false for two function pointers with the same value. Looking into this now (along with other less worrying problems =)

If you have something in comment 48 that works with MSVC's cl.exe but breaks with clang-cl, we (clang-cl team) consider that a clang-cl bug and want to fix it. Maybe you can make a repro and file a bug for it? Maybe we can fix that in clang-cl and then use that approach.

>If you have something in comment 48 that works with MSVC's cl.exe but breaks with clang-cl, we (clang-cl team) consider that a clang-cl bug and want to fix it. Maybe you can make a repro and file a bug for it? 

OK. I'm looking into the issue since it may just be something silly on my part. 

>Maybe you can make a repro and file a bug for it?

Will try to get a repro too.

(note mostly for me) It seems like the problem is related to templating, since this (https://github.com/llvm-mirror/compiler-rt/blob/master/lib/fuzzer/FuzzerExtFunctionsWeakAlias.cpp#L35) comparison is incorrectly true when done in the template function but is false if I cast Fun and FunDef to void* and do the comparison in another function..

I have a reproducer and I'm more confident that there's a compiler bug.

Filed https://bugs.llvm.org/show_bug.cgi?id=40218.

The reproducer is so small that I'm surprised this hasn't broken so many things that it someone else noticed it. Maybe there is an issue my local set up.

Turns out the bug I filed in #53 is a known problem that I will have to hack around.

I solved the big problem (not being able to build libFuzzer with MSVC), but the bot is still red. Need to make some small chrome-side changes before it turns green.