Implementing this is not going to be completely straightforward; I think in order to do this we're going to need to change the `chromium` and/or `chromium_trybot` recipes a bit.

The reason for this is that I think we'd like to run this like a test, i.e., a GN failure for the non-primary config shouldn't abort the build (assuming the primary config doesn't fail, we should still compile it and run other tests). However, we will need to run this even if `analyze` thinks that there is no work to do for the primary config.

I think changing the recipe(s) is easy enough, but it does require a change.

Also, as a sketch of how this would be implemented, I think we want some sort of config file entries that can indicate which configs should be generated (verified) per-builder, and then a script that runs each config, generating the files into a temp directory under //out . 

Given that GN is relatively fast (a few seconds per config on linux), there's no advantage to trying to run this under swarming, and indeed doing so would be difficult because we wouldn't easily know what files we'd need. If we wanted to optimize things, we could use ninja to only re-generate the files if the build directory already existed and the args for the config didn't change, but I think it's probably better to keep things simpler for the start rather than over-designing up front to save a few seconds.

Having thought about this some more just now, I think my comments in #2 and #3 were probably off-base.

I suspect it's actually easier to write a completely new recipe that does the following:

- check out the code
- apply the patch
- Read a config file from somewhere in the repo that pulls a list of 
  (mastername, buildername) pairs to test
- Runs MB for each pair.

Given that these configs should always work, there's probably no reason to retry
patches, meaning that this recipe should be very simple to write.

@phajdan.jr - what do you think of this new idea? Feel like taking a stab at it? If not, bounce it back to me.

Let me raise the priority here, since this is heavily needed (e.g. https://bugs.chromium.org/p/chromium/issues/detail?id=714681, builds were broken for 4 days).

Deferring to Dirk/Erik for further assignment of this bug and what I should prioritize.

Erik, could you please help us there as per c#8?

Paweł, if you can take this it would be appreciated. It sounds like there's a clear path forward with the proposal in comment 6 and it's fairly important.

It seems like there's yet maybe another option here, which is to set up a bot in the CQ that generates the GN config but doesn't actually compile everything; maybe it would compile nothing, or maybe it could compile only a few targets?

If we wanted to take that approach, pretty much anyone could configure the bot and you wouldn't need to write a new recipe.

I'd be glad to assist you if you have any questions.

Based on the comments so far the sentiment seems to be even though marked as a priority, the bug has been sitting for ~9 months. Additionally, the task is described as easy, and Infra is moving to self-service for users where possible.

If any of these turn out not to be true (e.g. the task is harder than originally thought), we could revisit that of course.

Again, if you have any questions, please just contact me.

phajdan.jr@, may I ask you to clarify to whom the last comment is directed?

Hi Max, sorry, I think Paweł wanted to see if doing what Dirk mentioned in comment 11 was something you could help with. If you have some cycles but that's too high level of a description we can outline the actual steps. What do you think?

Thanks for clarifications! Ok, let's try to do this. There are a couple more things needed to be implemented on build infra side, so we'll try to address them altogether in Q3.

Unassigning myself as I've never actually had time / planned to do that.

Dirk, do you think this is something CCI team might be able to pick up?

(marking untriaged since it's already in the right component if so)

We're trying to resolve this with https://chromium-review.googlesource.com/c/chromium/src/+/1071875

That CL mentioned in c#21 doesn't seem to work. Several tests are failing with errors which do not really make any sense to me. A good reproducer, for example, jingle_unittests.

It can be built in an existing configuration:

dcheck_always_on = true
is_asan = true
is_component_build = false
is_debug = false
is_lsan = true
strip_absolute_paths_from_debug_symbols = true
symbol_level = 1
use_goma = true

And all the tests should pass.


If we build it in the same configuration + use_libfuzzer=true, some of the tests will fail. Looks like all of them are related to logging.

Filed  issue 847106  for c#22, as it affects the config used on code coverage bot as well.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2b4c02961570955e40d85646feb0865bb79fe661

commit 2b4c02961570955e40d85646feb0865bb79fe661
Author: Max Moroz <mmoroz@chromium.org>
Date: Tue May 29 17:54:14 2018

Make FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION depend on optimize_for_fuzzing GN arg.

Bug:  847106 , 646404 
Change-Id: I869ece467fc4783fa4c9dbb0984e8a146b5d7a7a
Reviewed-on: https://chromium-review.googlesource.com/1074830
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Oliver Chang <ochang@chromium.org>
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#562499}
[modify] https://crrev.com/2b4c02961570955e40d85646feb0865bb79fe661/build/config/compiler/BUILD.gn
[modify] https://crrev.com/2b4c02961570955e40d85646feb0865bb79fe661/build/config/sanitizers/BUILD.gn
[modify] https://crrev.com/2b4c02961570955e40d85646feb0865bb79fe661/build/config/sanitizers/sanitizers.gni

mmoroz@: since you're effectively workin on this, do you mind taking this bug?
Assigning to you for now; if you think Infra should own it though, please set back to Untriaged and mentioned what we can help with. Thanks!

The following revision refers to this bug:
  https://webrtc.googlesource.com/src.git/+/a61fa6ea363ce9eefd1b02aa38ad38ed47b6f52d

commit a61fa6ea363ce9eefd1b02aa38ad38ed47b6f52d
Author: Max Moroz <mmoroz@chromium.org>
Date: Wed May 30 04:28:28 2018

Rely on use_fuzzing_engine && optimize_for_fuzzing to define WEBRTC_UNSAFE_FUZZER_MODE.

Depends on https://chromium-review.googlesource.com/c/chromium/src/+/1074830.

Bug:  chromium:847106 ,  chromium:646404 , chromium:844647
Change-Id: I401d518646e032e24989d343c00c307563e23ddf
Reviewed-on: https://webrtc-review.googlesource.com/79187
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Henrik Lundin <henrik.lundin@webrtc.org>
Reviewed-by: Patrik Höglund <phoglund@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#23438}
[modify] https://crrev.com/a61fa6ea363ce9eefd1b02aa38ad38ed47b6f52d/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c256b85606bf30beee3e7a00fac38ed7739d9f11

commit c256b85606bf30beee3e7a00fac38ed7739d9f11
Author: webrtc-chromium-autoroll <webrtc-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed May 30 08:39:19 2018

Roll src/third_party/webrtc 183f4d9..700fc84 (3 commits)

https://webrtc.googlesource.com/src.git/+log/183f4d9..700fc84


git log 183f4d9..700fc84 --date=short --no-merges --format='%ad %ae %s'
2018-05-29 buildbot@webrtc.org Roll chromium_revision 577ba36372..9d94097261 (562638:562745)
2018-05-28 mmoroz@chromium.org Rely on use_fuzzing_engine && optimize_for_fuzzing to define WEBRTC_UNSAFE_FUZZER_MODE.
2018-05-29 buildbot@webrtc.org Roll chromium_revision eb7233cee8..577ba36372 (562409:562638)

Created with:
  gclient setdep -r src/third_party/webrtc@700fc84

The AutoRoll server is located here: https://webrtc-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_archive_rel_ng;master.tryserver.chromium.mac:mac_chromium_archive_rel_ng

BUG=chromium:None,chromium:847106,chromium:646404,chromium:844647,chromium:None

TBR=webrtc-chromium-sheriffs-robots@google.com

Change-Id: I8d7165af747083d3b3952b81d86d49ab7c347e6e
Reviewed-on: https://chromium-review.googlesource.com/1078073
Reviewed-by: webrtc-chromium-autoroll <webrtc-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: webrtc-chromium-autoroll <webrtc-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#562764}
[modify] https://crrev.com/c256b85606bf30beee3e7a00fac38ed7739d9f11/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e10ba490518329322a4bd1b7e57f65a44a197aa8

commit e10ba490518329322a4bd1b7e57f65a44a197aa8
Author: Max Moroz <mmoroz@chromium.org>
Date: Wed May 30 17:18:43 2018

[libFuzzer] Add use_libfuzzer=true GN flag to linux_chromium_asan_rel_ng trybot.

This flag should not affect any non fuzz targets, but having it enabled would
allow us to prevent some breakages, e.g. adding a fuzz target with unresolved
GN dependencies. See https://crbug.com/845774#c4 for more context.

Depends on https://chromium-review.googlesource.com/c/chromium/src/+/1074830 and
https://webrtc-review.googlesource.com/c/src/+/79187.

R=dpranke@chromium.org, inferno@chromium.org

Bug: 640271, 646404 ,845774
Change-Id: I3e96e9f15e1d0540a7667ea96059cf4d1b1a8eb8
Reviewed-on: https://chromium-review.googlesource.com/1071875
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#562882}
[modify] https://crrev.com/e10ba490518329322a4bd1b7e57f65a44a197aa8/tools/mb/mb_config.pyl

And now when it's finally in I should try uploading a fuzz target with a broken GN config!

Uploaded https://chromium-review.googlesource.com/c/chromium/src/+/1079459 to reproduce one of the recent breakages (issue 845689).

Yay, that works: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_asan_rel_ng/22827

Looks like this is done. We still have issue 640271 and issue 845774 though.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8585b5468aaa36749984917e05b9034792eb8e83

commit 8585b5468aaa36749984917e05b9034792eb8e83
Author: John Budorick <jbudorick@chromium.org>
Date: Fri Jun 01 06:03:38 2018

Revert "[libFuzzer] Add use_libfuzzer=true GN flag to linux_chromium_asan_rel_ng trybot."

This reverts commit e10ba490518329322a4bd1b7e57f65a44a197aa8.

Reason for revert: FindIt suspects this is responsible for  crbug.com/848278 

Original change's description:
> [libFuzzer] Add use_libfuzzer=true GN flag to linux_chromium_asan_rel_ng trybot.
> 
> This flag should not affect any non fuzz targets, but having it enabled would
> allow us to prevent some breakages, e.g. adding a fuzz target with unresolved
> GN dependencies. See https://crbug.com/845774#c4 for more context.
> 
> Depends on https://chromium-review.googlesource.com/c/chromium/src/+/1074830 and
> https://webrtc-review.googlesource.com/c/src/+/79187.
> 
> R=​dpranke@chromium.org, inferno@chromium.org
> 
> Bug: 640271, 646404 ,845774
> Change-Id: I3e96e9f15e1d0540a7667ea96059cf4d1b1a8eb8
> Reviewed-on: https://chromium-review.googlesource.com/1071875
> Commit-Queue: Max Moroz <mmoroz@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Abhishek Arya <inferno@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#562882}

TBR=inferno@chromium.org,dpranke@chromium.org,mmoroz@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 640271,  646404 , 845774
Change-Id: I701f573168a99996560664efb842627ff9c1ea70
Reviewed-on: https://chromium-review.googlesource.com/1081972
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563545}
[modify] https://crrev.com/8585b5468aaa36749984917e05b9034792eb8e83/tools/mb/mb_config.pyl

Available again since the revert in c#33.

-> untriaged to get into our queue.

mmoroz: what do you want here as far as:
 - gn configuration (same as linux_chromium_asan_rel_ng + use_libfuzzer=true? something else?)
 - build targets
 - test targets
 - OSes (just linux?)

The build part should be similar to https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Linux%20ASan, so:

1) GN config:
enable_nacl = false
ffmpeg_branding = "ChromeOS"
goma_dir = "/b/swarming/w/ir/cache/goma/client"
is_asan = true
is_component_build = false
is_debug = false
optimize_for_fuzzing = true
pdf_enable_xfa = true
proprietary_codecs = true
strip_absolute_paths_from_debug_symbols = true
use_goma = true
use_libfuzzer = true

It's also defined here: https://cs.chromium.org/chromium/src/tools/mb/mb_config.pyl?l=1381&rcl=9b0efaf66eb7329309a36179e8c07cb508f04977


2) build targets:

That might be a tough one, as we've discussed, the list is dynamic and is calculated by this code: https://cs.chromium.org/chromium/build/scripts/slave/recipes/chromium_libfuzzer.py?l=114&rcl=ae680ad4796b67db729532e35a53762c2e6fb829

If there is no way to do something like that, we have to find another solution. Maybe something like auto-generated file in with the list of targets. But I really hope that we can do "gn refs" on a bot.


3) test targets:

For start, I'd just run each of these fuzz targets with a single command line argument: "-runs=4". That would be just a quick sanity check that they aren't broken.

4) OSes:

Linux is enough for now!

Triager here. Few questions:

- Is this a dupe of bug 845774? Can we merge one into the other?

- It appears the CL that #33 reverted was sufficient to fix this bug. Since that's much less of a maintenance burden than a new set of CQ+CI bots, is it feasible to instead track down why the change was reverted in the first place, fix it, and reland?

- Chromium's main recipe does not support any "gn ref" logic to determine what to build. What's in the chromium_libfuzzer.py can be incorporated into it, but, like above, that's more of a tech burden than simply adding a flag to an existing builder.

All that said, if we still want new bots, I can handle it.

> - Is this a dupe of bug 845774? Can we merge one into the other?

Yes, those are dupes. Let me merge them.


> - It appears the CL that #33 reverted was sufficient to fix this bug. Since that's much less of a maintenance burden than a new set of CQ+CI bots, is it feasible to instead track down why the change was reverted in the first place, fix it, and reland?

Not really. That change was an attempt to get a little bit of CQ coverage for a relatively low price. The only thing it could verify is that enabling "use_libfuzzer=true" GN arg doesn't break the whole build generation process. Even if it worked out, we still would not build and run any fuzz targets, which is the most essential part.


> - Chromium's main recipe does not support any "gn ref" logic to determine what to build. What's in the chromium_libfuzzer.py can be incorporated into it, but, like above, that's more of a tech burden than simply adding a flag to an existing builder.

Can we execute shell commands from main recipe? I suspect that we can find some workaround for dynamic detection of fuzz targets, but it feels much better to me to find a way to support calling "gn refs" from main recipe.


> All that said, if we still want new bots, I can handle it.

Yes, please!

Issue 845774 has been merged into this issue.

> Can we execute shell commands from main recipe?

We can. There's nothing technically blocking it, just needs the cycles for someone to add that functionality.

I think it'd be best to add this to the main chromium & chromium_trybot recipe. That way, we could reuse it for any future bots, and we'll get patch applying & bot configurations for free. One idea is to

- Map a buildername to a specific target to run "gn refs" on via the testing/buildbot/ configs. We already have "additional_compile_targets" field in the configs at https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/waterfalls.pyl. I think we could reasonably add two more optional fields there, something like "target_refs_inclusion" and "target_refs_exclusion". (+kbr in case he has thoughts on this part)

- Make the recipe run gn refs on that target and add the output to the list of build targets. Prob somewhere between "run_mb()" and "compile()" in:
https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/api.py?rcl=bb5e3b9b5409bba053c841707e0c9c022660f1e0&l=592

Once that's done, it should be like any other bot and would just need some dumb config changes to get it running. Unless someone has a better idea, I'll plan on tackling that.

Not sure I understand all of the intricacies here, but in general I would advocate as general a solution as possible. If what really needs to happen is to differently compute the list of build targets, then let's define a contract between the Chromium recipe and some scripts in src/testing/ (maybe a new subdirectory like src/testing/target_scripts / target_computation / target_computation_scripts), have waterfalls.pyl specify this script and some args, and have the recipe invoke it at the appropriate time in order to override the results from "analyze".

Basically: encoding the fact that we want to run "gn refs" against a particular target sounds a little over-specific.

Either way works for us. If our buildbot is the only customer of "gn refs" and/or you anticipate that gn may go away in some reasonable time, having an extra script for detecting targets sounds like a safer choice indeed.

I don't anticipate either of those, but it'd be better to add less mechanism to the Chromium recipe and more mechanism in the Chromium workspace.

Re #41: That sounds reasonable, and I'll look into adding something along those lines. Though I do think it is possible to go too far in the other direction as well, ie: making things too general/open-ended. eg: What's to stop someone from adding a script that decides what targets to build by curling some file in google storage? We certainly don't want to be enabling that kind of behavior, and I worry that adding this kind of functionality may open the door to cases like that.

But I'm probably over-thinking things... Anyway, I'll try to get started on this soon.

Thinking this over a little more, I'm not sure how normal trybot analyze/filtering would best play into the dynamic set of targets we want built.

Essentially, we'd want the output of "gn refs" to get copied into the "additional_compile_targets" field in the input json of the analyze step. That way we refrain from building targets that the CL's change doesn't affect.

But to calculate the refs, we need a fully gen'ed out dir. We run "gn gen" for the first time in a build during the analyze step. So we need to gen prior to analyze, then run "gn refs", and subsequently attach its output to analyze's input.

A few options I can think of:
- Move the "generate_build_files" step before analyze. That would give us a build directory we can use to run "gn refs" prior to analyze.
- Add an option to mb's analyze flow to optionally run "gn refs" on specified labels and adding the output to the input.json it's passed. (Or maybe add a "refs" command to mb altogether and invoke that straight from the recipe.)

I'd lean towards option #2, but that would make this solution pretty tailor-made. +dirk/jbud for thoughts/ideas.

I don't think we should try to add this to the generic chromium/chromium_trybot recipes; what they want to do with the fuzz targets is too different from what we normally do on the bots.

In addition, I didn't think we wanted to build these targets at all, because I thought we thought the cycle time for building them was two hours or more?

mmoroz / inferno: do you have any stats on how often things break (and where), so we could gauge what the actual impact of various things might be?

Don't have an exact stat, but IMO most frequent case if compilation / linking failure, e.g. the most recent one: https://bugs.chromium.org/p/chromium/issues/detail?id=871587

Some older breakages to take a look at: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=libfuzzer+build+builds+broken+Pri%3D0&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids


We have to build those targets. But the good news is that with goma compilation is much faster than it used to be, e.g.:

- 23 min: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Linux%20ASan/19017
- 17 min: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Linux%20ASan/19016

to clarify: when I said that I thought we didn't want to build the targets, I meant instead that we had talked about just running `gn gen` in the given configuration.

If the cycle time is < 30 min, we should consider just adding the builder to the CQ. But, if there have only been ~8 breakages in 2+ years (I don't know how exhaustive you thought that list was), it probably wouldn't justify the resources; instead, we should have a sheriffed builder on the main waterfall and an optional trybot instead, and revert the changes that do break things.

I'd generally want a CQ config to catch a breakage every couple of weeks in order to put it into the CQ.

There are build breakages every week, that query isn't complete since it is hard to search for keywords.

>> I meant instead that we had talked about just running `gn gen` in the given configuration.

Yes, I've made an attempt to do so in issue 845774, but it turned out to be not really applicable and not sufficient for catching most of the breakages.

Re frequency, +1 to what Abhishek said. I've just tried to make a query without noisy issues, so it's clearly a tiny subset.

Okay, I kinda figured it happened more often :).

Let's plan to add at least a Linux CQ builder for this. Ben can you do that? It should be easy enough.

mmoroz/inferno - is a single builder likely to catch most of the compile failures?

Oh, we should talk about moving all of those builders to the main waterfall as well, if that's possible. Otherwise people will be confused about why we have an FYI config in the CQ.

Since these things just upload binaries and don't run any tests, I assume that most of the breakages are just compile breakages?

> mmoroz/inferno - is a single builder likely to catch most of the compile failures?

Yes, libFuzzer ASan config should cover the vast majority of breakages.


> Since these things just upload binaries and don't run any tests, I assume that most of the breakages are just compile breakages?

Yes!

> Ben can you do that? It should be easy enough.

Yes, that's what I started trying to do in #45.

> I don't think we should try to add this to the generic chromium/chromium_trybot recipes; what they want to do with the fuzz targets is too different from what we normally do on the bots.

So then we'd have a create a new recipe that'll handle waterfall-to-trybot mirroring, patch applying, analyzing, without-patch retrying, etc. I doubt we'd want to forgo any of that, and I doubt that could be simply boiled down to a set of recipe_module calls. Hence why I think adding it to the chromium recipe would be easier. (And do we really want different recipes running on the CQ? There's already config skew happening w/ the ios bots, which we'd like to move over to the chromium recipe.)

That said, I'm not a chromium_tests owner, so if you disagree with any of that LMK and I can go down the long road of creating a new recipe.

Sorry, I should've mentioned that part. 

If all these things are doing is building stuff (e.g., 'all') (and uploading on the CI bots), then we should use the chromium/chromium_trybot recipe, and figure out how to best modify it to do a slightly different upload step than what 'package_build' does.

[ And, to be clear: we should only build a static set of targets, if need be, 'all' ].

It was the part with either dynamically trying to figure out the list of targets, or run MB/GN multiple times, that I didn't want to push into the chromium recipe.

Today's breakage, compilation failure: https://bugs.chromium.org/p/chromium/issues/detail?id=874440

Another example from the past week, compilation failure as well: https://bugs.chromium.org/p/chromium/issues/detail?id=871587

Noted.

My plan is to add a chromium_libfuzzer_trybot recipe that'll do what chromium_libfuzzer does + all the needed trybot pulleys and levers.

I'll get to work on that once I'm done w/ bug 869533, which should be sometime this week hopefully.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ed2aa6585416aadc492bc64713f27a3a3769139d

commit ed2aa6585416aadc492bc64713f27a3a3769139d
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Aug 17 21:27:56 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/79eeb786b64cac9af67dfb2d39d048610cdaf09e

commit 79eeb786b64cac9af67dfb2d39d048610cdaf09e
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Aug 17 23:41:15 2018

Add luci configs for linux-libfuzzer-asan-rel trybot.

Bug:  646404 
Change-Id: Ie047903220a329f1d7731eec04c22c26fcf69b88
Reviewed-on: https://chromium-review.googlesource.com/1180419
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584242}
[modify] https://crrev.com/79eeb786b64cac9af67dfb2d39d048610cdaf09e/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/79eeb786b64cac9af67dfb2d39d048610cdaf09e/infra/config/global/luci-milo.cfg

BTW, I'm working on the CQ bot and am seeing compile failures:
https://ci.chromium.org/swarming/task/3f862951862e1410?server=chromium-swarm.appspot.com

Turns out that that's actually not a false positive, and the fuzzers are broken again FYI :)
https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Linux%20ASan/20751

https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Linux%20ASan shows green now, can you check again

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3ce2b24c81f8b0ff416049113567a735d1065e99

commit 3ce2b24c81f8b0ff416049113567a735d1065e99
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Sep 06 01:37:50 2018

Add 'gn refs' and 'gn ls' methods to GN recipe_module.

The logic added here will be used to calculate a dynamic set of
compile targets for use in the fuzz trybot.

'refs' will be used to calculate all targets that depend on a
main fuzz target.

'ls' will be used to translate from GN labels to ninja targets (since
analyze takes labels but compile takes ninja targets... and we don't
want to fill out gn_isolate_map with all fuzz targets.)

Bug:  646404 
Change-Id: I0022e904d325657ab7b0a11d54d3f339eae67295
Reviewed-on: https://chromium-review.googlesource.com/1188630
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Garrett Beaty <gbeaty@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/3ce2b24c81f8b0ff416049113567a735d1065e99/scripts/slave/README.recipes.md
[modify] https://crrev.com/3ce2b24c81f8b0ff416049113567a735d1065e99/scripts/slave/recipe_modules/gn/__init__.py
[modify] https://crrev.com/3ce2b24c81f8b0ff416049113567a735d1065e99/scripts/slave/recipe_modules/gn/api.py
[add] https://crrev.com/3ce2b24c81f8b0ff416049113567a735d1065e99/scripts/slave/recipe_modules/gn/tests/refs.py
[add] https://crrev.com/3ce2b24c81f8b0ff416049113567a735d1065e99/scripts/slave/recipe_modules/gn/tests/ls.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/f370c189922db1db831979191b7f7e6ae3bbdd13

commit f370c189922db1db831979191b7f7e6ae3bbdd13
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Sep 07 23:36:12 2018

Add a chromium_libfuzzer_trybot recipe.

Like chromium_libfuzzer.py, but with trybot logic.

See https://ci.chromium.org/swarming/task/3fc953e2435d4d10?server=chromium-swarm.appspot.com
for an example run (on a patch that matches a filter exclusion).

chromium_libfuzzer.py could use some refactoring to be more inline with
this recipe, but I'll get to that after this lands.

Bug:  646404 
Change-Id: I8bca844e89e93f7e5395165346035fd6873f6d9a
Reviewed-on: https://chromium-review.googlesource.com/1182010
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/f370c189922db1db831979191b7f7e6ae3bbdd13/scripts/slave/README.recipes.md
[add] https://crrev.com/f370c189922db1db831979191b7f7e6ae3bbdd13/scripts/slave/recipes/chromium_libfuzzer_trybot.expected/basic_linux_tryjob.json
[add] https://crrev.com/f370c189922db1db831979191b7f7e6ae3bbdd13/scripts/slave/recipes/chromium_libfuzzer_trybot.py
[add] https://crrev.com/f370c189922db1db831979191b7f7e6ae3bbdd13/scripts/slave/recipes/chromium_libfuzzer_trybot.expected/basic_linux_tryjob_with_compile.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7d6ba28e92bec865a7b7876c35b4621d56fb37d8

commit 7d6ba28e92bec865a7b7876c35b4621d56fb37d8
Author: Ben Pastene <bpastene@chromium.org>
Date: Sat Sep 08 01:18:05 2018

Add mb_config entry for linux-libfuzzer-asan-rel trybot.

Bug:  646404 
Change-Id: I9c2b313fb869a21190a95ed96542c2197759ce20
Reviewed-on: https://chromium-review.googlesource.com/1214462
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#589745}
[modify] https://crrev.com/7d6ba28e92bec865a7b7876c35b4621d56fb37d8/tools/mb/mb_config.pyl

Apologies for the delay. There's a trybot up and running now at https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-libfuzzer-asan-rel

It should be fully functional, but it's only got a single machine in its pool atm. I'll work on scaling it up in the days to come. Then if everything looks good we can flip the 100%-mandatory-CQ-bot switch.

Worth noting that the waterfall bot's red once again https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Libfuzzer%20Upload%20Linux%20ASan
That should become much less common once we get the trybot on the CQ ;)

Thanks a ton Ben!

Max, can you verify the trybot, and confirm it looks good as per our requirements ?

Thanks for the heads up! Filed a bug for the breakage (issue 882570), checking the bot now.

Looks good to me!!! How does it get GN args? From the existing config or it is duplicated somewhere?

I'm wondering as I've noticed two extra arguments ("dcheck_always_on = true" and "symbol_level = 1"). Also, we may eventually change some args, will those get reflected on the trybot automatically?

> How does it get GN args? From the existing config or it is duplicated somewhere?

The gn args are configured here:
https://chromium.googlesource.com/chromium/src/+/a1f0a5bbe3467622fa75162c6e389b201a58cbd3/tools/mb/mb_config.pyl#1425

It's identical to the "Libfuzzer Upload Linux ASan" bot (and mostly uses the same config), except it uses "release_trybot" instead of "release_bot", which affects symbol_level and dchecks_always_on args like you noticed:
https://chromium.googlesource.com/chromium/src/+/a1f0a5bbe3467622fa75162c6e389b201a58cbd3/tools/mb/mb_config.pyl#2071

We can tweak/remove those two args if you'd like, but it's the setup we generally apply to every trybot, so prob best to keep it there otherwise.


> Also, we may eventually change some args, will those get reflected on the trybot automatically?

It probably won't. If you change the waterfall bot's config (https://chromium.googlesource.com/chromium/src/+/a1f0a5bbe3467622fa75162c6e389b201a58cbd3/tools/mb/mb_config.pyl#1421) just be sure to also change the trybot's (https://chromium.googlesource.com/chromium/src/+/a1f0a5bbe3467622fa75162c6e389b201a58cbd3/tools/mb/mb_config.pyl#1425)

They're right next to each other, so shouldn't be too much of a hassle ;)

Got it! Thanks for the info, Ben!

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/330bd7d6e144325cfde38e2af74daf2646e1a502

commit 330bd7d6e144325cfde38e2af74daf2646e1a502
Author: Ben Pastene <bpastene@chromium.org>
Date: Mon Sep 10 23:48:29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/efe769401edc3eb1141f62ddc27f59bd42f87c30

commit efe769401edc3eb1141f62ddc27f59bd42f87c30
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Sep 11 00:30:43 2018

Set linux-libfuzzer-asan-rel trybot at 10% CQ experiment.

Blocked on adding more bots to the pool: http://crrev.com/i/676080

Bug:  646404 
Change-Id: I46021d3173e430287afffde7b9da77bddb6c5487
Reviewed-on: https://chromium-review.googlesource.com/1217037
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590135}
[modify] https://crrev.com/efe769401edc3eb1141f62ddc27f59bd42f87c30/infra/config/branch/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/57df9546cfcbd3500fd5107113540ebb4918fa89

commit 57df9546cfcbd3500fd5107113540ebb4918fa89
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Sep 11 22:40:00 2018

Bump linux-libfuzzer-asan-rel's CQ experiment to 50%.

It appears to have enough bots. Will bump more if needed.

TBR=jbudorick@chromium.org

Bug:  646404 
Change-Id: I4046177c2adb2916ec293cc1961753577739e4c9
Reviewed-on: https://chromium-review.googlesource.com/1220703
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590510}
[modify] https://crrev.com/57df9546cfcbd3500fd5107113540ebb4918fa89/infra/config/branch/cq.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/b50678c63664da4f8e5ccbb4d3fa4701ebb13026

commit b50678c63664da4f8e5ccbb4d3fa4701ebb13026
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Sep 13 19:58:23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/829991703decdbb5ba77b95302eaf4f7ad3d5366

commit 829991703decdbb5ba77b95302eaf4f7ad3d5366
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Sep 13 21:16:08 2018

Bump linux-libfuzzer-asan-rel to 100% CQ experiment.

Should have enough bots to handle the load (barring any natural
disasters.)

Bug:  646404 
Change-Id: I53f49cfa4d7c9c67626578d7ef36b40cb2d82a53
Reviewed-on: https://chromium-review.googlesource.com/1225850
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591157}
[modify] https://crrev.com/829991703decdbb5ba77b95302eaf4f7ad3d5366/infra/config/branch/cq.cfg

Ben, how is the experiment going? I've just noticed the breakage what would've been caught by the bot (issue 885205), and so I'm wondering when we can enable the bot for real? :)

Everything looks good. Uploaded https://chromium-review.googlesource.com/c/chromium/src/+/1229076 earlier to turn it on for real. Will send that out now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/93dfedb453ae1eec02767ecf87c636d9d046bffb

commit 93dfedb453ae1eec02767ecf87c636d9d046bffb
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Sep 18 19:18:43 2018

Promote linux-libfuzzer-asan-rel trybot to full CQ blocker.

Cycle time looks good. (Caps out at about 30min for analyze exclusions.)
Flake rate looks like it's pretty much 0.

Though the bot is currently red due to a broken webrtc roll (that this
bot would have caught). Need to wait on reverting that before this can
land.

Bug:  646404 
Change-Id: I26ddec32ad1b90165068c0c17946c2ac6cdc330b
Reviewed-on: https://chromium-review.googlesource.com/1229076
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#592138}
[modify] https://crrev.com/93dfedb453ae1eec02767ecf87c636d9d046bffb/infra/config/branch/cq.cfg

After #80, the trybot's now blocking the CQ.

Just in time too, since it looks like its preventing the latest webrtc roll from breaking things:
https://chromium-review.googlesource.com/c/chromium/src/+/1232056/2

Awesome!!!!!!

SUPEEER AWESOME!!! YOU ROCK!!!!

Circling back, is there anything else that needs to be done here? The bot's been up and running at https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-libfuzzer-asan-rel for a bit now. Have there been any breakages that it should have caught but didn't? If so, LMK and I can look into why.

If not, I'd like to make one small refactor to the recipe (https://chromium-review.googlesource.com/c/chromium/tools/build/+/1266297), then close this out. We can talk about adding different fuzz platforms to the CQ (eg: win, mac) but nothing's quite as cheap and easy to get as linux bots, so it might be tough to scrape together the capacity. Hopefully the single linux bot here is sufficient.

Ben, I think the bot is doing a great job so far. It's really awesome to have it.

Speaking of other configs, IMO we would need a very similar config for https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Afl%20Upload%20Linux%20ASan, as it used to break separately from libFuzzer builds (it uses another instrumentation and fuzzing engine) and is very important on its own (+metzman@).

Also, metzman@ is enabling Windows configuration for libFuzzer these days, and so it would be very helpful to have a builder for that too.

Should we file separate bugs to discuss these more?

Btw, there was one breakage that was not caught (issue 892470), but it seems like it broke some other builders too, so I'm not sure how it was landed.

#85: yes, please file separate bugs for those.

Filed issue 893612 and issue 893614 for AFL on Linux and libFuzzer on Windows respectively.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3af823e79cd6cee8c427ef7b4260a98f54a17896

commit 3af823e79cd6cee8c427ef7b4260a98f54a17896
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 10 18:50:14 2018

Change chromium_libfuzzer.py to use the GN recipe module.

Bug:  646404 
Change-Id: I0a7690ea544934cadda0f3208fd42755c588df05
Reviewed-on: https://chromium-review.googlesource.com/c/1266297
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Windows_ASan.json
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_ASan_Debug.json
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_ASan.json
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/README.recipes.md
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/kitchen_paths.json
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.py
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_MSan.json
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Mac_ASan.json
[modify] https://crrev.com/3af823e79cd6cee8c427ef7b4260a98f54a17896/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_UBSan.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/537b85d99457e136016f3d7bc9a1834cb386983b

commit 537b85d99457e136016f3d7bc9a1834cb386983b
Author: Stephen Martinis <martiniss@chromium.org>
Date: Wed Oct 10 20:18:40 2018

Revert "Change chromium_libfuzzer.py to use the GN recipe module."

This reverts commit 3af823e79cd6cee8c427ef7b4260a98f54a17896.

Reason for revert: https://crbug.com/894183

Original change's description:
> Change chromium_libfuzzer.py to use the GN recipe module.
> 
> Bug:  646404 
> Change-Id: I0a7690ea544934cadda0f3208fd42755c588df05
> Reviewed-on: https://chromium-review.googlesource.com/c/1266297
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Commit-Queue: Ben Pastene <bpastene@chromium.org>

TBR=martiniss@chromium.org,bpastene@chromium.org,gbeaty@chromium.org

Change-Id: I9bf468ef289f59b27d7e9f2e6188a27e97d7807a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  646404 
Reviewed-on: https://chromium-review.googlesource.com/c/1274366
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Windows_ASan.json
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_ASan_Debug.json
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_ASan.json
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/README.recipes.md
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/kitchen_paths.json
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.py
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_MSan.json
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Mac_ASan.json
[modify] https://crrev.com/537b85d99457e136016f3d7bc9a1834cb386983b/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_UBSan.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/bd106103636bfc20b0fa978775e1a4e5656fc287

commit bd106103636bfc20b0fa978775e1a4e5656fc287
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 30 00:09:25 2018

Reland: "Change chromium_libfuzzer.py to use the GN recipe module."

Original CL:
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1266297

The fix:
Set the cwd as the src-checkout when using gn.py.

Without the fix:
https://ci.chromium.org/swarming/task/40dac11a8ceda310?server=chromium-swarm.appspot.com

With the fix:
https://ci.chromium.org/swarming/task/40dada3af1389510?server=chromium-swarm.appspot.com

Bug:  646404 
Change-Id: I17ba02def567202e92e652a9d9e54ed14908165b
Reviewed-on: https://chromium-review.googlesource.com/c/1306573
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Windows_ASan.json
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_ASan_Debug.json
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_ASan.json
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/README.recipes.md
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/kitchen_paths.json
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.py
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_MSan.json
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Mac_ASan.json
[modify] https://crrev.com/bd106103636bfc20b0fa978775e1a4e5656fc287/scripts/slave/recipes/chromium_libfuzzer.expected/full_chromium_fyi_Libfuzzer_Upload_Linux_UBSan.json

With the reland of the refactor in #91, I think we can finally close this out. There's now a couple fuzz bots on the CQ (linux and win asan). Hopefully they'll prove useful w/ the fuzz marathon ;)

Feel free to reopen if there's anything else that's needed.