We should replace the //testing/buildbot/*.json files with something else (maybe GN-based targets)? |
||||||||||||||||||||||||
Issue description
Currently //BUILD.gn has a bunch of targets that were defined for the GYP->GN migration.
Now that we're all-GN, the targets are just confusing, and we should clean things up.
In addition, I'd like to move the top-level to something closer to the following:
- one main target for each of the main "types" of builds, e.g.
- a target that builds the "main" binaries like chrome and content_shell
- a target that builds all of the tests needed on the "main" waterfalls
(chromium.{android,linux,mac,win}
- a target that builds the "official" targets (incl. tests) needed by
chromium.chrome and the internal official builders
- a target that builds the stuff needed for the perf builders
- a target that builds the stuff needed for the webkit builders
- a target that builds the stuff needed for clusterfuzz
- and so forth. This list should be fairly frozen, but I'm probably missing
a few.
- a target that builds "nearly everything" that depends on the above plus
includes other miscellaneous targets that we just want to keep building.
This would replace "All"/"gn_all".
I would abandon any hope of having "All" match "all"; given the way GN works, particularly with some of the templates like mojo that generate a bunch of targets,
it's too hard (or at least not worth the effort) to do this.
In particular, one goal for the "types" targets is to replace the lists of test targets in the //testing/buildbot/*.json files with something like `gn desc //out/Default //:chromium_builder_perf deps`, so that it's easier maintain (and figure out) what is run on each builder.
What do people think of these ideas?
,
Nov 4 2016
Removing targets sounds great.
I'm not sure about adding bot-flavored targets. We used to have those ("chromium_builder_tests"), and they never actually matched the tests running on the bots (unless you want to make those targets automatically grab the list of tests from the testing/buildbot json file? Ah, you mention this at the end. How'd you do the metadata like cipd version and whatnot?) What's the use case for this?
"...that we just want to keep building." <- In the sense of "not all targets should be required to actually compile"?
I don't know what "All" does in the gn world or if it's used anywhere.
It sounds like this bug covers several things:
- Removing old gyp compat cruft (sure, we should do this)
- Replacing buildbot json files (imho we should do this, but what to do exactly probably requires some design work)
I'd probably not mix "well, duh" changes and "possibly invasive" changes in the same bug.
,
Nov 4 2016
I don't see the value in artificially adding build targets for the tests that a given builder runs. The JSON files in src/testing/buildbot are a good way, in my opinion, of describing the fine-grained decisions of which tests to run on which machines. Since build targets already have to exist for those tests, grouping them further in the build files seems redundant. The ancient "chromium_gpu_builder" GYP targets were finally removed after the GN conversion ( Issue 299367 ) and I wouldn't like to see them revived.
,
Nov 4 2016
,
Nov 4 2016
thakis@chromium.org wrote in comment #2: > What's the use case for this? I want to replace as much of //testing/buildbot/*.json with more maintainable things as possible. I believe we need a more powerful language than JSON to describe what we want, and I believe that not only is GN about the right level of power but that the list of targets are most naturally maintained in the build files themselves. > How'd you do the metadata like cipd version and whatnot?) I'm not sure what you're thinking of here, but that stuff isn't in the //testing/buildbot/*.json files > "...that we just want to keep building." <- In the sense of "not all targets should be required to actually compile"? No, in the sense of "stuff that we want to make sure is compiled in the CQ even if it isn't a test that runs". We should *also* build "all", but we can't afford to build "all" in the CQ on Mac or Win. > I don't know what "All" does in the gn world or if it's used anywhere. There is still the magic "All" target for the xcode ide generator. Otherwise it is a synonym for gn_all, which is used to pull in all of the various targets we want to build explicitly. I would like to eliminate most of the stuff in gn_all and replace it just with the builder targets and maybe an additional "misc" if it is really needed.
,
Nov 4 2016
> but that stuff isn't in the //testing/buildbot/*.json files it is: https://cs.chromium.org/chromium/src/testing/buildbot/chromium.android.json?q=chromium.android.json&sq=package:chromium&dr&l=20
,
Nov 4 2016
Thanks for explaining. I'm on board with replacing the json files with something else (but maybe in a different bug, and with some design discussions), and the rest (removing old targets, All, gn_all) sounds good to me too.
,
Nov 5 2016
kbr@chromium wrote in n comment #3 that: > I don't see the value in artificially adding build targets for the tests that a given builder runs. I agree that we shouldn't both have build targets and stuff in the JSON files. I want to get rid of the JSON files by pushing the needed logic into the BUILD files if possible (since I don't think we need much here). > The JSON files in src/testing/buildbot are a good way, in my opinion, of describing the > fine-grained decisions of which tests to run on which machines. No, they're a bad way to do this, which is why you had to write a script that generated them for the GPU builders :). The only real complexity in the JSON files has to do with the swarming dimensions for the GPU builders (and, perhaps, to a lesser extent for the Perf builders). I'm not sure what to do about this since I don't know how to best encode the swarming dimensions in GN files, or if we should even do that. Maybe we need a separate config file just for those, or maybe we should have a separate GPU recipe for those. Or maybe the existing mechanism works okay for the GPU builders. But it's terrible for the non-GPU builders and we shouldn't force people to have to use the JSON files just for consistency in that regard. I would like to be in a world where we can use all of GN's power to figure out how to manage and configure things, rather than duplicating it in a poorer way in either static JSON or dynamic but uncontrolled and unqueryable Python.
,
Nov 5 2016
At any rate, I agree with thakis@ that I'm mixing up two things in the description, and also I suspect that no one will disagree that we should clean up the gyp left over stuff. So, let's just focus this bug on the more controversial goal of eliminating the //testing/buildbot/*.json files. The current status quo for them is not good enough to me. At the very least I'd like to see us move to something closer to what the ios builders use (in //ios/build/bots) but even that is probably not powerful enough to be that useful for desktop. (Note that I also want to push a lot of the logic currently in MB into the GN files. We have too many ways to configure things).
,
Nov 5 2016
thakis wrote in comment #6 that: > > but that stuff isn't in the //testing/buildbot/*.json files > it is: https://cs.chromium.org/chromium/src/testing/buildbot/chromium.android.json?q=chromium.android.json&sq=package:chromium&dr&l=20 Ah, okay, good point. I don't particularly like that we have things specified via both cipd and install-build-deps. I think ideally we'd have something that installed stuff as needed at runhooks time and then the GN data deps would propagate them into the isolates as needed.
,
Nov 5 2016
I agree that the src/testing/buildbot JSON files are difficult to maintain and would like to help move them in a better direction. It would be great if recipes lived in the Chromium workspace, because there are already descriptions of the waterfalls in tools/build/scripts/slave/recipe_modules/chromium_tests/ , and in some sense, the test lists should be contained in the waterfall descriptions. What if we did this iteratively? We could write a generator script like src/content/test/gpu/generate_buildbot_json.py for some of the other JSON files in src/testing/buildbot. Once we have a better description of the bots and the tests that run on them, it might be easier to understand how to group them, and then how to migrate away from the generator script. (A while back I looked at chromium.linux.json but couldn't visualize the commonality between the test machines, and therefore how to express it more tersely. I think a good plan would be to write a program to parse the existing JSON files and find which tests run on many bots, and are therefore eligible for some sort of grouping together. That could then help in creating the waterfall description and thus the generator script. The generator script should be verified by seeing whether it can produce the exact same JSON files as currently exist.)
,
Nov 5 2016
Let's get together and actually chat about this. I think there's a possibility for an basically-all-GN solution here that will be really good for everyone involved. I think we can improve this substantially before the recipes move over (in fact, I think the two projects should be viewed independently).
,
Nov 7 2016
+1 to a better way to denote what tests run on which builders. I think there are a lot of similarities in all of the existing json files in src/testing/buildbot where we could come up with a more concise and common way to generate this json or another way to describe the tests. I authored a script for the perf recipe, modeling after GPUs generate_buildbot_json.py in src/tools/perf/generate_perf_json.py. We have roughly 300 tests to generate entries for ~120 testers, so this will end up being well over 30K test entries in this json file once the entire waterfall is swarmed. This might be a separate issue, but these seem to be mostly boilerplate wrappers around mostly similar swarming dimensions and slightly different arguments. I would love to see if there was a less verbose way to describe these test entries in the buildbot json, but I realize that would take updates to the recipe code itself that reads the json. Either way I think we could author something where the user doesn't have to care about manipulating the json file itself (as you can see it would be very infeasible in the perf case). Please include me on threads and/or bugs related to this topic as it definitely impacts the perf swarming efforts.
,
Nov 7 2016
Things I would like to see in a replacement (this is brainstorm-level quality, not a hard list of requirements): - have some way to make sure "similar" bots run about the same number of tests. Currently, making gfx_unittests run on all Android bots is a 300+ line patch. - but still need exceptions for certain bots (eg run fewer tests on tsan bots, for cycle time reasons) - it'd be nice if you add a dep on some library in some config, the bot testing that config automatically starts running tests for that library. (This is probably overdesigned, but we used to use angle on android for a long time without running angle tests, and running them would've found a bug that was super tricky to debug without it.) It feels a bit like this is in the contradictory problem space where you want things to be very simple and repetitive so that machines can easily modify the data, but also normalized and factored so that humans can modify it :-/ The current system is in the former camp, our previous system (a single list of tests which are tweaked with conditionals; buildbot side) was in the latter camp. Maybe the ability for code to modify the test lists isn't so important here, and now that things are src-side we can move back to the other model. A good first step would be to talk to folks (smut for ios, gpu folks for gpu) and collect a list of requirements.
,
Nov 7 2016
Adding some colour to the shed; - There's going to be more Swarming metadata than just the dimensions. As a concrete example, I'm planning to add timeouts per config/step pairs. There's currently no way to describe this in GN. To add colour to what thakis@ stated in comment #14, I'd like this to be "machine writeable" up to an extent, that the timeouts may be updated in an automated manner by looking at the P(99.99) value via a script. -> The Swarming specific metadata may have to be open ended for new metadata. It is de-facto in JSON but not in GN. - Specifying the command line would help transition off mb. That brings back to the "test" target idea which in part defines the command line to use. -> One reason I bring this up is that I want to move us from specifying the command line in the isolated file to the "swarming trigger" invocation. I can explain offline why they may be forced upon us sooner than expected. - This happens to be a step towards "ninja run_test_remotely" so even if it's clearly not a goal here, I like the idea to be moving test description into GN. :D But let's not scope-creep here.
,
Nov 7 2016
I'm not sure how important "machine-writable" is for this stuff. I can see the desire specifically for maruel's timeout example, and there's an obvious desire for being able to suppress test failures in a machine-writable way, but I'm not sure how many other cases there really are. It's not clear to me what the best solution(s) are here. I think the command line should mostly be contained in GN or things below GN like they effectively are today on android and ios. I think we need a way to have the trigger mechanism be able to add on additional flags for user convenience, but I don't think most (non-swarming) flags should be specified that way. I don't know what the best way to manage all of the different swarming dimensions or perf dimensions are, but putting that in GN feels somewhere between a real stretch and flat-out the wrong place, and so that's a tough one. I also want to really minimize the differences between running a test locally in-checkout (not isolated), running a test locally-isolated, and running a test on swarming. Ideally it'd be the same command with slightly different flags. I agree w/ maruel@ that this is close to "ninja run_test_remotely" and I also agree that we shouldn't scope-creep too much :).
,
Nov 14 2016
See also bug 665183 for how MB figures into this.
,
Jan 6 2017
I'm going to punt this to Paweł to own it now :).
,
Jan 18 2017
,
Jan 19 2017
,
Jan 19 2017
,
Jan 19 2017
Per the bug that was just merged into this one: I think a good start would be to first generate all of the src/testing/buildbot/*.json files from a script. src/content/test/gpu/generate_buildbot_json.py might be a reasonable starting point. Once the files are all autogenerated it should be easy to change them to something else. A first step might be to write a script which reads in all the JSON files and tries to find commonalities between various builders and testers. I found a lot of exceptions about which tests run on which machines. This would lead to a lot of verbosity in a description of the waterfalls. See Issue 636179 for more details.
,
Feb 10 2017
,
Feb 28 2017
,
Mar 15 2017
One problem with not having autogeneration of these JSON files is that someone can update one JSON, but miss others. This recently has caused android_optional_gpu_tests_rel (a CQ bot for some projects) to get broken. If we had these JSONs autogenerated, then we could have had another script which is responsible for updating stuff common to GPU.FYI bots (whose JSON is already autogenerated) and other Android bots, like cipd_packages. This would have prevented this breakage.
,
Mar 15 2017
,
Apr 26 2017
,
Jun 26 2017
,
Jun 30 2017
Chime in this late, but I wonder whether we can borrow some of the designs of gcl?
,
Jun 30 2017
You are not the first person to suggest that. I'm not very familiar w/ GCL but I will take a look at it before doing anything much.
,
Jun 30 2017
There is also go/rethinking-config (Googler only, sorry), which hopefully is relevant.
,
Jul 11 2017
,
Jul 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88 commit 4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88 Author: Dirk Pranke <dpranke@chromium.org> Date: Wed Jul 12 04:18:39 2017 Clean up //testing/buildbot/*.json files to remove obsolete builders. Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I0b58757a8c56cf1ae9a5ba82fe2d485651f3c52e Bug: 662541 Reviewed-on: https://chromium-review.googlesource.com/558381 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#485851} [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/content/test/gpu/generate_buildbot_json.py [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/chromium.full.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.mac.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/chromium.memory.fyi.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.perf.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/chromium.win.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/chromium_memory_trybot.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/chromium_win8_trybot.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/gn_isolate_map.pyl [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/testing/buildbot/manage.py [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/tryserver.blink.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/tryserver.chromium.linux.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/tryserver.chromium.mac.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/tryserver.chromium.perf.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/tryserver.chromium.win.json [delete] https://crrev.com/ea94040c26822246f57e699988b5a160fc6a3742/testing/buildbot/tryserver.v8.json [modify] https://crrev.com/4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88/tools/perf/core/perf_data_generator.py
,
Jul 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/377951a272696effa274c2786a00d18e45c497b6 commit 377951a272696effa274c2786a00d18e45c497b6 Author: Michael Achenbach <machenbach@chromium.org> Date: Wed Jul 12 09:15:33 2017 Revert "Clean up //testing/buildbot/*.json files to remove obsolete builders." This reverts commit 4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88. Reason for revert: V8 bot was not obsolete (obsolete name maybe). Why was it deleted? It is part of the V8 CQ. https://crbug.com/741505 Original change's description: > Clean up //testing/buildbot/*.json files to remove obsolete builders. > > Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel > Change-Id: I0b58757a8c56cf1ae9a5ba82fe2d485651f3c52e > Bug: 662541 > Reviewed-on: https://chromium-review.googlesource.com/558381 > Commit-Queue: Dirk Pranke <dpranke@chromium.org> > Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org> > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > Cr-Commit-Position: refs/heads/master@{#485851} TBR=phajdan.jr@chromium.org,dpranke@chromium.org,kbr@chromium.org,martiniss@chromium.org Change-Id: I056aa71fcae296adc08aa4baf7f2f465d2e1511c No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 662541 ,741505 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Reviewed-on: https://chromium-review.googlesource.com/566961 Reviewed-by: Michael Achenbach <machenbach@chromium.org> Commit-Queue: Michael Achenbach <machenbach@chromium.org> Cr-Commit-Position: refs/heads/master@{#485902} [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/content/test/gpu/generate_buildbot_json.py [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.full.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.mac.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.memory.fyi.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.perf.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium.win.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium_memory_trybot.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/chromium_win8_trybot.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/gn_isolate_map.pyl [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/manage.py [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/tryserver.blink.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/tryserver.chromium.linux.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/tryserver.chromium.mac.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/tryserver.chromium.perf.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/tryserver.chromium.win.json [add] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/testing/buildbot/tryserver.v8.json [modify] https://crrev.com/377951a272696effa274c2786a00d18e45c497b6/tools/perf/core/perf_data_generator.py
,
Jul 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/124475bd4b63567946a4d606b753954a08231c28 commit 124475bd4b63567946a4d606b753954a08231c28 Author: Dirk Pranke <dpranke@chromium.org> Date: Fri Jul 14 00:57:09 2017 Re-land "Clean up //testing/buildbot/*.json files to remove obsolete builders." This reverts commit 377951a272696effa274c2786a00d18e45c497b6 and re-lands 4053f7ad29076abc4bd3eb267ba8d6fd0b52fa88, after fixing the issues that led it to be reverted :). R=machenbach@chromium.org TBR=kbr@chromium.org, martiniss@chromium.org BUG= 662541 Change-Id: I7cfb2837736d7ca86c2cbda5cee109c6797e59bb Reviewed-on: https://chromium-review.googlesource.com/569444 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Michael Achenbach <machenbach@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#486560} [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/content/test/gpu/generate_buildbot_json.py [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/chromium.full.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.mac.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/chromium.memory.fyi.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.perf.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/chromium.win.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/chromium_memory_trybot.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/chromium_win8_trybot.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/gn_isolate_map.pyl [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/testing/buildbot/manage.py [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/tryserver.blink.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/tryserver.chromium.linux.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/tryserver.chromium.mac.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/tryserver.chromium.perf.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/tryserver.chromium.win.json [delete] https://crrev.com/928d360b185de165acb06cddfdba459f0100a666/testing/buildbot/tryserver.v8.json [modify] https://crrev.com/124475bd4b63567946a4d606b753954a08231c28/tools/perf/core/perf_data_generator.py
,
Oct 4 2017
I'm not actively working on this, and we have no ETA at the moment ...
,
Oct 4 2017
,
Oct 4 2017
I'd like to once again advocate for coming up with some sort of waterfall and test group description, and auto-generating the existing JSON files from it. https://cs.chromium.org/chromium/src/content/test/gpu/generate_buildbot_json.py contains such a waterfall description. It could be done incrementally and in a way that the JSON files would be guaranteed unmodified during the cutover. Once the waterfall description is done, it would be easy to massage it into a new form.
,
Oct 4 2017
kbr@ - you're advocating the approach I more-or-less described in the go/coattail doc, so I don't think you need to do any more advocating (at least, not to me). What we need is for someone to have cycles to actually work on it :(.
,
Nov 24 2017
I've started work on this. The first goal will be to autogenerate the existing JSON files from a description of the waterfalls and test suites. Once that is done, it should be straightforward to do something radically different with those descriptions. So far I've got chromium.win.json autogenerating as-is from a waterfall description. Once more of the waterfalls are handled I'll send something out for review.
,
Nov 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3c84840b7173be0296b463fd7c2942660412798a commit 3c84840b7173be0296b463fd7c2942660412798a Author: Kenneth Russell <kbr@chromium.org> Date: Sun Nov 26 23:11:08 2017 Alphabetize webkit_layout_tests in chromium.mac.json. This will make it slightly easier to autogenerate this file with zero modifications from the original. BUG= 662541 TBR=dpranke@chromium.org NOTRY=true Change-Id: Ib3b7b6865bf8dd05c76414b9c8050936dca4c236 Reviewed-on: https://chromium-review.googlesource.com/789808 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#519199} [modify] https://crrev.com/3c84840b7173be0296b463fd7c2942660412798a/testing/buildbot/chromium.mac.json
,
Nov 27 2017
Adding ashley since we are actively changing our perf json. Good to be in the loop if the standard is changing.
,
Nov 28 2017
Sounds good. My intent is to autogenerate the existing JSON files as-is. To start I'm working on the waterfalls whose JSON isn't auto-generated -- so, not touching the Perf or GPU waterfalls. Once the initial script is checked in, we can all collaborate on it to hopefully subsume functionality in the Perf and GPU generation scripts, and get one unified description of the waterfalls and the tests they run.
,
Nov 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/90107cc3ef8c450104e70314be53ed810e501579 commit 90107cc3ef8c450104e70314be53ed810e501579 Author: Kenneth Russell <kbr@chromium.org> Date: Tue Nov 28 17:54:46 2017 Alphabetize and fix a couple of entries in chromium.linux.json. These fixes will make it easier to autogenerate this file with full fidelity. BUG= 662541 TBR=dpranke@chromium.org NOTRY=true Change-Id: Iab5fc98e13b9ce4c4b61d29fa72d08bc06b3c8b1 Reviewed-on: https://chromium-review.googlesource.com/790131 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#519738} [modify] https://crrev.com/90107cc3ef8c450104e70314be53ed810e501579/testing/buildbot/chromium.linux.json
,
Nov 29 2017
sgtm thanks!
,
Dec 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bdccc14a98ac7793a92e17254896061b9df28754 commit bdccc14a98ac7793a92e17254896061b9df28754 Author: Kenneth Russell <kbr@chromium.org> Date: Sat Dec 02 01:13:56 2017 Alphabetize more entries in chromium.linux.json and chromium.android.json. This will make it possible to autogenerate these files with 100% fidelity. Stop skipping the associated testers on chromium.android.json in the manage.py script. This was the reason the gtests got out of order in the first place. BUG= 662541 R=dpranke@chromium.org NOTRY=true Change-Id: I2efc4ed6e7f7564737709d5eda607f5d4543a2a8 Reviewed-on: https://chromium-review.googlesource.com/804748 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#521164} [modify] https://crrev.com/bdccc14a98ac7793a92e17254896061b9df28754/testing/buildbot/chromium.android.json [modify] https://crrev.com/bdccc14a98ac7793a92e17254896061b9df28754/testing/buildbot/chromium.linux.json [modify] https://crrev.com/bdccc14a98ac7793a92e17254896061b9df28754/testing/buildbot/manage.py
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/eb60cbd2c389c97ca8809138e226c09eed75d4b8 commit eb60cbd2c389c97ca8809138e226c09eed75d4b8 Author: Kenneth Russell <kbr@chromium.org> Date: Tue Dec 05 07:54:28 2017 Autogenerate chromium.{android,linux,mac,windows}.json. Add a new generation script, and a description of these waterfalls and the test suites that run on them. This CL subsumes the maintenance of 21,000+ lines of hand-written JSON. Many simplifications are now possible. The initial goal here was to replicate the existing JSON files verbatim, in order to guarantee that the behavior of the waterfalls would be unmodified. Many exceptions have been identified with various test suites and bots that can now be easily removed. The new script is unit tested and has 100% code coverage modulo a few exceptions. New presubmit checks will enforce this rigor going forward. Follow-on work will autogenerate the remainder of Chromium's waterfalls and unify with the GPU bots' generation script. BUG= 662541 NOTRY=true Change-Id: I9a916d6f341bbd1495f0a61761628d7f4fb227ff Reviewed-on: https://chromium-review.googlesource.com/789793 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#521634} [modify] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/PRESUBMIT.py [modify] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/chromium.android.json [modify] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/chromium.linux.json [modify] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/chromium.mac.json [modify] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/chromium.win.json [add] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/generate_buildbot_json.py [add] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/generate_buildbot_json_coveragetest.py [add] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/generate_buildbot_json_unittest.py [add] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/test_suite_exceptions.pyl [add] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/test_suites.pyl [add] https://crrev.com/eb60cbd2c389c97ca8809138e226c09eed75d4b8/testing/buildbot/waterfalls.pyl
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/139f8641d3b7a3959f7bcaf03f9fdd0760789bf6 commit 139f8641d3b7a3959f7bcaf03f9fdd0760789bf6 Author: Kenneth Russell <kbr@chromium.org> Date: Tue Dec 05 08:51:43 2017 Remove artificial builder/tester split from generation script. Everything's now just a "machine". Absence of the test_suites key is no longer an error. Fixed two builders which were actually running the check_network_annotations.py script. BUG= 662541 TBR=dpranke@chromium.org NOTRY=true Change-Id: I7c28088676974cdda402cf020d097e69614ecf12 Reviewed-on: https://chromium-review.googlesource.com/808084 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#521643} [modify] https://crrev.com/139f8641d3b7a3959f7bcaf03f9fdd0760789bf6/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/139f8641d3b7a3959f7bcaf03f9fdd0760789bf6/testing/buildbot/generate_buildbot_json_unittest.py [modify] https://crrev.com/139f8641d3b7a3959f7bcaf03f9fdd0760789bf6/testing/buildbot/test_suites.pyl [modify] https://crrev.com/139f8641d3b7a3959f7bcaf03f9fdd0760789bf6/testing/buildbot/waterfalls.pyl
,
Dec 5 2017
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b3a0d30f53d462da8d35e66a720a6e34b64389f2 commit b3a0d30f53d462da8d35e66a720a6e34b64389f2 Author: Dirk Pranke <dpranke@chromium.org> Date: Tue Dec 05 19:10:13 2017 Disable coverage tests in //testing/buildbot if coverage is not installed. At the moment we can't assume that the Python `coverage` module is installed everywhere we might want to run the presubmit tests in //testing/buildbot. The right fix for this is to add coverage to the .vpython spec, but in the meantime this disables the check if it would fail. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: If71bfe7774788776865c8f1f9ecf38aafaddc308 Reviewed-on: https://chromium-review.googlesource.com/809464 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#521772} [modify] https://crrev.com/b3a0d30f53d462da8d35e66a720a6e34b64389f2/testing/buildbot/PRESUBMIT.py
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a1cbe21a735b9869f89ea5d7ee8ba00b60055d77 commit a1cbe21a735b9869f89ea5d7ee8ba00b60055d77 Author: Dirk Pranke <dpranke@chromium.org> Date: Tue Dec 05 21:45:04 2017 Lock down //testing/buildbot/*.json files temporarily. We are busily converting over the test configs for the waterfalls to the new `generate_buildbot_json` tooling; until everything is converted, and we have docs written for how to use this, we want only a very small number of people making changes to these files. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I45432632c914a721fc2704e1b57a02399c624a2f Reviewed-on: https://chromium-review.googlesource.com/809684 Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#521842} [add] https://crrev.com/a1cbe21a735b9869f89ea5d7ee8ba00b60055d77/testing/buildbot/GENERATION_OWNERS [modify] https://crrev.com/a1cbe21a735b9869f89ea5d7ee8ba00b60055d77/testing/buildbot/OWNERS
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b3cca6f0dd0c9550f10afeb1cdbe9105849f6e7e commit b3cca6f0dd0c9550f10afeb1cdbe9105849f6e7e Author: Kenneth Russell <kbr@chromium.org> Date: Tue Dec 05 22:02:39 2017 Remove nonexistent Android bots from chromium.linux waterfall. These machines were moved to the chromium.android waterfall a while back but their JSON entries were forgotten. Remove the definitions of these bots and the test suite exceptions referring to them. (The presubmit checks caught all of the stray exceptions.) Add cast_junit_test_lists to Cast Android (dbg) on chromium.android, which is presuambly where it should have been placed during this migration. BUG= 662541 Change-Id: Iaa0453b17e06e31096ecf4e770557ad6aa7d7fd6 Reviewed-on: https://chromium-review.googlesource.com/809633 Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#521847} [modify] https://crrev.com/b3cca6f0dd0c9550f10afeb1cdbe9105849f6e7e/testing/buildbot/chromium.android.json [modify] https://crrev.com/b3cca6f0dd0c9550f10afeb1cdbe9105849f6e7e/testing/buildbot/chromium.linux.json [modify] https://crrev.com/b3cca6f0dd0c9550f10afeb1cdbe9105849f6e7e/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/b3cca6f0dd0c9550f10afeb1cdbe9105849f6e7e/testing/buildbot/waterfalls.pyl
,
Dec 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a commit 1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a Author: Dirk Pranke <dpranke@chromium.org> Date: Thu Dec 07 04:44:23 2017 Auto-generate //testing/buildbot/chromium.json. This CL converts the chromium waterfall config to be auto-generated using the new generate_buildbot_json framework. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I56c403369b4e0811fe604930d8da97048d434eda Reviewed-on: https://chromium-review.googlesource.com/809958 Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#522339} [modify] https://crrev.com/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a/testing/buildbot/chromium.json [modify] https://crrev.com/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a/testing/buildbot/generate_buildbot_json_unittest.py [modify] https://crrev.com/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a/testing/buildbot/test_suites.pyl [modify] https://crrev.com/1b7670959ab3c8eb7e49e3fdb9c3d0e5dda81f7a/testing/buildbot/waterfalls.pyl
,
Dec 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ad33ed3adb4ffc05903c1d4095ff264579583437 commit ad33ed3adb4ffc05903c1d4095ff264579583437 Author: Ned Nguyen <nednguyen@google.com> Date: Thu Dec 07 19:55:21 2017 Disable parallelism on telemetry_unittests everywhere Bug: 753495, 662541 Change-Id: I5c819540d9a34a0b9e1e679a16cd4059f097ceee Reviewed-on: https://chromium-review.googlesource.com/814935 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#522516} [modify] https://crrev.com/ad33ed3adb4ffc05903c1d4095ff264579583437/testing/buildbot/chromium.mac.json [modify] https://crrev.com/ad33ed3adb4ffc05903c1d4095ff264579583437/testing/buildbot/chromium.win.json [modify] https://crrev.com/ad33ed3adb4ffc05903c1d4095ff264579583437/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/ad33ed3adb4ffc05903c1d4095ff264579583437/testing/buildbot/test_suites.pyl
,
Dec 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/56a3e86a239a7de8d54ae386201d76dcc2e2bc9f commit 56a3e86a239a7de8d54ae386201d76dcc2e2bc9f Author: Kenneth Russell <kbr@chromium.org> Date: Fri Dec 08 03:11:34 2017 Auto-generate //testing/buildbot/chromium.chromiumos.json. This converts chromium.chromiumos.json to use the new generate_buildbot_json tool. R=jbudorick@chromium.org, kbr@chromium.org BUG= 662541 Change-Id: I5f4dce65ad962a53678e7868b4e78863d3b4d26b Reviewed-on: https://chromium-review.googlesource.com/810267 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#522694} [modify] https://crrev.com/56a3e86a239a7de8d54ae386201d76dcc2e2bc9f/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/56a3e86a239a7de8d54ae386201d76dcc2e2bc9f/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/56a3e86a239a7de8d54ae386201d76dcc2e2bc9f/testing/buildbot/test_suites.pyl [modify] https://crrev.com/56a3e86a239a7de8d54ae386201d76dcc2e2bc9f/testing/buildbot/waterfalls.pyl
,
Dec 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e3f024683620742d281fcfd402588cf798560305 commit e3f024683620742d281fcfd402588cf798560305 Author: John Budorick <jbudorick@chromium.org> Date: Fri Dec 08 03:32:02 2017 Auto-generate //testing/buildbot/chromium.lkgr.json. Bug: 662541 Change-Id: I2d29a8a863abdad044025d587aa2f9fec6927b71 Reviewed-on: https://chromium-review.googlesource.com/815943 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#522700} [modify] https://crrev.com/e3f024683620742d281fcfd402588cf798560305/testing/buildbot/chromium.lkgr.json [modify] https://crrev.com/e3f024683620742d281fcfd402588cf798560305/testing/buildbot/waterfalls.pyl
,
Dec 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/35daa69c631f847e64e4066a8309357a5db195d0 commit 35daa69c631f847e64e4066a8309357a5db195d0 Author: Dirk Pranke <dpranke@chromium.org> Date: Fri Dec 08 07:08:37 2017 Auto-generate //testing/buildbot/chromium.chrome.json. This CL switches the chromium.chrome configuration to be auto-generated using the new generate_buildbot_json tool. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: Icfdb366e408b67c7f488d45bfa977272f011b95f Reviewed-on: https://chromium-review.googlesource.com/809961 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#522730} [modify] https://crrev.com/35daa69c631f847e64e4066a8309357a5db195d0/testing/buildbot/chromium.chrome.json [modify] https://crrev.com/35daa69c631f847e64e4066a8309357a5db195d0/testing/buildbot/waterfalls.pyl
,
Dec 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dd1d2a026dfc81ea420b47226cabc715237b3ead commit dd1d2a026dfc81ea420b47226cabc715237b3ead Author: Dirk Pranke <dpranke@chromium.org> Date: Fri Dec 08 20:51:59 2017 Auto-generate //testing/buildbot/chromium.sandbox.json. This converts the chromium.sandbox JSON file to be auto-generated from generate_buildbot_json.py. R=jbudorick@chromium.org, kbr@chromium.org BUG= 662541 Change-Id: I0bd822ee2cefd7c633981aa0c5c6d1d783510f71 Reviewed-on: https://chromium-review.googlesource.com/809392 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#522868} [modify] https://crrev.com/dd1d2a026dfc81ea420b47226cabc715237b3ead/testing/buildbot/chromium.sandbox.json [modify] https://crrev.com/dd1d2a026dfc81ea420b47226cabc715237b3ead/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/dd1d2a026dfc81ea420b47226cabc715237b3ead/testing/buildbot/waterfalls.pyl
,
Dec 8 2017
,
Dec 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fbda3c53c3e7b0bf7abae3cd732838ff0115dee5 commit fbda3c53c3e7b0bf7abae3cd732838ff0115dee5 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Dec 08 23:57:24 2017 Autogenerate chromium.clang.json. A couple of tiny diffs remain, compared to the original JSON file. These were incorrect in their original state, so are being left as is. This waterfall was difficult to autogenerate, as there were many seemingly arbitrary exclusions of certain tests from entire groups of machines. In order to avoid adding large copy-pastes of exclusion lists to the exceptions, many one-off test suites were created that contain only small numbers of tests. This will increase the maintenance burden for test_suites.pyl. The plan is to merge these test suites back together immediately after landing this CL, thereby running more tests on the Clang waterfall, and unifying machines' configurations. BUG= 662541 NOTRY=true Change-Id: I9c4efdbb919273124acf257c8722740fdf22a48d Reviewed-on: https://chromium-review.googlesource.com/818075 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#522926} [modify] https://crrev.com/fbda3c53c3e7b0bf7abae3cd732838ff0115dee5/testing/buildbot/chromium.clang.json [modify] https://crrev.com/fbda3c53c3e7b0bf7abae3cd732838ff0115dee5/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/fbda3c53c3e7b0bf7abae3cd732838ff0115dee5/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/fbda3c53c3e7b0bf7abae3cd732838ff0115dee5/testing/buildbot/test_suites.pyl [modify] https://crrev.com/fbda3c53c3e7b0bf7abae3cd732838ff0115dee5/testing/buildbot/waterfalls.pyl
,
Dec 9 2017
,
Dec 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ca14c76f6831b2e5597bdc6370b49001521ca7ca commit ca14c76f6831b2e5597bdc6370b49001521ca7ca Author: John Budorick <jbudorick@chromium.org> Date: Sat Dec 09 14:19:18 2017 Auto-generate //testing/buildbot/chromium.memory.json. Bug: 662541 Change-Id: I282729691d2f2e81b0ba5e3d0212ef6aec88f775 Reviewed-on: https://chromium-review.googlesource.com/818568 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#522989} [modify] https://crrev.com/ca14c76f6831b2e5597bdc6370b49001521ca7ca/testing/buildbot/chromium.memory.json [modify] https://crrev.com/ca14c76f6831b2e5597bdc6370b49001521ca7ca/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/ca14c76f6831b2e5597bdc6370b49001521ca7ca/testing/buildbot/test_suites.pyl [modify] https://crrev.com/ca14c76f6831b2e5597bdc6370b49001521ca7ca/testing/buildbot/waterfalls.pyl
,
Dec 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/20eae9a7b5195d2e7ef40f877774a5e40e61f75a commit 20eae9a7b5195d2e7ef40f877774a5e40e61f75a Author: Dirk Pranke <dpranke@chromium.org> Date: Sat Dec 09 18:12:14 2017 Convert chromium.swarm to generate_buildbot_json. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I6ae85069dbc6ca615dcf1cde6e19946ed4bf781d Reviewed-on: https://chromium-review.googlesource.com/818490 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523005} [modify] https://crrev.com/20eae9a7b5195d2e7ef40f877774a5e40e61f75a/testing/buildbot/chromium.swarm.json [modify] https://crrev.com/20eae9a7b5195d2e7ef40f877774a5e40e61f75a/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/20eae9a7b5195d2e7ef40f877774a5e40e61f75a/testing/buildbot/test_suites.pyl [modify] https://crrev.com/20eae9a7b5195d2e7ef40f877774a5e40e61f75a/testing/buildbot/waterfalls.pyl
,
Dec 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/81ff51c5b4d8784e1a6e890670b379166a2154cd commit 81ff51c5b4d8784e1a6e890670b379166a2154cd Author: Dirk Pranke <dpranke@chromium.org> Date: Sat Dec 09 19:24:28 2017 Convert chromium.webkit to generate_buildbot_json. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I7b73a552f6f3d308341cfca8a6ae5c88b8114065 Reviewed-on: https://chromium-review.googlesource.com/818473 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523008} [modify] https://crrev.com/81ff51c5b4d8784e1a6e890670b379166a2154cd/testing/buildbot/chromium.webkit.json [modify] https://crrev.com/81ff51c5b4d8784e1a6e890670b379166a2154cd/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/81ff51c5b4d8784e1a6e890670b379166a2154cd/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/81ff51c5b4d8784e1a6e890670b379166a2154cd/testing/buildbot/test_suites.pyl [modify] https://crrev.com/81ff51c5b4d8784e1a6e890670b379166a2154cd/testing/buildbot/waterfalls.pyl
,
Dec 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5767e45410af9483a213bfb0a15435bc946414f4 commit 5767e45410af9483a213bfb0a15435bc946414f4 Author: Dirk Pranke <dpranke@chromium.org> Date: Sat Dec 09 19:34:38 2017 Convert .angle, .webrtc, .webrtc.fyi, and .v8.branches to generators. This converts the chromium.angle, chromium.webrtc, chromium.webrtc.fyi, and client.v8.branches JSON files to be generated by generate_buildbot_json.py R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I4c1f9f0938b6461264d979625bdd7b69cb8641e8 Reviewed-on: https://chromium-review.googlesource.com/818625 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523009} [modify] https://crrev.com/5767e45410af9483a213bfb0a15435bc946414f4/testing/buildbot/chromium.angle.json [modify] https://crrev.com/5767e45410af9483a213bfb0a15435bc946414f4/testing/buildbot/chromium.webrtc.fyi.json [modify] https://crrev.com/5767e45410af9483a213bfb0a15435bc946414f4/testing/buildbot/chromium.webrtc.json [modify] https://crrev.com/5767e45410af9483a213bfb0a15435bc946414f4/testing/buildbot/client.v8.branches.json [modify] https://crrev.com/5767e45410af9483a213bfb0a15435bc946414f4/testing/buildbot/waterfalls.pyl
,
Dec 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2e2e951436f15c39b802972f8f1c4ec01b38d56a commit 2e2e951436f15c39b802972f8f1c4ec01b38d56a Author: Dirk Pranke <dpranke@chromium.org> Date: Sat Dec 09 19:43:45 2017 Convert chromium.goma to generate_buildbot_json. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I152e12b9b8d92a7fef53557b44cae22ac8b092e6 Reviewed-on: https://chromium-review.googlesource.com/818399 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523010} [modify] https://crrev.com/2e2e951436f15c39b802972f8f1c4ec01b38d56a/testing/buildbot/chromium.goma.json [modify] https://crrev.com/2e2e951436f15c39b802972f8f1c4ec01b38d56a/testing/buildbot/test_suites.pyl [modify] https://crrev.com/2e2e951436f15c39b802972f8f1c4ec01b38d56a/testing/buildbot/waterfalls.pyl
,
Dec 9 2017
,
Dec 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8fd63c1df9fc32c8f5d57f4af15de0331bde7004 commit 8fd63c1df9fc32c8f5d57f4af15de0331bde7004 Author: Dirk Pranke <dpranke@chromium.org> Date: Sun Dec 10 18:06:27 2017 Convert client.v8.chromium to generate_buildbot_json. R=kbr@chromium.org, jbudorick@chromium.org BUG= 662541 Change-Id: I4c308056fcb7a411edf6d6b5e50ac29f49fdc5a8 Reviewed-on: https://chromium-review.googlesource.com/818725 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523025} [modify] https://crrev.com/8fd63c1df9fc32c8f5d57f4af15de0331bde7004/testing/buildbot/client.v8.chromium.json [modify] https://crrev.com/8fd63c1df9fc32c8f5d57f4af15de0331bde7004/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/8fd63c1df9fc32c8f5d57f4af15de0331bde7004/testing/buildbot/test_suites.pyl [modify] https://crrev.com/8fd63c1df9fc32c8f5d57f4af15de0331bde7004/testing/buildbot/waterfalls.pyl
,
Dec 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/25ba3b23d4d4aa33c724335b05c8feae0a16703e commit 25ba3b23d4d4aa33c724335b05c8feae0a16703e Author: Kenneth Russell <kbr@chromium.org> Date: Sun Dec 10 22:37:16 2017 Reorganize chromium.fyi for autogeneration. The following changes were made: - On "Android VR Tests": - Alphabetize step names by name, when test was identical. - Add a missing "merge" script from one step. - On a few bots, remove useless swarming parameters when "can_use_on_swarming_builders" was false. - Made telemetry_unittests run serially, as on other waterfalls. - Alphabetized step names on "Site Isolation Android". With these changes, this waterfall's JSON can be autogenerated with 100% fidelity. BUG= 662541 TBR=dpranke@chromium.org, bpastene@chromium.org, jbudorick@chromium.org Change-Id: I9df67955cfe1ac61338c5cfef290e7231fc6dd0f Reviewed-on: https://chromium-review.googlesource.com/818741 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523031} [modify] https://crrev.com/25ba3b23d4d4aa33c724335b05c8feae0a16703e/testing/buildbot/chromium.fyi.json
,
Dec 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/04f944830e7394ceb1b14858db12debf15bda6ec commit 04f944830e7394ceb1b14858db12debf15bda6ec Author: Kenneth Russell <kbr@chromium.org> Date: Mon Dec 11 08:37:13 2017 Revise chromium.android.fyi.json for autogeneration. The following changes are made: - For many non-Swarmed tasks: - Remove merge script, as this is only used with Swarming. - Add explicit: "can_use_on_swarming_builders": false - Add output_links to chrome_public_test_apk on Oreo Phone Tester. - Remove redundant "shards: 1" from telemetry_perf_unittests on x86 Cloud Tester. These are all no-ops. With them, this file can be autogenerated with 100% fidelity. BUG= 662541 TBR=dpranke@chromium.org, jbudorick@chromium.org, bpastene@chromium.org Change-Id: I9a5f1e0eebfe5da694f4c30a372e74f9757d593d Reviewed-on: https://chromium-review.googlesource.com/818635 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#523057} [modify] https://crrev.com/04f944830e7394ceb1b14858db12debf15bda6ec/testing/buildbot/chromium.android.fyi.json
,
Dec 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8ceeabfac539fb5e4ea35a52072dc1cefb03873e commit 8ceeabfac539fb5e4ea35a52072dc1cefb03873e Author: Kenneth Russell <kbr@chromium.org> Date: Mon Dec 11 17:53:28 2017 Autogenerate chromium.fyi.json and chromium.android.fyi.json. The latter was the first time where odd configuration differences were seen between bots on different waterfalls with the same name, and between gtests and instrumentation tests with the same names. Functionality was added to the generation script to be able to target these cases precisely, so that the existing JSON files could be autogenerated exactly as they are. (A couple of shortcuts were made in the coverage test in the meantime and left as TODOs.) Includes refactorings to several test suites for better reuse. BUG= 662541 Change-Id: I91975c6b9ab0a7fa378de79f32c12709531c2bbe Reviewed-on: https://chromium-review.googlesource.com/818210 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#523128} [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/chromium.android.fyi.json [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/generate_buildbot_json_unittest.py [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/test_suites.pyl [modify] https://crrev.com/8ceeabfac539fb5e4ea35a52072dc1cefb03873e/testing/buildbot/waterfalls.pyl
,
Dec 13 2017
,
Dec 15 2017
I'm going to call this "done enough". I've filed bug 795306 as a project/roll-up bug to track additional work, in order to make it clearer what's left and what next steps might be.
,
Dec 15 2017
,
Dec 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1af992036bb56704037df77e4617d9a13d0738f4 commit 1af992036bb56704037df77e4617d9a13d0738f4 Author: Dirk Pranke <dpranke@chromium.org> Date: Thu Dec 21 03:00:25 2017 Restore the //testing/buildbot/OWNERS. Now that we've converted all of the .json files to being generated by generate_buildbot_json.py, it's safe (enough) to restore the old OWNER rules. There's still a bunch of work to do to continue cleaning up the organization and documenting how to configure things. R=kbr@chromium.org BUG= 662541 Change-Id: Ie0f46b204a35da62d9136b83294270a53162cd19 Reviewed-on: https://chromium-review.googlesource.com/838424 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#525575} [modify] https://crrev.com/1af992036bb56704037df77e4617d9a13d0738f4/testing/buildbot/OWNERS
,
Jan 18 2018
Ken and Dirk, thank you so much for doing this. Awesome work :-) (somewhat belatedly; i've been away for a while and i'm catching up)
,
Mar 10 2018
,
May 17 2018
,
May 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5a89036b61de53c4083da98659cfb63c53923784 commit 5a89036b61de53c4083da98659cfb63c53923784 Author: Kenneth Russell <kbr@chromium.org> Date: Thu May 17 22:33:41 2018 Make generate_buildbot_json_coveragetest mandatory. coverage is now part of Chromium's vpython spec. This script is crucial to Chrome's infrastructure, and changes should not be allowed unless the standard of code coverage is maintained. Bug: 843842 , 662541 , 792130 Change-Id: I9164fe331045d13e08c1e1d40c79a01bac368ff9 Reviewed-on: https://chromium-review.googlesource.com/1063257 Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#559711} [modify] https://crrev.com/5a89036b61de53c4083da98659cfb63c53923784/testing/buildbot/PRESUBMIT.py
,
May 19 2018
|
||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||
Comment 1 by dpranke@chromium.org
, Nov 4 2016