Failing unittests don't show repro instructions |
||
Issue descriptionI've recently starting picking up some chrome development coming from V8 and I think the ergonomics of the test outputs could be improved. The following 2 steps are on every output page generated by V8 bots to help devs to repro their tests quickly and reliably. 1. GN Args: Display at the top of every log file to make sure you get a an easy way to repro 2. Command: Display the command that was used to invoke the test suite or a single crashing test. Chrome Example: --------------- a) Failing build: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/131496 b) Failing webkit_unit_test output: https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8942114123272082112%2F%2B%2Fsteps%2Fwebkit_unit_tests__with_patch_%2F0%2Fstdout b) Failing test output: https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8942114123272082112%2F%2B%2Fsteps%2Fwebkit_unit_tests__with_patch_%2F0%2Flogs%2FScriptStreamingTest.SmallScripts%2F0 Output b): - Nicely displays the command run for the tests - Lacks the build args though. I have to infer quite a bit of information to make sure I will properly repro this. Ouput c): - No gn args - No Command show to run failing test in isolation This is the place I most likely click since I don't want to grep through the complete output of b). However, there is no additional information there that helps me debug the failing test. V8 Example: ----------- a) Failing build bot: https://ci.chromium.org/p/v8/builders/luci.v8.try/v8_linux64_asan_rel_ng_triggered/b8942111319761167904 b) Failing test262 step: https://logs.chromium.org/v/?s=v8%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8942111319761167904%2F%2B%2Fsteps%2FTest262%2F0%2Fstdout c) Failing test262 "date-is-nan-throws" test: https://logs.chromium.org/v/?s=v8%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8942111319761167904%2F%2B%2Fsteps%2FTest262%2F0%2Flogs%2Fdate-is-nan-throws%2F0 Note how we have a nice header in output c) which tells me how to repro this with complete gn-args and command.
,
Jul 2
One can actually find needed information in the build, but it's really well-hidden. For GN args, one needs to look for generate_build_files step and the args are found in the stdout, e.g. [1] for the Chrome example above. For command used to execute test it's a little harder because tests are running on swarming, i.e. on a different machine: * on the failing step, there is one or more links named like "shard #N (failed)" - open them in a new tab(s), e.g. see attached image1.png * each link opens a page to the swarming UI, e.g. see [2], where one needs to find "Isolated Inputs" row in the table and click on the link to the hash nearby, e.g. see attached image2.png * one the resulting page, e.g. see [3], one can see the command used to run ALL tests on this shard In order to run just the failing test, which is what developers typically want, one needs to find the source code for the test launcher, which can be either a Python script or a C++ binary. For example, in this case web webkit_unit_tests binary is used, which is defined in [4]. To learn about it's arguments, I've downloaded the binary from the isolate using the following commands $ git clone https://github.com/luci/luci-py $ python luci-py/client/isolateserver.py download -I isolateserver.appspot.com --namespace default-gzip -f ffd7b077263858e888a000811be041d6fc91b711 ffd7b077263858e888a000811be041d6fc91b711 where ffd7b077263858e888a000811be041d6fc91b711 is the hash found on the isolate page next to webkit_unit_tests, see attached image3.png and then ran with --help: $ chmod u+x download/ffd7b077263858e888a000811be041d6fc91b711 $ ./download/ffd7b077263858e888a000811be041d6fc91b711 --help which allowed me to learn about --gtest_filter argument, so I guess one can run just the failing test with $ cd <chromium-checkout>/out/Release $ ./webkit_unit_tests --gtest_filter=AdTrackerSimTest.ResourceLoadedWhileExecutingAdScript --brave-new-test-launcher --test-launcher-bot-mode --asan=0 --msan=0 --tsan=0 --cfi-diag=0 but didn't test this. IMHO this is too complicated for developers who have no idea about swarming and isolate. The difficult part here is that Chromium has many different test launchers and all of them need to be updated to print out the command used to run individual tests, which can then be extracted from the output to be added to individual test logs. [1]: https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8942114123272082112%2F%2B%2Fsteps%2Fgenerate_build_files__with_patch_%2F0%2Fstdout [2]: https://chromium-swarm.appspot.com/task?id=3e73bd49595a0710&refresh=10&show_raw=1 [3]: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=38e3cfd2763710e10a3ec7230d88fefc58ef9aa0 [4]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/controller/BUILD.gn?l=74&rcl=923247620d3e7be6e61686bf12f6586d65f5c4ea
,
Jul 18
|
||
►
Sign in to add a comment |
||
Comment 1 by cbruni@chromium.org
, Jul 2