Bot test steps should provide a command line to run the test locally |
|||||||||||
Issue descriptionI needed to run telemetry_perf_unittests to diagnose a flake (https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/116501), but it took 20 minutes of two engineer's time to figure out how to do so. The error when you try to run that target is not very useful: $ cr run telemetry_perf_unittests ninja: Entering directory `/usr/local/google/code/clankium/src/out_android/Release' ninja: no work to do. C 0.005s Main Could not find apk or executable for telemetry_perf_unittests E 0.006s Main Unrecognized error occurred. Traceback (most recent call last): File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 834, in main return RunTestsCommand(args) File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 629, in RunTestsCommand return RunTestsInPlatformMode(args) File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 715, in RunTestsInPlatformMode test_instance = test_instance_factory.CreateTestInstance(args, infra_error) File "/usr/local/google/code/clankium/src/build/android/pylib/base/test_instance_factory.py", line 18, in CreateTestInstance args, device_dependencies.GetDataDependencies, error_func) File "/usr/local/google/code/clankium/src/build/android/pylib/gtest/gtest_test_instance.py", line 306, in __init__ error_func('Could not find apk or executable for %s' % self._suite) File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 661, in infra_error sys.exit(constants.INFRA_EXIT_CODE) SystemExit: 87 After some hunting it turned out we need to specify a --browser parameter, but that doesn't actually work: $ cr run telemetry_perf_unittests --browser=android-chromium ninja: Entering directory `/usr/local/google/code/clankium/src/out_android/Release' ninja: no work to do. usage: test_runner.py [-h] {gtest,instrumentation,junit,linker,monkey,perf,python} ... test_runner.py: error: unrecognized arguments: --browser=android-chromium Error 2 executing command ['/usr/local/google/code/clankium/src/build/android/test_runner.py', 'gtest', '-s', 'telemetry_perf_unittests', '--release', '--browser=android-chromium'] More spelunking later and we figured out this command was needed: tools/perf/run_tests -v --browser=android-chromium Can we please add a print somewhere obvious in the bot stdout so the next person who needs to do this doesn't have a bad time? Thanks!
,
Feb 13 2017
Annie - what do you think about this?
,
Feb 13 2017
Dirk, is there a "right way" to do this?
,
Feb 13 2017
There are two "right" ways to do this. The first is to follow the instructions on the build step to click through to the swarming task page: https://chromium-swarm.appspot.com/task?id=3440d4071b72f010 which tells you to reproduce the test locally by running: python swarming.py reproduce -S chromium-swarm.appspot.com 3440d4071b72f010 which isn't quite right (you'd have to know that the script is in //tools/swarming_client/swarming.py). Second, you can also reproduce it locally (without swarming) via python tools/mb/mb.py run -m tryserver.chromium.android \ -b android_n5x_swarming_rel out/foo telemetry_perf_unittests Which will run `gn gen` in out/foo using the build args the bot used, build telemetry_perf_unittests, and then run it. Or, if you want to reproduce the environment and command line but reuse your existing build, just do: python tools/mb/mb.py run out/Release telemetry_perf_unittests Eventually I'd like for the mb.py intermediary to go away and for us to just be able to say `out/Release/bin/telemetry_perf_unittests_run` or some such, much like you can do today on Android. The documentation for all of this stuff is woefully inadequate at the moment :(.
,
Feb 13 2017
Yeah, that's the thing I'm trying to figure out, how could we have made this easier for alexclarke@ to figure out? Is it short enough to add a step annotation?
,
Feb 13 2017
I don't think we have a completely consistent way of answering this for every kind of test, and so I'm a bit reluctant to try and change something until we did (or thought about this a bit more globally). So, maybe we should clear this from Infra>Client>Perf and reassign to me or something?
,
Feb 13 2017
Works for me.
,
Feb 14 2017
,
Feb 21 2017
,
Feb 25 2017
,
Feb 25 2017
I"m not sure when I'm going to get around to this, so I'll mark this as available.
,
Apr 26 2017
,
Apr 26 2018
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue. Sorry for the inconvenience if the bug really should have been left as Available. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 27 2018
Still useful, but we're all bit swamped atm, so back into our backlog. |
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by vadimsh@chromium.org
, Feb 13 2017