New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 691500 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 3
Type: Feature



Sign in to add a comment

Bot test steps should provide a command line to run the test locally

Project Member Reported by alexclarke@chromium.org, Feb 13 2017

Issue description

I needed to run telemetry_perf_unittests to diagnose a flake (https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/116501), but it took 20 minutes of two engineer's time to figure out how to do so.  The error when you try to run that target is not very useful:

$ cr run telemetry_perf_unittests 
ninja: Entering directory `/usr/local/google/code/clankium/src/out_android/Release'
ninja: no work to do.
C    0.005s Main  Could not find apk or executable for telemetry_perf_unittests
E    0.006s Main  Unrecognized error occurred.
Traceback (most recent call last):
  File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 834, in main
    return RunTestsCommand(args)
  File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 629, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 715, in RunTestsInPlatformMode
    test_instance = test_instance_factory.CreateTestInstance(args, infra_error)
  File "/usr/local/google/code/clankium/src/build/android/pylib/base/test_instance_factory.py", line 18, in CreateTestInstance
    args, device_dependencies.GetDataDependencies, error_func)
  File "/usr/local/google/code/clankium/src/build/android/pylib/gtest/gtest_test_instance.py", line 306, in __init__
    error_func('Could not find apk or executable for %s' % self._suite)
  File "/usr/local/google/code/clankium/src/build/android/test_runner.py", line 661, in infra_error
    sys.exit(constants.INFRA_EXIT_CODE)
SystemExit: 87

After some hunting it turned out we need to specify a --browser parameter, but that doesn't actually work:

$ cr run telemetry_perf_unittests --browser=android-chromium
ninja: Entering directory `/usr/local/google/code/clankium/src/out_android/Release'
ninja: no work to do.
usage: test_runner.py [-h]
                      {gtest,instrumentation,junit,linker,monkey,perf,python}
                      ...
test_runner.py: error: unrecognized arguments: --browser=android-chromium
Error 2 executing command ['/usr/local/google/code/clankium/src/build/android/test_runner.py', 'gtest', '-s', 'telemetry_perf_unittests', '--release', '--browser=android-chromium']

More spelunking later and we figured out this command was needed:

tools/perf/run_tests -v --browser=android-chromium

Can we please add a print somewhere obvious in the bot stdout so the next person who needs to do this doesn't have a bad time?  Thanks!
 
Components: -Infra Infra>Client>Perf
Cc: sullivan@chromium.org
Annie - what do you think about this?
Cc: nednguyen@chromium.org dpranke@chromium.org
Dirk, is there a "right way" to do this?
There are two "right" ways to do this.

The first is to follow the instructions on the build step to click through to the swarming task page:

https://chromium-swarm.appspot.com/task?id=3440d4071b72f010

which tells you to reproduce the test locally by running:

python swarming.py reproduce -S chromium-swarm.appspot.com 3440d4071b72f010

which isn't quite right (you'd have to know that the script is in //tools/swarming_client/swarming.py).

Second, you can also reproduce it locally (without swarming) via

    python tools/mb/mb.py run -m tryserver.chromium.android \
        -b android_n5x_swarming_rel out/foo telemetry_perf_unittests

Which will run `gn gen` in out/foo using the build args the bot used, build telemetry_perf_unittests, and then run it.

Or, if you want to reproduce the environment and command line but reuse your existing build, just do:

    python tools/mb/mb.py run out/Release telemetry_perf_unittests

Eventually I'd like for the mb.py intermediary to go away and for us to just be able to say `out/Release/bin/telemetry_perf_unittests_run` or some such, much like you can do today on Android.

The documentation for all of this stuff is woefully inadequate at the moment :(.




Yeah, that's the thing I'm trying to figure out, how could we have made this easier for alexclarke@ to figure out? Is it short enough to add a step annotation?
I don't think we have a completely consistent way of answering this for every kind of test, and so I'm a bit reluctant to try and change something until we did (or thought about this a bit more globally).

So, maybe we should clear this from Infra>Client>Perf and reassign to me or something?
Components: -Infra>Client>Perf Infra
Owner: dpranke@chromium.org
Works for me.
Cc: -nednguyen@chromium.org

Comment 9 by s...@google.com, Feb 21 2017

Components: -Infra Infra>Platform
Labels: -Pri-2 Pri-3
Status: Available (was: Untriaged)
Cc: phajdan.jr@chromium.org estaab@chromium.org
Labels: OS-All
Owner: ----
I"m not sure when I'm going to get around to this, so I'll mark this as available.
Components: Infra>Client>Chrome
Project Member

Comment 13 by sheriffbot@chromium.org, Apr 26 2018

Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Status: Available (was: Untriaged)
Still useful, but we're all bit swamped atm, so back into our backlog.

Sign in to add a comment