Move last-try restart behind a flag |
||||
Issue descriptionI got confused when my test device restarted itself after running instrumentation tests for a change I was writing. My change had made the test fail, but I was trying to do TDD :-P To make it worse, when I reverted my code (and the tests passed) the device stopped restarting after the tests, leading me to think I'd hit some horrendous Android breaking bug, which I could find no sign of in the device logs. It turns out that the test class had @RetryOnFailure. It would be good if we could have some test output explaining that the device has been restarted to retry the test. I was also running the tests with '--num-retries=0', so restarting the device to retry doesn't make sense. Could we disable this behaviour when num-retries=0? Also, would it be an idea to add a convenience flag to the test runner such as "developer-run" which developers could use to get behaviour that makes sense on a developer machine (instead of a trybot or tester).
,
Dec 1 2017
As for a dev run flag -- that's kind of what --fast-local-dev tries to do: https://codesearch.chromium.org/chromium/src/build/android/test_runner.py?rcl=46a41a802079d801a4d7b3f9d8ad2bf0fd941243&l=176
,
Dec 1 2017
Yeah, I'm not able to reproduce this locally w/ a failing test and --num-retries=0. If you've got any more details on what you saw, potentially including a patch, that'd be helpful.
,
Dec 1 2017
Talked to tedchoc@ and mdjones@, who expressed interest in not having the last-try restart enabled for local test execution. That sounds reasonable to me, and I think the best way to do so at the moment would be to put it behind a flag and pass that to bot test executions.
,
Dec 1 2017
It seems that recovery logic I added is doing more harm than good. Whoops. I'll hide it behind a bot-only flag.
,
Dec 4 2017
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1ddc023aa30dcedf41e5f9dcc62b351e8712a689 commit 1ddc023aa30dcedf41e5f9dcc62b351e8712a689 Author: Benjamin Pastene <bpastene@chromium.org> Date: Tue Dec 05 19:44:15 2017 android: Add test runner option to recover devices before last attempt. Will enable on bots by either: - adding option to testing spec - adding option to all isolated tests via mb - adding option to cmdline in recipe Leaning towards option 3 atm. R=jbudorick@chromium.org Bug: 790936 Change-Id: Ieb9e0e8237e691887e383578f1860cd77e03e5cb Reviewed-on: https://chromium-review.googlesource.com/804914 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Benjamin Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#521791} [modify] https://crrev.com/1ddc023aa30dcedf41e5f9dcc62b351e8712a689/build/android/pylib/local/device/local_device_environment.py [modify] https://crrev.com/1ddc023aa30dcedf41e5f9dcc62b351e8712a689/build/android/pylib/local/device/local_device_test_run.py [modify] https://crrev.com/1ddc023aa30dcedf41e5f9dcc62b351e8712a689/build/android/test_runner.py
,
Dec 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/766d48f58f34794918d8c547dc9a9cf957e3ef04 commit 766d48f58f34794918d8c547dc9a9cf957e3ef04 Author: Benjamin Pastene <bpastene@chromium.org> Date: Mon Dec 18 21:47:42 2017 Add --recover-devices arg to all android tests. Had to add some special-case logic to generate_buildbot_json.py Bug: 790936 Change-Id: Ie255718f6c5fc022e203970f4278937285bed7f3 Reviewed-on: https://chromium-review.googlesource.com/810027 Commit-Queue: Benjamin Pastene <bpastene@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#524810} [modify] https://crrev.com/766d48f58f34794918d8c547dc9a9cf957e3ef04/testing/buildbot/chromium.android.fyi.json [modify] https://crrev.com/766d48f58f34794918d8c547dc9a9cf957e3ef04/testing/buildbot/chromium.android.json [modify] https://crrev.com/766d48f58f34794918d8c547dc9a9cf957e3ef04/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/766d48f58f34794918d8c547dc9a9cf957e3ef04/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/766d48f58f34794918d8c547dc9a9cf957e3ef04/testing/buildbot/generate_buildbot_json_unittest.py |
||||
►
Sign in to add a comment |
||||
Comment 1 by jbudorick@chromium.org
, Dec 1 2017Status: Available (was: Untriaged)