Fix up nvda_chrome_tests.py |
||
Issue descriptionI'd like to use nvda_chrome_tests.py to better test chromium<>nvda. I propose the following changes: * Add a check to ensure that nvda.exe isn't running * Make testing for speech more 'fuzzing'. ** Currently, the NVDA speech output is read in each time the test asks to wait for speech. In order for the wait method to pass, the speech text must match exactly. We should change this so that a test can simply say 'wait for the text "foopy"', and we'll look for the word 'foopy' regardless of the speech before it. Doing tihs will allow us to run test cases in different orders/ * Convert so that it can run in python 3. optional * Update some of the tests to match reality. Also, we should try hard to make the tests as robusts as they can be. Right now, we have a high failure rate (flakiness) because the test doesn't exactly match. In some cases this is timing, and in other cases it's because of our dynamic UI (reload button is a stop button during load). This bug will track this work.
,
Aug 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d7108c824fafeae73d2ab90524cf9b0b066bd839 commit d7108c824fafeae73d2ab90524cf9b0b066bd839 Author: Doug Turner <dougt@chromium.org> Date: Thu Aug 03 08:50:49 2017 Fix up nvda_chrome_tests.py to run under Python 3. See "Print Is A Function". Bug: 742592 Change-Id: Ic2744a01a4149a466dab9a7c1208d36238206008 Reviewed-on: https://chromium-review.googlesource.com/599567 Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Commit-Queue: Doug Turner <dougt@chromium.org> Cr-Commit-Position: refs/heads/master@{#491681} [modify] https://crrev.com/d7108c824fafeae73d2ab90524cf9b0b066bd839/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dccc49f05a9acaa65d3103f8b076d3eee7361b5b commit dccc49f05a9acaa65d3103f8b076d3eee7361b5b Author: Doug Turner <dougt@chromium.org> Date: Thu Aug 03 17:42:09 2017 Fix up octal use in nvda_chrome_tests.py to run under Python 3. Bug: 742592 Change-Id: I5d263b00a4d5d1ab0489217f09e73a353c3021b0 Reviewed-on: https://chromium-review.googlesource.com/599568 Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Commit-Queue: Doug Turner <dougt@chromium.org> Cr-Commit-Position: refs/heads/master@{#491780} [modify] https://crrev.com/dccc49f05a9acaa65d3103f8b076d3eee7361b5b/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fe594161342f240db05901da4d583f3dd9989e18 commit fe594161342f240db05901da4d583f3dd9989e18 Author: Doug Turner <dougt@chromium.org> Date: Thu Aug 03 18:21:32 2017 Exit early from nvda_chrome_tests.py if nvda.exe is running. Bug: 742592 Change-Id: I92fe2330345f60d599e6fd7c77d5c74c20237268 Reviewed-on: https://chromium-review.googlesource.com/599569 Commit-Queue: Doug Turner <dougt@chromium.org> Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Cr-Commit-Position: refs/heads/master@{#491797} [modify] https://crrev.com/fe594161342f240db05901da4d583f3dd9989e18/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d07794506ed64430f85c18824253897740e86e63 commit d07794506ed64430f85c18824253897740e86e63 Author: Doug Turner <dougt@chromium.org> Date: Thu Aug 03 22:57:24 2017 _WaitForSpeech should not wait for speech output that happened in the past. Currently, nvda_chrome_tests.py _WaitForSpeech will block for a period of time until NVDA speaks some given text. The implementation reads output from a log file that NVDA writes to. This read just starts at the start of the file and builds a list of what was said. This means that each call to _WaitForSpeech() must contain the prior expected text. This CL simply remembers the last offsite and when reading the file again we ignore old stuff. I don't think that re-reading the file is a big deal since we're barely testing with this approach. At some point, if it becomes a problem, we might think of something more clever. Bug: 742592 Change-Id: I47b417f7aa7b7cf8f6f6361d4581ab0ea562a042 Reviewed-on: https://chromium-review.googlesource.com/599373 Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Commit-Queue: Doug Turner <dougt@chromium.org> Cr-Commit-Position: refs/heads/master@{#491869} [modify] https://crrev.com/d07794506ed64430f85c18824253897740e86e63/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dad723268c7ffb48d897d4d30238110b0c8c6ccb commit dad723268c7ffb48d897d4d30238110b0c8c6ccb Author: Doug Turner <dougt@chromium.org> Date: Thu Aug 03 23:51:51 2017 NVDA test method _WaitForSpeech should ignore unrelated speech. Current, we have a method _WaitForSpeech which blocks on NVDA to speak exactly the given set of text. What this CL does is modifies _WaitForSpeech such that it only tests for the ordered set of given text and ignores text that may have happened prior to the text we're looking for. Because of this change, I also renamed the method to _TestForSpeech Bug: 742592 Change-Id: I37f2ff890a2f8860afac17b0011300d8fd34dd3f Reviewed-on: https://chromium-review.googlesource.com/599607 Commit-Queue: Doug Turner <dougt@chromium.org> Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Cr-Commit-Position: refs/heads/master@{#491881} [modify] https://crrev.com/dad723268c7ffb48d897d4d30238110b0c8c6ccb/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0a0540f1f8f8400d7db82c2f96eb3310f2c6e987 commit 0a0540f1f8f8400d7db82c2f96eb3310f2c6e987 Author: Doug Turner <dougt@chromium.org> Date: Fri Aug 04 00:55:09 2017 Reduce the tests in nvda_chrome_tests.py to things that pass more often. nvda_chrome_tests.py tests the interactions between Chrome <> NVDA. The tests currently are very flakey and this is an attempt to prune things that simply no longer work anymore. The intent is not to just leave it as is, but to continue to add tests that work more often to these test cases. Bug: 742592 Change-Id: I3d6c5cc82dcbf63de154fe2dd79a8b6b5f41a7ff Reviewed-on: https://chromium-review.googlesource.com/599589 Commit-Queue: Doug Turner <dougt@chromium.org> Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Cr-Commit-Position: refs/heads/master@{#491896} [modify] https://crrev.com/0a0540f1f8f8400d7db82c2f96eb3310f2c6e987/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1441b968d8153d4d01345ad46264053f2513549a commit 1441b968d8153d4d01345ad46264053f2513549a Author: Doug Turner <dougt@chromium.org> Date: Fri Aug 04 01:21:06 2017 nvda_chrome_tests.py should not block on speech during setup. The setup of these tests should not be testing for speech. Instead if speech doesn't work, the test cases will catch it. Bug: 742592 Change-Id: I656a40ae8a304ac1161941929d88409aacc144ae Reviewed-on: https://chromium-review.googlesource.com/599590 Commit-Queue: Doug Turner <dougt@chromium.org> Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Cr-Commit-Position: refs/heads/master@{#491903} [modify] https://crrev.com/1441b968d8153d4d01345ad46264053f2513549a/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/113e2d4177a9aef7f6c5b9a3268c5057145fa7e4 commit 113e2d4177a9aef7f6c5b9a3268c5057145fa7e4 Author: Doug Turner <dougt@chromium.org> Date: Fri Aug 04 01:52:04 2017 Update nvda_chrome_tests.py to the latest pywinauto API. Bug: 742592 Change-Id: Id572d0f9961a0c2795ae9abbc73696a05a7b94bf Reviewed-on: https://chromium-review.googlesource.com/599627 Commit-Queue: Doug Turner <dougt@chromium.org> Reviewed-by: Alice Boxhall <aboxhall@chromium.org> Cr-Commit-Position: refs/heads/master@{#491913} [modify] https://crrev.com/113e2d4177a9aef7f6c5b9a3268c5057145fa7e4/tools/accessibility/nvda/nvda_chrome_tests.py
,
Aug 4 2017
|
||
►
Sign in to add a comment |
||
Comment 1 by dougt@chromium.org
, Jul 17 2017