We are repeating 23 getPrimaryService(s) tests over 60 files and 12 getCharacteristic(s) tests over 34 files. For each test that we repeat, the code stays the same except for one line.
For example, getPrimaryService/device-disconnected.html, getPrimaryServices/device-disconnected.html and getPrimaryServices/device-disconnected-with-uuid.html have all the same code except for one line, the line that calls a different variation of getPrimaryService(s) i.e. getPrimaryService(‘heart_rate’), getPrimaryServices(), getPrimaryServices(‘heart_rate’).
The number of such tests is going to increase as we implement getIncludedService(s), getDescriptor(s) and start requiring frames to be connected.
Our objective is to reduce the number of lines we have to manually write and consolidate the test code in one place. This will improve readability so that it’s easier to make sure we are testing the same across all function variations, reduce maintenance burden and reduce the cost of writing new tests.
Two possibilities:
1. Add a helper function for each test.
Pros:
- Test code lives in one place
- No need for code generation script
Cons:
- Fuzzer can’t fuzz test since it doesn’t fuzz resource files. (Though, we already have our own fuzzer.)
- We still need to manually write ~3 files for each test.
2. Use a generate.py script to generate each tests
Pros:
- Test code lives in one place
- All test files are generated for us
- Because all tests are generated and checked in fuzzer can fuzz the tests
- There is precedence https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/http/tests/fetch/README?q=generate.py+file:%5Esrc/third_party/WebKit/LayoutTests/&sq=package:chromium&l=17&dr=C
Cons:
- Checking in generated code
- Need to manually run script to generate test files. We can include a presubmit script to warn if test files are incorrectly modified, though
I briefly talked to haraken. He mentioned that generating tests is indeed possible but we should make sure the number of tests that will use this is worth the added complexity.
@jyasskin: Do you have a preference? I'm leaning towards 2. because of the fuzzer issue and less manual writing.
Comment 1 by ortuno@chromium.org
, Oct 11 2016