e.g. "query all results in dom/* that pass on 2 browsers other than Chrome"
qyearsley@ mentioned that the perf dashboard's schema mirrors the hierarchy of the test suite. We could structure our tests like so:
TestFile '/'
TestFile '2dcontext'
TestFile 'building-paths'
TestFile 'compositing'
TestFile '2d.composite.canvas.copy.html'
TestFile '2d.composite.canvas.destination-atop.html'
We'd also have to store the test results as a scalar property on TestFile in order to be filterable. I'm not sure how best to do this. Here's my best idea now:
TestFile has properties that are the percent of all passing subtests:
Testfile.chrome_pass_rate = 100
Testfile.edge_pass_rate = 100
Testfile.firefox_pass_rate = 100
So you could answer the query on the 1st line with:
TestFile.query(ancestor=ndb.Key(TestFile, '/', TestFile, 'dom'),
chrome_pass_rate < 100,
firefox_pass_rate = 100,
safari_pass_rate = 100)
e.g. "query all results in dom/* that pass on 2 browsers other than Chrome"
qyearsley@ mentioned that the perf dashboard's schema mirrors the hierarchy of the test suite. We could structure our tests like so:
TestFile '/'
TestFile '2dcontext'
TestFile 'building-paths'
TestFile 'compositing'
TestFile '2d.composite.canvas.copy.html'
TestFile '2d.composite.canvas.destination-atop.html'
We'd also have to store the test results as a scalar property on TestFile in order to be filterable. I'm not sure how best to do this. Here's my best idea now:
TestFile has properties that are the percent of all passing subtests:
Testfile.chrome_pass_rate = 100
Testfile.edge_pass_rate = 100
Testfile.firefox_pass_rate = 100
So you could answer the query at the beginning of this description with:
TestFile.query(ancestor=ndb.Key(TestFile, '/', TestFile, 'dom'),
chrome_pass_rate < 100,
firefox_pass_rate = 100,
safari_pass_rate = 100)
So, I don't think this approach will work out. In order to achieve this with AE Datastore, we'll need separate indexes for every permutation of browser order, which gets large really quickly. It also doesn't easily support adding new browsers.
This task might lend itself more to precomputation. We already do some precomputation for pass rates within TestFiles. This would require adding a BrowserResult model, but after uploading a new set of results we could set a flag on each browser result to indicate whether it's failing and others are passing. Then querying by that flag would be pretty straightforward:
BrowserResult.query(parent=ndb.key(Directory, 'html', TestFile, 'test.html'),
failing_individually=True)
The datastore schema has been updated to be hierarchical, like in #0, and we now have pass rate computed properties like TestFile.chrome_pass_rate. Closing.
Comment 1 by jeffcarp@chromium.org
, Jan 18 2017