standardize test results parsing/structs |
||||||
Issue descriptionIn an ideal world, there would be a proto definition for test results, and we'd just use the generated language bindings in apps that produce or consume test results. That would eliminate a lot of errors, ambiguity, workarounds and special cases in our code. Instead, today we have multiple test result json parsers (at least 2: one for test-results.appspot.com/upload, and another one for sheriff-o-matic's analyzers) and multiple generators (various python recipes produce them directly, some use gtest->json converter, others?). First, we should see what happens when we propose switching from arbitrary JSON to protos. Is there a lot of justifiable pushback from test writers/maintainers?
,
Dec 4 2017
The JSON results format is considered to be the current standard: https://bit.ly/chromium-json-test-results-format Any test suite that isn't generating this format needs to be modified to do so; I'm not sure if we still have any that aren't. It's not actually clear to me that switching to a proto *by itself* will fix any of the "errors, ambiguity, workarounds and special cases" that you refer to. I do think that any time to try and force people to be more compliant, you can clean things up, and often switching to a new format can help with that. I think there is value in trying to reduce the number of variants of the existing format. It's less clear to me if there is any real value to switching to protos beyond that, unless it would buy us something like better integration with other systems. Most of our test harnesses are lightweight python wrappers and have no real mechanism for integrating w/ proto files and proto-generated stubs.
,
Dec 20 2017
,
Mar 15 2018
Another example pain: https://bugs.chromium.org/p/chromium/issues/detail?id=820428
,
Mar 15 2018
@seanmccullough - I feel like we don't have a clear path forward here. Here's my take: 1) There is a standardized "wire" format. There aren't currently standardized client libraries for parsing this file. 2) I'm not sure which things you're asking for: 2a) improvements to the spec for the format? 2b) client libraries for the format? 2c) switching from the existing spec to protos I think you'd be happy w/ 2c) which would get you 2a) and 2b) as well, but be somewhat expensive since we'd have to come up with a new spec and update all of the existing users. However, if we chose not to do that, are there things in the 2a/2b categories that we could do? e.g., TYP parses things for Python, but I don't think we have an agreed-upon library for Go, and not everything in Python uses TYP now.
,
Mar 15 2018
[removed proto mention from description since that was prematurely prescriptive] Agree this problem description isn't quite clear enough to act on. I've observed a few different problems with the JSON format and I think I just conflated them all because protos fix them. I'll try to collect my thoughts offline and formulate a better problem statement. This isn't at all urgent of course, but the recent discussion about adding "is_regression" in https://bugs.chromium.org/p/chromium/issues/detail?id=822078 did get me thinking again about schema evolution and how changes to them are managed with and without formal specs. Leaving this here for future reference, since it may help with 2a or 2b: http://json-schema.org/
,
Apr 28 2018
,
Apr 28 2018
,
Apr 28 2018
,
Aug 1
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by bugdroid1@chromium.org
, Oct 30 2017