On the other hand, if we wanted to ingest results from the current Travis CI runs, it would be more convenient to offer an HTTP API.

For  issue 647861 , would that be moot if WebRTC tests were being run and updated in the dashboard frequently? Would the tests be run against some other browser/configuration than we want by default, perhaps?

I'm inclined to say that we should keep it simple and only store full runs, and to do each run against the same commit of web-platform-tests. Results from pre-release Edge would fit into that scheme though. Has that been requested by someone at Microsoft?

> For  issue 647861 , would that be moot if WebRTC tests were being run and updated in the dashboard frequently? Would the tests be run against some other browser/configuration than we want by default, perhaps?

I'm not familiar with the specifics but agouaillard is building a WebRTC test setup comprising multiple computers, which I think allows running a number of tests that wouldn't otherwise be testable on one machine.

> Has that been requested by someone at Microsoft?

Not explicitly, but when we were at the WPT convergence one of the topics being discussed was how to easily get results from pre-release versions of Edge.

Edge is one deliverable configuration of the project. While the project is mainly for webrtc INTEROPERABILITY (i.e. two browsers in the same call), we designed it to be able to run test suites that use one browser at a time like WPT (the fact that both google project manager and myself are maintainers of /webrtc and /mediastream in WPT has ab-so-lu-te-ly nothing to do with that decision :D )

it is available in our system both through sauce labs, and soon through a pre-configured VM:

Original VM images are available here: https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

Edge [webrtc] FAQ is here: http://internaut.com:8080/~baboba/iit-tutorial/MicrosoftEdgeRTCQA.pdf

Note that the term interoperability is also used to mean that an API has the same behavior in multiple implementations even in cases where two implementations never communicate with each other like in the WebRTC case. (From https://docs.google.com/document/d/1LSuLWJDP02rlC9bOlidL6DzBV5kSkV5bW5Pled8HGC8/edit?usp=sharing)

For the WebRTC testing effort, is there a need to do something other than run web-platform-tests in various browsers?

in the webrtc scope, "interoperability" often means the capacity to have two browsers interoperate with each other using webrtc (p2p), as opposed to having a web app behaving the same way across browsers. 

The main difference here is that you run wpt in one browser at a time, and/or in parallel in multiple browser, but one browser at  time. For webrtc, you need at least 2 browsers running a common, asymmetrical and asynchronous test, at a time.

we separated the test engine, whose duty is to manage the selenium grids and the dispatch of the test across the grids, from the tests/result callback, and number of browser used in each test. That allow us to treat WPT as a special case where N=1, the test is to be written based now wpt, and the callback will be the wptdashboard callback URL.

Are these tests part of web-platform-tests, or will they be? Browser-to-browser isn't something that can currently be tested using wptrunner I assume, so is there a test launcher that lives outside of web-platform-tests, at minimum?

our work is separated for now. the repository should be public soon.

what we call "test" in our context is a java class that control N browsers through web driver and run the same (JS) test on both of them. In the N=2 case, we are using appRTC as a test base, and we created a java wrapper around to instrument the browsers (accept calls, ...). It will be separated from wpt.

However, since we wrote the engine to be independent of the number of participating browsers in one test, one can use our engine with N=1 to run wpt. The advantage here is that you don't have to modify wpt to know about web driver or sauce lab, or anything new. That s the key point of our design: to keep things separated. By separating the test engine, the test itself, and the test callback, one can add browser configuration or service provider without modifying the test code. One can also change the dashboard, or the visualisation depending of the usage. for some run you will want the result in wptdashoard, but if you use this in a CI environment, you might want to redirect the result elsewhere. As long as the test output results in format understandable by the dashboards, you can change the dashboard at each run if you want.

We plan to provide the glue, i.e. the java test that runs wpt and return results to wptdashboard, to show how we could populate the wptdashboard DB using a few runs of our test engine. We are finishing a first dashboard for apart right now, and wpt should come after, before end of April.

Do you have plans for merging this into web-platform-tests proper? If running multiple browsers at once is critical for judging WebRTC interoperability, then maybe wptrunner or something should know how to do that.

There's a lot of ongoing work on infrastructure for wpt, and it'd be great if the WebRTC tests got all of those benefits for free. Currently, it's the ability to add/modify tests together with any Chromium code change and have those changes automatically exported. Also test stability checking when modifying tests in the upstream web-platform-tests repository. Eventually this will come together so that one can change tests in a Chromium code review, and know how those tests will run in all other engines.

And then there's the dashboard of course :)

bob@bocoup.com FYI

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This has been done now, by robertma@. (And this was the last open issue here with "WPT Dashboard" in the title, project is now at https://github.com/web-platform-tests/wpt.fyi)