I checked out Browserstack. Using wptrunner didn't work for me, so I set up a simple script using the WebDriver remote API to hit a test on w3c-test.org, and extracted the JSON results from the DOM.

I ran it against Safari on macOS and iOS on one test and uploaded the results:
https://wptdashboard.appspot.com/t/dom/events/Event-dispatch-bubbles-false.html 

The pricing is $79/mo for unlimited runs (in serial). I think this might be a viable option for running tests on harder to test platforms like iOS Safari.

Cool!

Yeah definitely expense a monthly plan for yourself if you find it helpful/practical (Philip and I and several others on the team already have our own browserstack account).

Philip is also looking into some sort of team license to one of these (so don't get the annual plan, hopefully we can just switch it over to our team license once that's in place).

If you could try Sauce Labs as well, it'd be interesting to learn which seems easiest to work with and most reliable for this kind of thing.

Yep, will do!

markdittmer said he had a lot of trouble with reliability with sauce labs - selenium connections dying or something.  But if we're doing a site license with them then presumably we'd get some support.

(Updating the title to specify Safari since we're missing data on it)

FYI there is a preliminary PR for Safari in wptrunner at [1], and an outdated sauce branch at [2].

For running on macOS in general https support isn't there yet (one needs to install certs into the OS cert store I think). For running on platforms that don't give you root that may be impossible to do. I feel like I'd rather have a conversation with Mozilla people about whether we can run jobs on our new CI infrastructure (for which I think macOS support is forthcoming) than depend on something like browserstack here. Of course that might not support mobile so easily, but I at least am not too bothered by that yet.

[1] https://github.com/w3c/wptrunner/pull/225
[2] https://github.com/w3c/wptrunner/tree/jgraham/sauce

That makes sense, maybe we can talk more in-depth about it at the convergence Monday? re: Safari support - thanks! For Safari I was doing an initial run of the testharness tests by grabbing the JSON out of the __testharness__results__ div. I hope that's not too janky.

Did you do a run, or is this actually a WontFix?

Yes, I did a run of Safari on Browserstack before BlinkOn (the results should be up on the dashboard).

I see, thanks :)