https://chromeperf.appspot.com/report?sid=c2f014e393753b77ebe300b0ed6d8f1a1f0de45a09d95bd32c610ff58ad64835 is some data we have.

It looks like things are mostly good here. It does make me sad to see that we're running less often (compare the number of data points for mean_frame_time). Although, the graph for swarming looks a bit weird; we might be missing data?

https://chromeperf.appspot.com/report?sid=c7ca22d1ba0330ae083563e8df205757b9199e08a4b90eb8a200cda4cce7ceb3 is a bit better of a graph. Doesn't use the old BenchmarkDuration metric.

+rendering experts: we are trying to migrate our Android bots in the perf lab to swarming infrastructure. For this bug, we are trying to validate whether the test results in the new infrastructure look odd comparing with the existing ones. Beside "benchmark_duration", do you have suggestions for which other metrics we should be looking at?


Smoothness metrics:
avg_surface_fps
first_gesture_scroll_update_latency
frame_lengths
frame_time_discrepancy
frame_times
input_event_latency_discrepancy
jank_count
main_thread_scroll_latency_discrepancy
max_frame_delay
mean_frame_time
mean_input_event_latency
mean_main_thread_scroll_latency
mean_pixels_approximated
mean_pixels_checkerboarded
percentage_smooth
queueing_durations

https://chromeperf.appspot.com/report?sid=013c3a223001bf118a8e357e4e74054559b11ef5a3eb446e5a799ff3c66d40a6 is a bit nicer of a graph to look at (replaces the link I pasted in #3)

How do I construct a graph like the one in #5, with multiple bots on the same axis?

Quickly skimming the full set of metrics shouldn't take long, should it? I don't think we'll need to pick a small number of metrics.

Thanks Tim, we will construct graph like the one in #5 with all the metrics then.

It looks like we're missing some data on the swarming bot. It hasn't had new data since.... march 3rd? I'll look into this.

So the bot is supposedly uploading the data. https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf.fyi%2FAndroid_Swarming_N5X_Tester%2F500%2F%2B%2Frecipes%2Fsteps%2Fsmoothness.top_25_smooth_Dashboard_Upload%2F0%2Fstdout is a log of it supposedly uploading data. But going to the link it gives us doesn't do anything.

sullivan@ (or someone else who knows the perf dashboard) do you know why data isn't showing up? I can't seem to see any data on the perf dashboard past build 403 (about march 9th)

I am seeing the data just fine here: https://chromeperf.appspot.com/report?sid=ebef4b72d983a1a7eadd0c6babfd2debe4e181bed9b320060841861d72b364b2&rev=463170

When a new bot is added, it needs to be made public otherwise you have to log in to see the data, maybe that's the problem?

In https://chromeperf.appspot.com/report?sid=ebef4b72d983a1a7eadd0c6babfd2debe4e181bed9b320060841861d72b364b2&rev=463170, I only see data up until April 1st. I don't see any data after that, even though it's been supposedly uploading data since then just fine. I see some, but not more recent data.

Ok, I figured out that I'm not crazy. We don't have some summary metrics, apparently because some tests are failing. I think? sullivan@ seems confident that it's not an error anywhere

I made a sort of graph here: https://chromeperf.appspot.com/report?sid=80f2674385ac30902cc4c5fa522eac91a55b16be7a7a252901c2552d88ff5092

It seems to show that the bots are mostly the same. The graph is sadly fairly manual to create. I only did the graph for youtube.com, and I don't think I can do it with a summary metric. I'm not that familiar with the perf dashboard.

Thanks for the graph, Stephen. For youtube case, I find them pretty comparable, with the swarming bot seems to produce more stable data.

+Victor/Tim: can any of you help validate these perf data from swarming Android bot as well?

Why are the swarming bots sampling less frequently?

frame_lengths/times looks a bit fishy, as there's movement when swarming that isn't showing up elsewhere.

Other than that, LGTM.

Swarming bots take longer right now, because of the way we run tests, so they run less frequently. We're working on getting the cycle time down, but for now I think they run less often, which results in less sampling :/

Yeah, I see some movement. Not sure where it is coming from. The device off swarming has similar jumps in data, so maybe it's just random noise.

Looks like we're good here.