+sullivan, what are ref builds?
like a reference build of webview as a baseline?

Yes, it's a build of chrome stable that runs on the same device as the ToT build under test. Then if there is a device swap, we can see easily on the graph that the ref build jumped too, and know it's not a real performance regression.

Do you know where the ref builds are stored or where in the infra code perf tests on the reference builds are run? Are tests on the reference builds only run once or run continually? I can work on this but am not sure where to start.

Dave, Kari, can you help answer the questions in #3?

oh, I think I see how it is done now. There is a --browser=reference flag on the perf test runner. We might have to add --browser=webview-reference to fix this? I'll work on it.

It's a bit more complicated than that. Telemetry team is talking about it offline because it might be better to fix some complexity than add more, and I'll update this bug in a bit.

Any updates on how this should be fixed?

 Issue 654647  has been merged into this issue.

eakuefner, nednguyen: Kari mentioned you had a discussion about the right way to do WebView ref tests. Do either of you remember the results?

I think we mused about this a few quarters ago in the context of the Options/Platform/BrowserFinder overhaul that we designed but never implemented, for lack of an immediate motivation: https://docs.google.com/document/d/14GYqOmG2A8L1-biEDFSvgY9bFWWm7ruaCzgHd8Vmx00/edit

Understanding the full context of the refactor is probably not necessary, but I talked to Kari offline and we think a sketch of the plan to add --browser=webview-reference looks something like the following:

1. Add a webview-stable dependency to chrome_binaries.json: https://github.com/catapult-project/catapult/blob/master/common/py_utils/py_utils/chrome_binaries.json
2. Use this dependency to provide a new browser type 'webview-reference on Android that works like 'reference' currently does: https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/backends/chrome/android_browser_finder.py#L93
3. Modify the dashboard's recognition of what a reference build is to match on --browser=*-reference in addition to --browser=reference

This is a rough sketch, so it also might require touching some surrounding code to work properly, but those are all the big places where code would need to be modified.

Interested to hear if Ned has any additional input.

"--browser=webview-reference" or "--browser=reference-webview" is fine (I prefer the latter)

+Emily: how are you currently generate the commandline for reference benchmark? We may need to the script for the webview case.

It would just be an extra arg added to the src side buildbot json in src/testing/buildbot/*.  It is generated through generate_perf_json.py: https://cs.chromium.org/chromium/src/tools/perf/generate_perf_json.py?q=generate_perf&sq=package:chromium&l=1

You will see the logic for adding a reference benchmark you can add the flag in those instances.

Any update here? Should this be P1?

 Issue 715977  has been merged into this issue.

Taking myself off of owner here in case someone else wants to take this up

I don't think we have any one to tackle this now. 

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I think the discussions on issue 799950 on the --browser selection mechanism are also relevant.