I think the key problem here is v8 team are using the perf dashboard the way that are not well supported. 

The key focus of the product has always been around enable teams to get regression alerts. The use cases of watching over the performance graphs and quater-to-quarter evaluation are organic uses, thus not currently well designed yet.

I am not sure what to do about this bug. Probably better to bring this overall issue to the v8-benchmarking discussion.

Also to clarify the bug, this is only a builder renaming, there is no bots that get switched. So if we do perf dashboard migration from old name to new name (at least for data started since this quarter), there would be no gap

I think we need to have a fundamental discussion on how to use the dashboard then.


Most of the V8 folks doing performance work have to look at the graphs the old-fashioned way, there is simply no possibility to get positive feedback for sub-1% improvements in most cases.
Plus you always have to compare trade-offs, something that's also very hard to get from just looking at the regressions themselves.


Indeed, V8 use-cases are probably not well understood here.
I don't think any of these issues are new, in fact I remember having similar discussions 2 years ago.

Having a fundamental discussion sgtm. For this immediate bug though, I think we can still fix some of the bleeding by doing data migration.

Simon: if you think the data migration is the right solution, can you help with triaging?

+dtu, +ushesh

re: #c0

- Why is there no announcement email that bots are going to be renamed?

Sorry about that but unfortunately the name change was only relayed to us after the change went live as well, and I discussed concerns with benchmarking team about the possibility of data loss if we didn't do some sort of migration.


- Why is there no clear indication on chromeperf marking bots as deprecated?

Believe it takes 2 weeks for the dashboard to consider something deprecated (ie. no data being uploaded anymore).


- Why are bots switched in the middle of a quarter without overlap?

Emily summarized our discussion in a doc here with some possible approaches to migrating the data: https://docs.google.com/document/d/1NhMuaeYKxpRy56l-2Wvh0KJ0SIJ2FV6sSDK5qv_7TNk/edit



It kind of sounds like v8's use case is to look at fixed reports over time periods of a few months? Which, if that's the case, isn't unreasonable or unsupported.

Dave and I are discussing adding an OKR around establishing some best-practices for these kind of breaking changes.