Set up a controlled environment for build performance monitoring |
|||||
Issue descriptionSplitting this off from bug 779252 to track the larger effort ... Quoting from #c9 on that bug: I think we need to get a set of "control" builders in place, where we can assess how different environments are performing, and what the relative tradeoffs are. Then we can both better diagnose things like "why is builder X so slow" and also "is it better to use config X or config Y"? Recent examples of the sort of questions I'd like to be able to answer better: - when and where did bot_update get slow? Is it worse on Win? does location or GCE vs. Golo matter? What times should we expect? Is it possible that a source change landed that is affecting this? - when and where did the archive_layout_tests step get so slow? Why is gsutil 10x slower on windows than linux? why is a cloud<->cloud copy inside a single bucket the same speed as a client->cloud copy - is moving bots from ESX VMs in the golo to GCE VMs actually a good idea? is an n1-standard-8 VM actually equivalent to a 8-core VM on one of our 720s? - should we add more cores, or more memory, or switch to SSDs, or all of the above? Would one n1-standard-32 VM be better than 4 n1-standard-8 VMs? - what's the relative impact of goma config changes, or even building w/o goma? - what about jumbo builds? E.g., I have jumbo builders on chromium.fyi now, but they're not all that useful for evaluating the impact since I don't have comparable non-jumbo builders to compare them to. - we can now fully cross-compile win chromium on linux hosts. Is that faster than a win-native build? What about running the Linux subsystem on Win10? What about the Linux subsystem + an ext4-formatted filesystem? Obviously, actually being able to answer any of these questions requires additional work, but having controlled data is probably required for some of them, and would probably make most of the rest easier.
,
Nov 2 2017
By the way, can we rely on monitored metrics of buildbots to know which resource is not sufficient in each build steps? https://cs.chromium.org/chromium/infra/go/src/infra/monitoring/sysmon/?sq=package:chromium
,
Nov 3 2017
Good question; I don't know.
,
Nov 14 2017
From https://bugs.chromium.org/p/chromium/issues/detail?id=779252#c19 : ESX win10 SSD: vm350-m1 ESX win10 Normal Disk: vm348-m1 ESX trusty SSD: vm65-m1 ESX trusty Normal Disk: vm349-m1
,
Dec 18 2017
,
Feb 1 2018
martiniss@, as per my comments in your doc, I'm going to punt this to you to own now.
,
Dec 4
I made a bit of progress here; I made some docker containers. I haven't worked on this in a long time. Closing for now, someone could revisit this later potentially. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by dpranke@chromium.org
, Oct 30 2017