New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 779834 link

Starred by 0 users

Issue metadata

Status: WontFix
Owner:
Closed: Dec 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 779252



Sign in to add a comment

Set up a controlled environment for build performance monitoring

Project Member Reported by dpranke@chromium.org, Oct 30 2017

Issue description

Splitting this off from bug 779252 to track the larger effort ...

Quoting from #c9 on that bug:

I think we need to get a set of "control" builders in place, where we can assess how different environments are performing, and what the relative tradeoffs are. Then we can both better diagnose things like "why is builder X so slow" and also "is it better to use config X or config Y"?

Recent examples of the sort of questions I'd like to be able to answer better:

- when and where did bot_update get slow? Is it worse on Win? does location or GCE vs. Golo matter? What times should we expect? Is it possible that a source change landed that is affecting this? 

- when and where did the archive_layout_tests step get so slow? Why is gsutil 10x slower on windows than linux? why is a cloud<->cloud copy inside a single bucket the same speed as a client->cloud copy

- is moving bots from ESX VMs in the golo to GCE VMs actually a good idea? is an n1-standard-8 VM actually equivalent to a 8-core VM on one of our 720s?

- should we add more cores, or more memory, or switch to SSDs, or all of the above? Would one n1-standard-32 VM be better than 4 n1-standard-8 VMs?

- what's the relative impact of goma config changes, or even building w/o goma?

- what about jumbo builds? E.g., I have jumbo builders on chromium.fyi now, but they're not all that useful for evaluating the impact since I don't have comparable non-jumbo builders to compare them to.

- we can now fully cross-compile win chromium on linux hosts. Is that faster than a win-native build? What about running the Linux subsystem on Win10? What about the Linux subsystem + an ext4-formatted filesystem?

Obviously, actually being able to answer any of these questions requires additional work, but having controlled data is probably required for some of them, and would probably make most of the rest easier.

 
Blockedon: 779252
Cc: bpastene@chromium.org
By the way, can we rely on monitored metrics of buildbots to know which resource is not sufficient in each build steps?
https://cs.chromium.org/chromium/infra/go/src/infra/monitoring/sysmon/?sq=package:chromium
Good question; I don't know. 
From https://bugs.chromium.org/p/chromium/issues/detail?id=779252#c19 :

ESX win10 SSD: vm350-m1
ESX win10 Normal Disk: vm348-m1

ESX trusty SSD: vm65-m1
ESX trusty Normal Disk: vm349-m1
Labels: -Pri-3 Pri-1
Owner: martiniss@chromium.org
Status: Assigned (was: Started)
martiniss@, as per my comments in your doc, I'm going to punt this to you to own now.
Status: WontFix (was: Assigned)
I made a bit of progress here; I made some docker containers. I haven't worked on this in a long time. Closing for now, someone could revisit this later potentially.

Sign in to add a comment