GCE builds are far slower than usual since Jan 31 |
||||||||
Issue descriptionThis has hit moblab-generic-vm-pre-cq worst. See issue 810061 But the conclusion there is that GCE builds are running slower (the BuildPackages stage there took 27% longer, UnitTest 50% longer than average) in the last couple days. Pri-1 because not dealing with this soon will normalize the new average. Pri-0 if other attempts at dealing with its effects on moblab-generic-vm-pre-cq fail us.
,
Feb 8 2018
Any graph/dashboard for "the BuildPackages stage there took 27% longer, UnitTest 50% longer than average"?
,
Feb 8 2018
+jingjing I read those numbers off the beautiful new legoland UI. I don't know where we get that data / what exactly it's comparing against.
,
Feb 8 2018
The data was pulled from CIDB, and it is got from averaging over 30 days for the same stage on the same build config (not builder, so it is less accurate in that regard).
,
Feb 9 2018
Here's some findings. I go through all paladins within 30 days and plot their time cost for buildPackages here: https://colab.corp.google.com/drive/1GekXDQ7S4kHRNWZqpPMvJbvYI_iz-97z (We can have them here https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildHealthDashboard, don't we) Some conclusions: Do all builders get slower? No. e.g. amd64-generic-paladin. Do some of the builders get slower? Absolutely yes. Most of our effective paladins go slower, including pre-cqs. When did them get slower? About 1.31.2018. Move it to Chase-Pending for more ideas.
,
Feb 12 2018
,
Feb 26 2018
,
May 15 2018
,
May 16 2018
How are we deciding about these between Infra>Client>ChromeOS>Build and Infra>Client>ChromeOS>CI ?
,
Jul 13
,
Aug 31
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by pprabhu@chromium.org
, Feb 7 2018