New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 818877 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: ----



Sign in to add a comment

"Out of capacity" on Windows perf bots when Microsoft pushes an update

Project Member Reported by eakuefner@chromium.org, Mar 5 2018

Issue description

Link to buildbot status page: https://ci.chromium.org/buildbot/chromium.perf/Win%2010%20Perf/

Looks like the swarming bot may be down?

 
Owner: eakuefner@chromium.org
Status: Assigned (was: Untriaged)
The bot isn't down: https://chromium-swarm.appspot.com/bot?id=build133-m1&sort_stats=total%3Adesc is running just fine.

The bot's windows OS was updated though. Looks like we need to change the dimensions source side to be updated to that.

https://cs.chromium.org/chromium/src/tools/perf/core/perf_data_generator.py?l=311 needs to be changed to use "Windows-10-16299.248"

Ethan, can you do that?
I'm doing this for  crbug.com/819780  so I'll just include this one.
Owner: simonhatch@chromium.org
Project Member

Comment 4 by bugdroid1@chromium.org, Mar 8 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/56fd063683018627c56f813874e1ba28661e8baa

commit 56fd063683018627c56f813874e1ba28661e8baa
Author: Simon <simonhatch@chromium.org>
Date: Thu Mar 08 18:14:46 2018

Change dimensions for win high-DPI

They seem to have changed on the bot, changing here to match.

Bug:  819780 ,  818877 
Change-Id: I305892716bcb337df71a1f1e2d0937c0db8541b6
Reviewed-on: https://chromium-review.googlesource.com/953379
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Commit-Queue: Simon Hatch <simonhatch@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541839}
[modify] https://crrev.com/56fd063683018627c56f813874e1ba28661e8baa/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/56fd063683018627c56f813874e1ba28661e8baa/tools/perf/core/perf_data_generator.py

Status: Fixed (was: Assigned)
Cc: simonhatch@chromium.org
Owner: charliea@chromium.org
Status: Assigned (was: Fixed)
Reopening this because we still need to do this for Win 10 Perf. I'll go ahead and do that.
Cc: nednguyen@chromium.org
Summary: "Out of capacity" on Windows perf bots when Microsoft pushes an update (was: Purple Bot on chromium.perf Win10 Perf)
+nednguyen@ FYI

I talked some to bevc@ about this. It looks like the problem is that sometimes, even if Windows Update is disabled, Microsoft pushes OS updates, which change the version number. Because we use that version number as one of the criteria in our swarming target, our benchmarks can no longer find a machine to run on and we start getting out of capacity errors.

You can see this clearly on build117-b1: it was at version Windows-10-16299.248 12 hours or so ago (http://bit.ly/2FKUmLJ, search for Windows-10-16299.248) but now seems to be at Windows-10-16299.309 (http://bit.ly/2DsRcud, search for Windows-10-16299.309) and experiencing the same "out of capacity" errors that the other bots were previously experiencing.

Ned suggested that we might not want to target based on the OS version at all (it's redundant - we already specify the bots that we want to target). However, we need to make sure that not specifying the target OS version doesn't have some confusing results. Apparently if you omit the GPU targeting criteria, for example, it means that you need to run on a machine with *no* GPU, rather than being able to run on a machine with *any* GPU.
Stupid question, since you specify the device id's in perf_data_generator, why do you need any of those other dimensions?
#8: we don't need most of other dimensions, except for gpu dimension (see 36 --> 40 in https://bugs.chromium.org/p/chromium/issues/detail?id=717744#c36)
Cc: eyaich@chromium.org
Components: Speed>Benchmarks>Waterfall
Project Member

Comment 11 by bugdroid1@chromium.org, Mar 15 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7650de1d462a76c0e41d924177caa823fc78cebf

commit 7650de1d462a76c0e41d924177caa823fc78cebf
Author: Charlie Andrews <charliea@chromium.org>
Date: Thu Mar 15 17:33:35 2018

Update the Windows target OS version strings to reflect system update

Windows silently updated and, because we target the exact version, the
benchmarks running on these machines started to fail with "out of
capacity" errors.

The long term solution here probably has to do with removing the OS
version from the targeting criteria altogether given the fact that we
already target the exact machines, but in the short term, this should
get our bots greener.

TBR=nednguyen@google.com

Bug:  818877 
Change-Id: I95ecfc8b3381eaa45c0d69d2b0caa747d1b1f9eb
Reviewed-on: https://chromium-review.googlesource.com/963146
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#543427}
[modify] https://crrev.com/7650de1d462a76c0e41d924177caa823fc78cebf/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/7650de1d462a76c0e41d924177caa823fc78cebf/tools/perf/core/perf_data_generator.py

Status: Fixed (was: Assigned)
Nice work, Charlie!

Sign in to add a comment