New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 618544 link

Starred by 4 users

Issue metadata

Status: Duplicate
Merged: issue 750250
Owner: ----
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: 2017-04-01
OS: Windows
Pri: 2
Type: Bug


Participants' hotlists:
speed-ops-backlog


Sign in to add a comment

Make chromium.perf Win x64 builders tree closers

Project Member Reported by sullivan@chromium.org, Jun 9 2016

Issue description

We'd be happy to move the builders to a different waterfall that funnels into the main sheriff-o-matic page if that makes it easier for everyone. They are release official builders if that helps with considerations.
 
Owner: estaab@chromium.org
Status: Assigned (was: Untriaged)
Let me try to pull some stats from existing builders to see if blamelist size and cycle times are on par before moving this. Next update by 6/15.
Cc: jam@chromium.org phajdan@google.com
+jam

Please check with chrome folks before adding new tree closers.

Comment 3 by jam@chromium.org, Jun 13 2016

Are you referring to https://build.chromium.org/p/chromium.perf/builders/Win%20x64%20Builder?numbuilds=200?

Can you give some background about why you want this to be a tree closer? i.e. what fails on this bot compared to other bots already on the main waterfall. Is it just the official part? There's a long standing bug to make the internal official builder show up on sheriffomatic.

I see that this builder takes 2 hours to cycle. After the code yellow, we tried to make every bot on the main waterfall cycle in less than half an hour.
This is official, non-clobber. We'd really like better coverage for that config. Agreed that the cycle time is long, so that we can get it as close to the released version of Chrome as possible. We are working on sourcing much more hardware, so that although the builds wouldn't get faster, there would be more of them, one for every revision. Not sure if that helps with the cycle time concern.

Comment 5 by jam@chromium.org, Jun 13 2016

(and now I see the email thread about this :) )

Do we need this to be a true closer, or just show up on sheriffomatic? And if we can get the internal official builder to show up on sheriffomatic, do we need this as well?

One last question: how come this fails most of the time? (i.e. the purple runs)
Cc: simonhatch@chromium.org fmea...@chromium.org
Our main goal is to get attention from the right people more quickly when the build fails. For example, you can see in  bug 616946 , it was reported June 2 but the author of the culprit CL was not cc-ed until June 6. It was fixed immediately after the cc, so the problem isn't that people don't help, it's just that we aren't able to get the right people on the problem quickly. If it just showed up on sheriffomatic, would that help?

I think that these builds have more problems than the official builders because they are non-clobber (so that they can build faster, they are of course still slow)

Adding Fadi and Simon, last week's bot health sheriffs, can you explain the cause of the purple runs?
My understanding is that the purple "bots" are bug 618751 (or at least one of them). The waterfall is confusing as it was showing the Win 64 builder as green most of my shift (despite the purple bots) as the one using the bare metal was succeeding.
It's down again in issue 620295
Components: Infra>SheriffOMatic
Components: -Infra
Components: Infra>Sheriffing>SheriffOMatic
Components: -Infra>Sheriffing
Labels: -Pri-3 Pri-2
I see that the builds are only about 10 minutes apart now and the history is very green, which seems not very burdensome for someone to sheriff.

https://build.chromium.org/p/chromium.perf/builders/Win%20x64%20Builder?numbuilds=200

How do perf folks want to proceed with this? Have the more frequent runs made sheriffing easy enough as is or should we get this showing in SoM or make a tree closer?

Apologies if this was resolved elsewhere already, I'm just going through my bug queue at the moment.
I think we should make this a tree closer. 
Components: Infra>Client>Perf
We'd still prefer to make this a tree closer. What ends up happening for us is that a wave of really tough failures come in, and then things are quiet, but they always get bad again and it would help a ton if this config closed the tree.
Owner: jam@chromium.org
jam@, this seems to cycle fast now so are you ok with making this a tree closer? I nominate you as the decision maker for this issue. :)
Labels: Milestone-PostSoMNG
Ping on this. jam@, can we make this a tree closer now that the builder has sufficient capacity?
Labels: -Pri-2 Pri-1

Comment 20 by jam@chromium.org, Aug 12 2016

I'm sorry for missing the pings above (I have a lot of bugs I'm on). In the future if I miss the pings here, please IM me.

It's great that the bot's cycle time is now fast. So to make sure I'm understanding correctly, the main issue is that we have 32 bit official builds on the waterall (https://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Win/builds/9847/steps/generate_build_files/logs/stdio), but we don't have 64 bit right? if so, it seems very reasonable that https://build.chromium.org/p/chromium.perf/builders/Win%20x64%20Builder?numbuilds=200 moves to chromium.chrome master where the 32 bit builder is. This would also be similar to chromium master, which does have 32 & 64 bit (non-official) chromium builds.
Owner: dtu@chromium.org
Thanks, jam!

Assigning to dtu to move the builder to the chromium.chrome master.

Comment 22 by dtu@chromium.org, Aug 12 2016

Cc: no...@chromium.org
We're currently triggering tests on chromium.perf from these builders. If we move masters, then we'd have to trigger cross-master, which means we have to use buildbucket, which doesn't support merging build requests. Is there something we can do about this?

Also, is this bug referring to just Win x64? Seems strange to have one config separate from the rest of the perf builders.

Comment 23 by jam@chromium.org, Aug 12 2016

Another option to avoid triggering cross builder is to just add a win64 builder to chromium.chrome. That would duplicate the perf master one, but we already have duplication for all the other builders on both masters.

Comment 24 by no...@chromium.org, Aug 12 2016

Buildbucket's poor support of build request merging won't affect you: the triggered builds will be scheduled on buildbot. The triggering build won't be able to see build results, but it does not care about them
just like https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.chromium.perf.fyi/master_site_config.py#17
Labels: -Milestone-PostSoMNG Milestone-Perf
Ping - please provide an update to your high priority bug. This bug is stale. Is it really P-1?
NextAction: 2017-01-02
NextAction: 2017-04-01
Owner: martiniss@chromium.org
I think we should just make another builder on chromium.chrome, which builds Windows on 64 bit, and leave the perf one where it is. We'd get coverage, but not have to worry about buildbucket triggering.

Although, maybe we don't have the necessary hardware for that?

Re #24: We can't do that anymore I think; the swarming builders are using buildbucket I think.
Components: Speed>Benchmarks>Waterfall
Labels: -Performance-Waterfall
Labels: -Pri-1 Pri-2
Owner: ----
Status: Available (was: Assigned)
I'm not working on this anymore.
Mergedinto: 750250
Status: Duplicate (was: Available)

Sign in to add a comment