New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 407399 link

Starred by 8 users

Issue metadata

Status: Fixed
OOO until December 17
Closed: Apr 2018
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug

Blocked on:
issue 407401
issue 407402
issue 414906
issue 701720
issue 704080
issue 821705
issue 824080

issue 760305
issue 407381

Sign in to add a comment

[META] Remove LKGR

Reported by, Aug 25 2014

Issue description

As discussed in:!searchin/chromium-dev/lkgr/chromium-dev/e4lQRKYbRJY/zNebDVTEuwcJ

We're interested in removing LKGR for Chromium.  This is the bug to track that work.  Please relate any issues blocking this removal.
Blockedon: chromium:407401
Blockedon: chromium:407402
Blocking: chromium:407381

Comment 4 by, Aug 25 2014

I had a recent discussion with someone who actually uses LKGR locally that made me wonder if we should take a different approach. Keep LKGR, but have it mean something similar to what it means for the canaries. Instead of the last green revision, it grabs the best revision of the last 40. And it updates every 30 minutes.

This bot would be relatively easy to maintain I think and would not require anyone to change their workflows.
I totally agree with Ojan in #4 and also gives us consistency on when we can expect build. 
That's fine.  But what is "best" defined as?  Across which bots does it check?  Presumably we could build a generic infrastructure for this and allow people to consume it with different sets of bots.  However it's not clear how you compare a build with 100 layouttest failures vs. one with 1 ASAN failure, etc.  I suspect those comparisons differ between use-cases.

My interest in this bug is about freeing our infrastructure team to forget some of our past mistakes and free their time to focus on better future projects.  With so few consumers of LKGR and so much code/infrastructure to maintain it, it seems like a very poor trade of our capacity and infrastructure-engineer resources.
Blockedon: chromium:414906

Comment 8 by, Oct 17 2014

Once  issue 407402  is resolved, we can change the lkgr waterfall to actually pull lkcr and then we can kill lkgr. We can't kill the lkgr waterfall, because those bots actually do things. But, we won't need to maintain lkgr and deal with it getting behind.
Labels: TE-NeedsFurtherTriage
Labels: -TE-NeedsFurtherTriage Te-NeedsFurtherTriage
Status: Assigned (was: Unconfirmed)
I think we need to finally get around to doing this as part of the recipe migration and the GN migration.

It looks like there's three classes of bots:

- bots whose sole purpose is to build at lkgr. We should just remove these.
- the telemetry upload harness bot. We should just move this to trunk ( bug 407402 )
- the clusterfuzz bots. We should move this to trunk and write a recipe
  that only uploads the builds to the bucket after we run and pass some list of
  tests ( bug 407401 )
Sounds good, thanks Dirk. Yes each bot can individually run the tests and hence wont be blocked on a revision.
This sounds good to me.

I started converting this waterfall to recipes some days ago, but left it on the backburner after a preliminary inspection because I couldn't replicate some builders' steps locally.

Which bots are going to be deleted? If I know this, I can skip them when converting the rest to recipes. (The recipe can be used even after the bots switch masters, so I'm planning to go ahead with the conversion.)
@aneeshm - I'm deleting the bots in .

I don't know that we'll want to land your conversion, but it would probably be at least helpful to me in converting things to proper recipes (whether we use the chromium recipe as a basis or, more likely, write new ones), so make sure you keep me involved in whatever you end up doing.
Project Member

Comment 14 by, Mar 18 2016

The following revision refers to this bug:

commit ffe53a8970e65115b9d1721b46930f157302f0ee
Author: <>
Date: Fri Mar 18 21:12:52 2016

Remove non-clusterfuzz, non-telemetry LKGR bots.

We don't need and don't want to support them any more.
BUG= 407399 

Review URL:

git-svn-id: svn:// 0039d316-1c4b-4281-b951-d872f2087c98


Labels: -Pri-2 Pri-3
Components: -Infra Infra>Client>Chrome
Labels: -TE-NeedsFurtherTriage TE-NeedsFurthertriage OS-All
Labels: -Pri-3 Pri-2
Labels: -Pri-2 Pri-3
Owner: ----
Status: Available (was: Assigned)
I'm not sure when I'll get around to this or what the priority of this really is, given current situations :(.

estaab@ - this is something to be aware of for the migration off of buildbot, too.
Looks like thakis converted these to recipes 9 months ago. Is that why I may need to be aware or is it something else?
Blockedon: 701720
Blockedon: 704080
 Issue 706617  has been merged into this issue.

Comment 24 by, Mar 30 2017

Owner: ----
okay, I give up, I don't know when I'm going to get to this :).
Labels: cit-pm-32
All the bots on the chromium.lkgr waterfall are now using lkcr and nothing seem to use the lkgr recipe module (according to$&type=cs) 

Is it fixed then? Should we just remove the lkgr recipe module or is there still some projects that don't use recipes?
No, we rather need to deprecate the lkcr ref (not the concept) due to the hacky way it runs on a master right now. Since lkgr is effectively unused, we could switch its configs to the same as lkcr, then switch all bots to use lkgr again, then deprecate lkcr and remove the hacky code.

Another option would be to get all chromium bots off lkcr and lkgr - and deprecate the whole thing. This requires the blocking bugs to get fixed.
Blockedon: 821705
Labels: -TE-NeedsFurtherTriage -Pri-3 Pri-1
Status: Assigned (was: Available)
Now lkcr-update is broken (see new blocker bug), which directly affects V8 trybots and clusterfuzz builders.

I'll raise prio and try to drive this. The title isn't really fitting though.

My master plan:
-> LKGR is not updated since a year: - so I assume nobody relies on it anymore.
1. Migrate the lkCr configs to lkGr
-> Then lkgr and lkcr should be the same
2. Migrate all clusterfuzz builders and V8 infra back to use lkgr
3. Rip out lkcr logic from the scripts
4. Migrate the script off from master1 to a bot (like V8 and webrtc already do)
5. Use the already built-in direct ref-update in lkgr-finder recipe
6. Remove lkgr-tag-pusher recipe and bots
7. Remove lkgr in status-app
@tandrii, iannucci, dpranke: I'm CC'ing you on all CLs for carrying out the plan above. Sergiy will do the main reviews. Feel free to ignore them, it's just FYI.
regular clusterfuzz builders on are actually on master, not on lkgr or lkcr. the name of the waterfall is obsolete.

Are you talking about v8 clusterfuzz builders, those should be on master as well right ?
Oh, you're right:

Seems to have changed recently. In this case, V8 is the sole user of Chromium's lkgr now. Seems to be reasonable for us to drive this effort here.
Blocking: 760305
Blockedon: 824080
Status: Fixed (was: Assigned)
Marking this as fixed. We didn't quite "remove" lkgr, but we did the following:

- Removed lkgr scripts from master1, unblocking buildbot deprecation
- Removed status-app dependencies in lkgr scripts, unblocking work for status in sheriff-o-matic
- All users of lkgr have their lkgr-finder instances running on infra slaves
- Tag pushing is now integrated in lkgr-finder recipe
- Documentation was updated

Sign in to add a comment