New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 8 users

Issue metadata

Status: Fixed
Closed: Apr 2018
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug

Blocked on:
issue 407401
issue 407402
issue 414906
issue 701720
issue 704080
issue 821705
issue 824080

issue 760305
issue 407381

Sign in to add a comment

Issue 407399: [META] Remove LKGR

Reported by, Aug 25 2014 Project Member

Issue description

As discussed in:!searchin/chromium-dev/lkgr/chromium-dev/e4lQRKYbRJY/zNebDVTEuwcJ

We're interested in removing LKGR for Chromium.  This is the bug to track that work.  Please relate any issues blocking this removal.

Comment 1 by, Aug 25 2014

Blockedon: chromium:407401

Comment 2 by, Aug 25 2014

Blockedon: chromium:407402

Comment 3 by, Aug 25 2014

Blocking: chromium:407381

Comment 4 by, Aug 25 2014

I had a recent discussion with someone who actually uses LKGR locally that made me wonder if we should take a different approach. Keep LKGR, but have it mean something similar to what it means for the canaries. Instead of the last green revision, it grabs the best revision of the last 40. And it updates every 30 minutes.

This bot would be relatively easy to maintain I think and would not require anyone to change their workflows.

Comment 5 by, Aug 26 2014

I totally agree with Ojan in #4 and also gives us consistency on when we can expect build.

Comment 6 by, Aug 26 2014

That's fine.  But what is "best" defined as?  Across which bots does it check?  Presumably we could build a generic infrastructure for this and allow people to consume it with different sets of bots.  However it's not clear how you compare a build with 100 layouttest failures vs. one with 1 ASAN failure, etc.  I suspect those comparisons differ between use-cases.

My interest in this bug is about freeing our infrastructure team to forget some of our past mistakes and free their time to focus on better future projects.  With so few consumers of LKGR and so much code/infrastructure to maintain it, it seems like a very poor trade of our capacity and infrastructure-engineer resources.

Comment 7 by, Sep 16 2014

Blockedon: chromium:414906

Comment 8 by, Oct 17 2014

Once  issue 407402  is resolved, we can change the lkgr waterfall to actually pull lkcr and then we can kill lkgr. We can't kill the lkgr waterfall, because those bots actually do things. But, we won't need to maintain lkgr and deal with it getting behind.

Comment 9 by, Oct 13 2015

Labels: TE-NeedsFurtherTriage

Comment 10 by, Mar 18 2016

Labels: -TE-NeedsFurtherTriage Te-NeedsFurtherTriage
Status: Assigned (was: Unconfirmed)
I think we need to finally get around to doing this as part of the recipe migration and the GN migration.

It looks like there's three classes of bots:

- bots whose sole purpose is to build at lkgr. We should just remove these.
- the telemetry upload harness bot. We should just move this to trunk ( bug 407402 )
- the clusterfuzz bots. We should move this to trunk and write a recipe
  that only uploads the builds to the bucket after we run and pass some list of
  tests ( bug 407401 )

Comment 11 by, Mar 18 2016

Sounds good, thanks Dirk. Yes each bot can individually run the tests and hence wont be blocked on a revision.

Comment 12 by, Mar 18 2016

This sounds good to me.

I started converting this waterfall to recipes some days ago, but left it on the backburner after a preliminary inspection because I couldn't replicate some builders' steps locally.

Which bots are going to be deleted? If I know this, I can skip them when converting the rest to recipes. (The recipe can be used even after the bots switch masters, so I'm planning to go ahead with the conversion.)

Comment 13 by, Mar 18 2016

@aneeshm - I'm deleting the bots in .

I don't know that we'll want to land your conversion, but it would probably be at least helpful to me in converting things to proper recipes (whether we use the chromium recipe as a basis or, more likely, write new ones), so make sure you keep me involved in whatever you end up doing.

Comment 14 by, Mar 18 2016

Project Member
The following revision refers to this bug:

commit ffe53a8970e65115b9d1721b46930f157302f0ee
Author: <>
Date: Fri Mar 18 21:12:52 2016

Remove non-clusterfuzz, non-telemetry LKGR bots.

We don't need and don't want to support them any more.
BUG= 407399 

Review URL:

git-svn-id: svn:// 0039d316-1c4b-4281-b951-d872f2087c98


Comment 15 by, Jun 23 2016

Labels: -Pri-2 Pri-3

Comment 16 by, Jun 23 2016

Components: -Infra Infra>Client>Chrome

Comment 17 by, Aug 30 2016

Labels: -TE-NeedsFurtherTriage TE-NeedsFurthertriage OS-All

Comment 18 by, Jan 15 2017

Labels: -Pri-3 Pri-2

Comment 19 by, Feb 25 2017

Labels: -Pri-2 Pri-3
Owner: ----
Status: Available (was: Assigned)
I'm not sure when I'll get around to this or what the priority of this really is, given current situations :(.

estaab@ - this is something to be aware of for the migration off of buildbot, too.

Comment 20 by, Feb 27 2017

Looks like thakis converted these to recipes 9 months ago. Is that why I may need to be aware or is it something else?

Comment 21 by, Mar 17 2017

Blockedon: 701720

Comment 22 by, Mar 22 2017

Blockedon: 704080

Comment 23 by, Mar 30 2017

 Issue 706617  has been merged into this issue.

Comment 24 by, Mar 30 2017


Comment 25 by, Apr 16 2017

Owner: ----
okay, I give up, I don't know when I'm going to get to this :).

Comment 26 by, Apr 16 2017


Comment 27 by, Apr 26 2017

Labels: cit-pm-32

Comment 28 by, May 25 2017

All the bots on the chromium.lkgr waterfall are now using lkcr and nothing seem to use the lkgr recipe module (according to$&type=cs) 

Is it fixed then? Should we just remove the lkgr recipe module or is there still some projects that don't use recipes?

Comment 29 by, Aug 2 2017

No, we rather need to deprecate the lkcr ref (not the concept) due to the hacky way it runs on a master right now. Since lkgr is effectively unused, we could switch its configs to the same as lkcr, then switch all bots to use lkgr again, then deprecate lkcr and remove the hacky code.

Another option would be to get all chromium bots off lkcr and lkgr - and deprecate the whole thing. This requires the blocking bugs to get fixed.

Comment 30 by, Mar 14 2018

Blockedon: 821705

Comment 31 by, Mar 14 2018

Labels: -TE-NeedsFurtherTriage -Pri-3 Pri-1
Status: Assigned (was: Available)
Now lkcr-update is broken (see new blocker bug), which directly affects V8 trybots and clusterfuzz builders.

I'll raise prio and try to drive this. The title isn't really fitting though.

My master plan:
-> LKGR is not updated since a year: - so I assume nobody relies on it anymore.
1. Migrate the lkCr configs to lkGr
-> Then lkgr and lkcr should be the same
2. Migrate all clusterfuzz builders and V8 infra back to use lkgr
3. Rip out lkcr logic from the scripts
4. Migrate the script off from master1 to a bot (like V8 and webrtc already do)
5. Use the already built-in direct ref-update in lkgr-finder recipe
6. Remove lkgr-tag-pusher recipe and bots
7. Remove lkgr in status-app

Comment 32 by, Mar 14 2018

@tandrii, iannucci, dpranke: I'm CC'ing you on all CLs for carrying out the plan above. Sergiy will do the main reviews. Feel free to ignore them, it's just FYI.

Comment 33 by, Mar 14 2018

regular clusterfuzz builders on are actually on master, not on lkgr or lkcr. the name of the waterfall is obsolete.

Are you talking about v8 clusterfuzz builders, those should be on master as well right ?

Comment 34 by, Mar 14 2018

Oh, you're right:

Seems to have changed recently. In this case, V8 is the sole user of Chromium's lkgr now. Seems to be reasonable for us to drive this effort here.

Comment 35 by, Mar 19 2018

Blocking: 760305

Comment 36 by, Mar 21 2018

Blockedon: 824080

Comment 37 by, Apr 9 2018

Status: Fixed (was: Assigned)
Marking this as fixed. We didn't quite "remove" lkgr, but we did the following:

- Removed lkgr scripts from master1, unblocking buildbot deprecation
- Removed status-app dependencies in lkgr scripts, unblocking work for status in sheriff-o-matic
- All users of lkgr have their lkgr-finder instances running on infra slaves
- Tag pushing is now integrated in lkgr-finder recipe
- Documentation was updated

Sign in to add a comment