New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 8 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug

Blocked on:
issue 407401
issue 407402
issue 414906
issue 701720
issue 704080
issue 821705
issue 824080

Blocking:
issue 760305
issue 407381



Sign in to add a comment
link

Issue 407399: [META] Remove LKGR

Reported by eseidel@chromium.org, Aug 25 2014 Project Member

Issue description

As discussed in:
https://groups.google.com/a/chromium.org/forum/#!searchin/chromium-dev/lkgr/chromium-dev/e4lQRKYbRJY/zNebDVTEuwcJ

We're interested in removing LKGR for Chromium.  This is the bug to track that work.  Please relate any issues blocking this removal.
 

Comment 1 by eseidel@chromium.org, Aug 25 2014

Blockedon: chromium:407401

Comment 2 by eseidel@chromium.org, Aug 25 2014

Blockedon: chromium:407402

Comment 3 by eseidel@chromium.org, Aug 25 2014

Blocking: chromium:407381

Comment 4 by ojan@chromium.org, Aug 25 2014

Cc: c...@chromium.org
I had a recent discussion with someone who actually uses LKGR locally that made me wonder if we should take a different approach. Keep LKGR, but have it mean something similar to what it means for the canaries. Instead of the last green revision, it grabs the best revision of the last 40. And it updates every 30 minutes.

This bot would be relatively easy to maintain I think and would not require anyone to change their workflows.

Comment 5 by infe...@chromium.org, Aug 26 2014

I totally agree with Ojan in #4 and also gives us consistency on when we can expect build.

Comment 6 by eseidel@chromium.org, Aug 26 2014

That's fine.  But what is "best" defined as?  Across which bots does it check?  Presumably we could build a generic infrastructure for this and allow people to consume it with different sets of bots.  However it's not clear how you compare a build with 100 layouttest failures vs. one with 1 ASAN failure, etc.  I suspect those comparisons differ between use-cases.

My interest in this bug is about freeing our infrastructure team to forget some of our past mistakes and free their time to focus on better future projects.  With so few consumers of LKGR and so much code/infrastructure to maintain it, it seems like a very poor trade of our capacity and infrastructure-engineer resources.

Comment 7 by dpranke@chromium.org, Sep 16 2014

Blockedon: chromium:414906

Comment 8 by ojan@chromium.org, Oct 17 2014

Once  issue 407402  is resolved, we can change the lkgr waterfall to actually pull lkcr and then we can kill lkgr. We can't kill the lkgr waterfall, because those bots actually do things. But, we won't need to maintain lkgr and deal with it getting behind.

Comment 9 by rnimmagadda@chromium.org, Oct 13 2015

Labels: TE-NeedsFurtherTriage

Comment 10 by dpranke@chromium.org, Mar 18 2016

Cc: -c...@chromium.org infe...@chromium.org dtu@chromium.org iannucci@chromium.org mar...@chromium.org nednguyen@chromium.org aneeshm@chromium.org phajdan.jr@chromium.org aga...@chromium.org
Labels: -TE-NeedsFurtherTriage Te-NeedsFurtherTriage
Owner: dpranke@chromium.org
Status: Assigned (was: Unconfirmed)
I think we need to finally get around to doing this as part of the recipe migration and the GN migration.

It looks like there's three classes of bots:

- bots whose sole purpose is to build at lkgr. We should just remove these.
- the telemetry upload harness bot. We should just move this to trunk ( bug 407402 )
- the clusterfuzz bots. We should move this to trunk and write a recipe
  that only uploads the builds to the bucket after we run and pass some list of
  tests ( bug 407401 )

Comment 11 by infe...@chromium.org, Mar 18 2016

Cc: och...@chromium.org mbarbe...@chromium.org
Sounds good, thanks Dirk. Yes each bot can individually run the tests and hence wont be blocked on a revision.

Comment 12 by aneeshm@chromium.org, Mar 18 2016

This sounds good to me.

I started converting this waterfall to recipes some days ago, but left it on the backburner after a preliminary inspection because I couldn't replicate some builders' steps locally.

Which bots are going to be deleted? If I know this, I can skip them when converting the rest to recipes. (The recipe can be used even after the bots switch masters, so I'm planning to go ahead with the conversion.)

Comment 13 by dpranke@chromium.org, Mar 18 2016

@aneeshm - I'm deleting the bots in https://codereview.chromium.org/1813123002/ .

I don't know that we'll want to land your conversion, but it would probably be at least helpful to me in converting things to proper recipes (whether we use the chromium recipe as a basis or, more likely, write new ones), so make sure you keep me involved in whatever you end up doing.

Comment 14 by bugdroid1@chromium.org, Mar 18 2016

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/ffe53a8970e65115b9d1721b46930f157302f0ee

commit ffe53a8970e65115b9d1721b46930f157302f0ee
Author: dpranke@chromium.org <dpranke@chromium.org>
Date: Fri Mar 18 21:12:52 2016

Remove non-clusterfuzz, non-telemetry LKGR bots.

We don't need and don't want to support them any more.

R=maruel@chromium.org
BUG= 407399 

Review URL: https://codereview.chromium.org/1813123002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299390 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/ffe53a8970e65115b9d1721b46930f157302f0ee/masters/master.chromium.lkgr/master_lkgr_cfg.py
[modify] https://crrev.com/ffe53a8970e65115b9d1721b46930f157302f0ee/masters/master.chromium.lkgr/slaves.cfg

Comment 15 by dpranke@chromium.org, Jun 23 2016

Labels: -Pri-2 Pri-3

Comment 16 by benhenry@chromium.org, Jun 23 2016

Components: -Infra Infra>Client>Chrome

Comment 17 by dpranke@chromium.org, Aug 30 2016

Labels: -TE-NeedsFurtherTriage TE-NeedsFurthertriage OS-All

Comment 18 by dpranke@chromium.org, Jan 15 2017

Labels: -Pri-3 Pri-2

Comment 19 by dpranke@chromium.org, Feb 25 2017

Cc: estaab@chromium.org
Labels: -Pri-2 Pri-3
Owner: ----
Status: Available (was: Assigned)
I'm not sure when I'll get around to this or what the priority of this really is, given current situations :(.

estaab@ - this is something to be aware of for the migration off of buildbot, too.

Comment 20 by estaab@chromium.org, Feb 27 2017

Looks like thakis converted these to recipes 9 months ago. Is that why I may need to be aware or is it something else?

Comment 21 by machenb...@chromium.org, Mar 17 2017

Blockedon: 701720

Comment 22 by machenb...@chromium.org, Mar 22 2017

Blockedon: 704080

Comment 23 by machenb...@chromium.org, Mar 30 2017

 Issue 706617  has been merged into this issue.

Comment 24 by efoo@chromium.org, Mar 30 2017

Owner: dpranke@chromium.org

Comment 25 by dpranke@chromium.org, Apr 16 2017

Owner: ----
okay, I give up, I don't know when I'm going to get to this :).

Comment 26 by nedngu...@google.com, Apr 16 2017

Cc: -nednguyen@chromium.org

Comment 27 by benhenry@chromium.org, Apr 26 2017

Labels: cit-pm-32

Comment 28 by sebmarchand@chromium.org, May 25 2017

All the bots on the chromium.lkgr waterfall are now using lkcr and nothing seem to use the lkgr recipe module (according to https://cs.chromium.org/search/?q=chromium_lkgr+package:%5Echromium$&type=cs) 

Is it fixed then? Should we just remove the lkgr recipe module or is there still some projects that don't use recipes?

Comment 29 by machenb...@chromium.org, Aug 2 2017

No, we rather need to deprecate the lkcr ref (not the concept) due to the hacky way it runs on a master right now. Since lkgr is effectively unused, we could switch its configs to the same as lkcr, then switch all bots to use lkgr again, then deprecate lkcr and remove the hacky code.

Another option would be to get all chromium bots off lkcr and lkgr - and deprecate the whole thing. This requires the blocking bugs to get fixed.

Comment 30 by machenb...@chromium.org, Mar 14 2018

Blockedon: 821705

Comment 31 by machenb...@chromium.org, Mar 14 2018

Labels: -TE-NeedsFurtherTriage -Pri-3 Pri-1
Owner: machenb...@chromium.org
Status: Assigned (was: Available)
Now lkcr-update is broken (see new blocker bug), which directly affects V8 trybots and clusterfuzz builders.

I'll raise prio and try to drive this. The title isn't really fitting though.

My master plan:
-> LKGR is not updated since a year: https://chromium.googlesource.com/chromium/src/+/refs/heads/lkgr - so I assume nobody relies on it anymore.
1. Migrate the lkCr configs to lkGr
-> Then lkgr and lkcr should be the same
2. Migrate all clusterfuzz builders and V8 infra back to use lkgr
3. Rip out lkcr logic from the scripts
4. Migrate the script off from master1 to a bot (like V8 and webrtc already do)
5. Use the already built-in direct ref-update in lkgr-finder recipe
6. Remove lkgr-tag-pusher recipe and bots
7. Remove lkgr in status-app

Comment 32 by machenb...@chromium.org, Mar 14 2018

Cc: tandrii@chromium.org dpranke@chromium.org
@tandrii, iannucci, dpranke: I'm CC'ing you on all CLs for carrying out the plan above. Sergiy will do the main reviews. Feel free to ignore them, it's just FYI.

Comment 33 by infe...@chromium.org, Mar 14 2018

Cc: jbudorick@chromium.org
regular clusterfuzz builders on
https://ci.chromium.org/p/chromium/g/chromium.lkgr/console are actually on master, not on lkgr or lkcr. the name of the waterfall is obsolete.

Are you talking about v8 clusterfuzz builders, those should be on master as well right ?

Comment 34 by machenb...@chromium.org, Mar 14 2018

Oh, you're right:
https://cs.chromium.org/chromium/build/masters/master.chromium.lkgr/master_lkgr_cfg.py?l=156

Seems to have changed recently. In this case, V8 is the sole user of Chromium's lkgr now. Seems to be reasonable for us to drive this effort here.

Comment 35 by machenb...@chromium.org, Mar 19 2018

Blocking: 760305

Comment 36 by machenb...@chromium.org, Mar 21 2018

Blockedon: 824080

Comment 37 by machenb...@chromium.org, Apr 9 2018

Status: Fixed (was: Assigned)
Marking this as fixed. We didn't quite "remove" lkgr, but we did the following:

- Removed lkgr scripts from master1, unblocking buildbot deprecation
- Removed status-app dependencies in lkgr scripts, unblocking work for status in sheriff-o-matic
- All users of lkgr have their lkgr-finder instances running on infra slaves
- Tag pushing is now integrated in lkgr-finder recipe
- Documentation was updated

Sign in to add a comment