New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 762070 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 765795

Blocking:
issue 748099



Sign in to add a comment

Revive db_cleanup

Project Member Reported by pho...@chromium.org, Sep 5 2017

Issue description

Let's resurrect the db_cleanup script. Fang commented on http://crosreview.com/c/329250 that it may be possible to revive it. I imagine we could just try reducing the db_cleanup script's batch size and throttling its QPS to make sure we don't overload the master db.

Chase-Pending & Pri-1 because it prevents P0 outages due to db slowness.

 
Blocking: 748099
Labels: -Chase-Pending Chase

Comment 3 by dchan@chromium.org, Sep 12 2017

What does the db_cleanup script removed ? What impact will this have from GE perspective ?
GE will no longer be able to get test results (only test results, does not affect build status / build logs / or even test logs) older than 3 months.

This is not a new proposal. iirc, we already have a stated test horizon of 3 months. This is just re-instating cleanup that we were not doing for other reasons.

Comment 5 by dshi@chromium.org, Sep 12 2017

Re #3, #4

GE is not affected. The script is trying to cleanup master scheduler database, not TKO database.

GE gets data from Sponge, so it's not affected.


Comment 6 by pho...@chromium.org, Sep 12 2017

Status: Started (was: Available)
Project Member

Comment 7 by bugdroid1@chromium.org, Sep 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4d11bc3a925dc6204c14142a4a862c645d1f7f31

commit 4d11bc3a925dc6204c14142a4a862c645d1f7f31
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Sep 13 22:12:01 2017

[autotest] db_cleanup: Add a "dry_run" flag

Add a dry_run option to db_cleanup for sanity.

BUG= chromium:762070 
TEST=None

Change-Id: I4e073fa98e6904a6c8e50d80279949c536d2e56f
Reviewed-on: https://chromium-review.googlesource.com/657239
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/4d11bc3a925dc6204c14142a4a862c645d1f7f31/contrib/db_cleanup.py

Project Member

Comment 8 by bugdroid1@chromium.org, Sep 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/9d6c6dce76d8491b2eb3623521e5b4e7737dbb5e

commit 9d6c6dce76d8491b2eb3623521e5b4e7737dbb5e
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Sep 13 22:12:02 2017

[autotest] db_cleanup: Rate limit deletions

Add a rate limiting flag to db_cleanup. This
makes it safer to run on production databases,
because we don't want to overload
the master database.

BUG= chromium:762070 
TEST=None

Change-Id: Ie9ed29c0a3ec11e9eabb64e32b88937db4b252f7
Reviewed-on: https://chromium-review.googlesource.com/657240
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/9d6c6dce76d8491b2eb3623521e5b4e7737dbb5e/contrib/db_cleanup.py

Project Member

Comment 9 by bugdroid1@chromium.org, Sep 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/45c53424e391613b8995993dc4baea04f4d49444

commit 45c53424e391613b8995993dc4baea04f4d49444
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Sep 13 22:12:02 2017

[autotest] db_cleanup: Enable on master

Currently, db_cleanup exits if it is not
running on a shard. Get rid of that logic.

BUG= chromium:762070 
TEST=None

Change-Id: I86c02ea34445e858c70d2f31b321d46e597eb2bf
Reviewed-on: https://chromium-review.googlesource.com/657241
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/45c53424e391613b8995993dc4baea04f4d49444/contrib/db_cleanup.py

Blockedon: 765795
Status: Fixed (was: Started)
Project Member

Comment 12 by bugdroid1@chromium.org, Feb 12 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/60b0c9e25addbb8d73dab2e52cde849124318adc

commit 60b0c9e25addbb8d73dab2e52cde849124318adc
Author: Paul Hobbs <phobbs@google.com>
Date: Mon Feb 12 18:20:11 2018

Project Member

Comment 13 by bugdroid1@chromium.org, Feb 12 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/480244e1916470885865c198e86281852c0b2ff9

commit 480244e1916470885865c198e86281852c0b2ff9
Author: Paul Hobbs <phobbs@google.com>
Date: Mon Feb 12 18:46:51 2018

Sign in to add a comment