New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 620572 link

Starred by 1 user
Status: WontFix
Owner: ----
Closed: Jun 2018
Cc:
vadimsh@chromium.org
█ dsansome@chromium.org
friedman@chromium.org
no...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature



Sign in to add a comment

Switch buildbot masters to sqlite from postgresql

Project Member Reported by dsansome@chromium.org, Jun 16 2016 
The database is used to store pending builds.

Having an external database as a dependency is fragile - it's not monitored and configuration and setup is done manually.

sqlite should easily be able to handle the load from one single-threaded process.

From postmortem: https://docs.google.com/document/d/187LmN11xtMc5EzyovSZNlKYjWklP0x5Ywbpl3DRbJ0c/edit

Comment 1 by dsansome@chromium.org, Jun 16 2016

Components: Infra>Platform>Buildbot
Project Member

Comment 4 by bugdroid1@chromium.org, Jul 1 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/80fbecca8758922ccf890cb51c2d99e1b1b78d48

commit 80fbecca8758922ccf890cb51c2d99e1b1b78d48
Author: dsansome <dsansome@chromium.org>
Date: Fri Jul 01 01:43:13 2016

Don't require a .dbconfig file on any masters.

After this CL lands we can start switching them back to sqlite one at a time.

BUG= 620572 

Review-Url: https://codereview.chromium.org/2094613002

[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.chrome/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.chromedriver/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.gatekeeper/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.gpu.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.gpu/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.infra.cron/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.infra/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.linux/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.lkgr/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.mac/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.memory.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.memory/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.perf.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.perf/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.swarm/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.webkit/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.webrtc.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.webrtc/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium.win/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromium/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromiumos.chromium/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromiumos.tryserver/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.chromiumos/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.dart.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.dart.packages/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.dart/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.drmemory/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.dynamorio/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.fletch/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.libvpx/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.libyuv/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.mojo/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.nacl.ports/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.nacl.sdk/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.nacl.toolchain/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.nacl/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.v8.branches/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.v8.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.v8.ports/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.v8/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.webrtc.fyi/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.webrtc.perf/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.client.webrtc/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.blink/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.chromium.angle/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.chromium.linux/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.chromium.mac/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.chromium.perf/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.chromium.win/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.client.mojo/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.libyuv/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.nacl/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.v8/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/masters/master.tryserver.webrtc/master.cfg
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/scripts/master/master_gen.py
[modify] https://crrev.com/80fbecca8758922ccf890cb51c2d99e1b1b78d48/scripts/master/skia/skia_master_utils.py
Project Member

Comment 5 by bugdroid1@chromium.org, Jul 1 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8c7a5b159b444e0193f9b2d8a094d352c226e7d1

commit 8c7a5b159b444e0193f9b2d8a094d352c226e7d1
Author: dsansome <dsansome@chromium.org>
Date: Fri Jul 01 04:01:57 2016

Revert buildbot 361b81b43fc7793b66facd4bac7abe0adb30f3d5.

This commit was only added in the 0.8.4 branch, and was never made to master.

The bug only happens when starting buildbot with "make start", but we start it
directly with twistd in Chrome: http://trac.buildbot.net/ticket/1992

BUG= 620572 

Review-Url: https://codereview.chromium.org/2108553002

[modify] https://crrev.com/8c7a5b159b444e0193f9b2d8a094d352c226e7d1/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 6 by bugdroid1@chromium.org, Jul 1 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8c7a5b159b444e0193f9b2d8a094d352c226e7d1

commit 8c7a5b159b444e0193f9b2d8a094d352c226e7d1
Author: dsansome <dsansome@chromium.org>
Date: Fri Jul 01 04:01:57 2016

Revert buildbot 361b81b43fc7793b66facd4bac7abe0adb30f3d5.

This commit was only added in the 0.8.4 branch, and was never made to master.

The bug only happens when starting buildbot with "make start", but we start it
directly with twistd in Chrome: http://trac.buildbot.net/ticket/1992

BUG= 620572 

Review-Url: https://codereview.chromium.org/2108553002

[modify] https://crrev.com/8c7a5b159b444e0193f9b2d8a094d352c226e7d1/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 8 by bugdroid1@chromium.org, Jul 1 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/5f0fc7590d1b5f11f5b0969cd7ce7e5d9729e256

commit 5f0fc7590d1b5f11f5b0969cd7ce7e5d9729e256
Author: dsansome <dsansome@chromium.org>
Date: Fri Jul 01 04:13:45 2016

Cherry-pick buildbot 6bd279caf945e51882a1126288355528db68c769:

print sqlite version when pool is loaded

This also checks for and logs about concurrency problems in older
versions of SQLite.  Refs #2005.

BUG= 620572 

Review-Url: https://codereview.chromium.org/2101993002

[modify] https://crrev.com/5f0fc7590d1b5f11f5b0969cd7ce7e5d9729e256/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 11 by bugdroid1@chromium.org, Jul 1 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8d24f5663861a7ae00e544f96e5c2e35787ecf4d

commit 8d24f5663861a7ae00e544f96e5c2e35787ecf4d
Author: dsansome <dsansome@chromium.org>
Date: Fri Jul 01 04:46:03 2016

Cherry-pick buildbot 95deef27d7c531ead19e0ac86a9aa1546d4ee7f9:

Re-run queries after certain OperationalErrors

Sometimes malformed SQL can generate an OperationalError, so this looks
at the message itself.  It's unclear how this will work with
localization.

The code uses an exponential backoff algorithm (with a relatively small
multiplier), and will retry for a day, which seems a reasonable
get-the-db-server-fixed time.

Fixes #2005.  Hoepfully for good!

BUG= 620572 

Review-Url: https://codereview.chromium.org/2103053002

[modify] https://crrev.com/8d24f5663861a7ae00e544f96e5c2e35787ecf4d/third_party/buildbot_8_4p1/buildbot/db/pool.py
[modify] https://crrev.com/8d24f5663861a7ae00e544f96e5c2e35787ecf4d/third_party/buildbot_8_4p1/buildbot/test/unit/test_db_pool.py
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/30c81b34a804043ae02f69a3066c8f410c5af7e2

commit 30c81b34a804043ae02f69a3066c8f410c5af7e2
Author: stip <stip@chromium.org>
Date: Thu Jul 07 16:37:20 2016

Revert of Revert "Fix masters_test presubmit_tests by using serialized sqlite access." (patchset #1 id:1 of https://codereview.chromium.org/2102003003/ )

Reason for revert:
possible cause of https://bugs.chromium.org/p/chromium/issues/detail?id=626354

Original issue's description:
> Revert 4e7e4672aae819f312c9691544652f251964007f to make multiple connections to sqlite databases again.
>
> This is safe after https://codereview.chromium.org/2103053002/
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/312d2566bfcb38bcaa25e93c27d9d862d05c7a8a

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2126033003

[modify] https://crrev.com/30c81b34a804043ae02f69a3066c8f410c5af7e2/third_party/buildbot_8_4p1/buildbot/db/enginestrategy.py
[modify] https://crrev.com/30c81b34a804043ae02f69a3066c8f410c5af7e2/third_party/buildbot_8_4p1/buildbot/master.py
Project Member

Comment 14 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/30c81b34a804043ae02f69a3066c8f410c5af7e2

commit 30c81b34a804043ae02f69a3066c8f410c5af7e2
Author: stip <stip@chromium.org>
Date: Thu Jul 07 16:37:20 2016

Revert of Revert "Fix masters_test presubmit_tests by using serialized sqlite access." (patchset #1 id:1 of https://codereview.chromium.org/2102003003/ )

Reason for revert:
possible cause of https://bugs.chromium.org/p/chromium/issues/detail?id=626354

Original issue's description:
> Revert 4e7e4672aae819f312c9691544652f251964007f to make multiple connections to sqlite databases again.
>
> This is safe after https://codereview.chromium.org/2103053002/
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/312d2566bfcb38bcaa25e93c27d9d862d05c7a8a

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2126033003

[modify] https://crrev.com/30c81b34a804043ae02f69a3066c8f410c5af7e2/third_party/buildbot_8_4p1/buildbot/db/enginestrategy.py
[modify] https://crrev.com/30c81b34a804043ae02f69a3066c8f410c5af7e2/third_party/buildbot_8_4p1/buildbot/master.py
Project Member

Comment 15 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/e9fc858ef555c00dc22fadf5514caed65c2b43e7

commit e9fc858ef555c00dc22fadf5514caed65c2b43e7
Author: stip <stip@chromium.org>
Date: Thu Jul 07 20:16:52 2016

Revert of Cherry-pick buildbot 95deef27d7c531ead19e0ac86a9aa1546d4ee7f9: (patchset #2 id:20001 of https://codereview.chromium.org/2103053002/ )

Reason for revert:
possible cause of
https://bugs.chromium.org/p/chromium/issues/detail?id=626354.

Original issue's description:
> Cherry-pick buildbot 95deef27d7c531ead19e0ac86a9aa1546d4ee7f9:
>
> Re-run queries after certain OperationalErrors
>
> Sometimes malformed SQL can generate an OperationalError, so this looks
> at the message itself.  It's unclear how this will work with
> localization.
>
> The code uses an exponential backoff algorithm (with a relatively small
> multiplier), and will retry for a day, which seems a reasonable
> get-the-db-server-fixed time.
>
> Fixes #2005.  Hoepfully for good!
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/8d24f5663861a7ae00e544f96e5c2e35787ecf4d

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2128953002

[modify] https://crrev.com/e9fc858ef555c00dc22fadf5514caed65c2b43e7/third_party/buildbot_8_4p1/buildbot/db/pool.py
[modify] https://crrev.com/e9fc858ef555c00dc22fadf5514caed65c2b43e7/third_party/buildbot_8_4p1/buildbot/test/unit/test_db_pool.py
Project Member

Comment 16 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/d69f3508e20628910774ac25a7c2370dc101b3ac

commit d69f3508e20628910774ac25a7c2370dc101b3ac
Author: stip <stip@chromium.org>
Date: Thu Jul 07 20:39:33 2016

Revert of Cherry pick buildbot cda35a4edcc822b0169412e2ad17c75973050807: (patchset #1 id:1 of https://codereview.chromium.org/2106713002/ )

Reason for revert:
possible cause of
https://bugs.chromium.org/p/chromium/issues/detail?id=626354.

Original issue's description:
> Cherry-pick buildbot cda35a4edcc822b0169412e2ad17c75973050807:
>
> be more forceful in sqlite version requirements
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/248f23a80cb3ecc4e02444523425617b36beacec

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2124383002

[modify] https://crrev.com/d69f3508e20628910774ac25a7c2370dc101b3ac/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 17 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/d801fa48de362aeb5634c7be1bb7e80055289cd2

commit d801fa48de362aeb5634c7be1bb7e80055289cd2
Author: stip <stip@chromium.org>
Date: Thu Jul 07 20:50:03 2016

Revert of Cherry-pick buildbot 7d31a0593e2e207289511c10f68a8bc4ba13b759: (patchset #1 id:1 of https://codereview.chromium.org/2103043002/ )

Reason for revert:
possible cause of
https://bugs.chromium.org/p/chromium/issues/detail?id=626354.

Original issue's description:
> Cherry-pick buildbot 7d31a0593e2e207289511c10f68a8bc4ba13b759:
>
> document buildbot.db.pool
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/3453b3593ae6688d1d6a1514fc185f6425c47ffd

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2131703002

[modify] https://crrev.com/d801fa48de362aeb5634c7be1bb7e80055289cd2/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 18 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8a713dd3408bd0dc54a3972ce0415457eab57acd

commit 8a713dd3408bd0dc54a3972ce0415457eab57acd
Author: stip <stip@chromium.org>
Date: Thu Jul 07 20:51:20 2016

Revert of Revert buildbot 361b81b43fc7793b66facd4bac7abe0adb30f3d5. (patchset #1 id:1 of https://codereview.chromium.org/2108553002/ )

Reason for revert:
possible cause of
https://bugs.chromium.org/p/chromium/issues/detail?id=626354.

Original issue's description:
> Revert buildbot 361b81b43fc7793b66facd4bac7abe0adb30f3d5.
>
> This commit was only added in the 0.8.4 branch, and was never made to master.
>
> The bug only happens when starting buildbot with "make start", but we start it
> directly with twistd in Chrome: http://trac.buildbot.net/ticket/1992
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/8c7a5b159b444e0193f9b2d8a094d352c226e7d1

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2128963002

[modify] https://crrev.com/8a713dd3408bd0dc54a3972ce0415457eab57acd/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 19 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/0bb4402f59c601c5c2dd4d3439e09ae5b2059648

commit 0bb4402f59c601c5c2dd4d3439e09ae5b2059648
Author: stip <stip@chromium.org>
Date: Thu Jul 07 23:49:42 2016

Revert of Cherry-pick buildbot de928a17f52dd91e17fab66d1cd5a60eed6aa360: (patchset #1 id:1 of https://codereview.chromium.org/2108563002/ )

Reason for revert:
possible cause of
https://bugs.chromium.org/p/chromium/issues/detail?id=626354.

Original issue's description:
> Cherry-pick buildbot de928a17f52dd91e17fab66d1cd5a60eed6aa360:
>
> add warning for older than sqlite-3.7 regarding WAL logs
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/fb705f915463ef3f773d55bb1aa2140ab1cfd463

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2130993002

[modify] https://crrev.com/0bb4402f59c601c5c2dd4d3439e09ae5b2059648/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 20 by bugdroid1@chromium.org, Jul 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/0c2979728e4968dee4dc2da001fcd251548c500f

commit 0c2979728e4968dee4dc2da001fcd251548c500f
Author: stip <stip@chromium.org>
Date: Fri Jul 08 00:01:11 2016

Revert of Cherry-pick buildbot 6bd279caf945e51882a1126288355528db68c769: (patchset #1 id:1 of https://codereview.chromium.org/2101993002/ )

Reason for revert:
possible cause of
https://bugs.chromium.org/p/chromium/issues/detail?id=626354

Original issue's description:
> Cherry-pick buildbot 6bd279caf945e51882a1126288355528db68c769:
>
> print sqlite version when pool is loaded
>
> This also checks for and logs about concurrency problems in older
> versions of SQLite.  Refs #2005.
>
> BUG= 620572 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/5f0fc7590d1b5f11f5b0969cd7ce7e5d9729e256

TBR=dsansome@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 620572 

Review-Url: https://codereview.chromium.org/2134453003

[modify] https://crrev.com/0c2979728e4968dee4dc2da001fcd251548c500f/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 21 by bugdroid1@chromium.org, Jul 19 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/74ccfc967788cc645bbfdc17f4fb81088ec05833

commit 74ccfc967788cc645bbfdc17f4fb81088ec05833
Author: dsansome <dsansome@chromium.org>
Date: Tue Jul 19 03:49:50 2016

Add a script to quickly migrate a master from postgresql to sqlite.

I intend to try this on one master first (probably master.chromium.infra) and let it run for a few days to see if there are any problems.

BUG= 620572 

Review-Url: https://codereview.chromium.org/2096593002

[add] https://crrev.com/74ccfc967788cc645bbfdc17f4fb81088ec05833/scripts/tools/migrate_psql_to_sqlite.py
Project Member

Comment 22 by bugdroid1@chromium.org, Jul 20 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/7d96879707fcb412ebb1136cfde4af00955fab53

commit 7d96879707fcb412ebb1136cfde4af00955fab53
Author: dsansome <dsansome@chromium.org>
Date: Wed Jul 20 03:02:26 2016

Re-land https://codereview.chromium.org/2101993002/

Cherry-pick buildbot 6bd279caf945e51882a1126288355528db68c769:

print sqlite version when pool is loaded

This also checks for and logs about concurrency problems in older
versions of SQLite.  Refs #2005.

BUG= 620572 

Review-Url: https://codereview.chromium.org/2162793002

[modify] https://crrev.com/7d96879707fcb412ebb1136cfde4af00955fab53/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 24 by bugdroid1@chromium.org, Jul 20 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/d55ae245e3194142a45fb34a7c8e86cbd00d0409

commit d55ae245e3194142a45fb34a7c8e86cbd00d0409
Author: dsansome <dsansome@chromium.org>
Date: Wed Jul 20 03:42:15 2016

Re-land https://codereview.chromium.org/2108553002/

Revert buildbot 361b81b43fc7793b66facd4bac7abe0adb30f3d5.

This commit was only added in the 0.8.4 branch, and was never made to master.

The bug only happens when starting buildbot with "make start", but we start it
directly with twistd in Chrome: http://trac.buildbot.net/ticket/1992

BUG= 620572 

Review-Url: https://codereview.chromium.org/2165433002

[modify] https://crrev.com/d55ae245e3194142a45fb34a7c8e86cbd00d0409/third_party/buildbot_8_4p1/buildbot/db/pool.py
Project Member

Comment 27 by bugdroid1@chromium.org, Jul 20 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/17ba0a73166bf637528205686ac74f8466eadfc4

commit 17ba0a73166bf637528205686ac74f8466eadfc4
Author: dsansome <dsansome@chromium.org>
Date: Wed Jul 20 04:20:32 2016

Re-land https://codereview.chromium.org/2103053002/

Cherry-pick buildbot 95deef27d7c531ead19e0ac86a9aa1546d4ee7f9:

Re-run queries after certain OperationalErrors

Sometimes malformed SQL can generate an OperationalError, so this looks
at the message itself.  It's unclear how this will work with
localization.

The code uses an exponential backoff algorithm (with a relatively small
multiplier), and will retry for a day, which seems a reasonable
get-the-db-server-fixed time.

Fixes #2005.  Hoepfully for good!

BUG= 620572 

Review-Url: https://codereview.chromium.org/2162553003

[modify] https://crrev.com/17ba0a73166bf637528205686ac74f8466eadfc4/third_party/buildbot_8_4p1/buildbot/db/pool.py
[modify] https://crrev.com/17ba0a73166bf637528205686ac74f8466eadfc4/third_party/buildbot_8_4p1/buildbot/test/unit/test_db_pool.py
Project Member

Comment 28 by bugdroid1@chromium.org, Jul 20 2016 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/c25fb3a8e16531a9b8bec19c4ff228de52ea7db7

commit c25fb3a8e16531a9b8bec19c4ff228de52ea7db7
Author: dsansome <dsansome@google.com>
Date: Wed Jul 20 23:53:20 2016

Comment 30 by dsansome@chromium.org, Oct 21 2016

Cc: dsansome@chromium.org
Owner: chrishall@chromium.org
Chris will add some instrumentation to buildbot's db code in build/third_party/buildbot_8_4p1/buildbot/db/*.py.

Look in buildbot/status/web/baseweb.py for an example, and http_metrics.py in infra/packages/infra_libs.

We want:
Query count
Query response (success/failure)
Rows touched/returned
Duration

Fields:
SQL verb (SELECT/UPDATE/...)
Table

Comment 31 by dsansome@chromium.org, Dec 6 2016

Owner: dsansome@chromium.org
Project Member

Comment 32 by bugdroid1@chromium.org, Dec 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1

commit fbc10b815db1028a7d6167f9bb0f9d2255c20ac1
Author: David Sansome <dsansome@chromium.org>
Date: Wed Dec 07 05:08:49 2016

Instrument buildbot's database queries.

BUG= 620572 

Change-Id: I829597a8b00119fa47f1db435cbc7a42cb6ba9c7
Reviewed-on: https://chromium-review.googlesource.com/416701
Reviewed-by: Mike Stipicevic <stip@chromium.org>
Commit-Queue: Dave Sansome <dsansome@chromium.org>

[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/buildrequests.py
[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/builds.py
[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/buildsets.py
[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/changes.py
[add] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/monitoring.py
[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/schedulers.py
[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/sourcestamps.py
[modify] https://crrev.com/fbc10b815db1028a7d6167f9bb0f9d2255c20ac1/third_party/buildbot_8_4p1/buildbot/db/state.py

Comment 33 by benhenry@chromium.org, Jan 18 2017 

Ping - please provide an update to your high priority bug. This bug is stale. Is it really P-1?

Comment 34 by dsansome@chromium.org, Jul 6 2017

Owner: ----
Status: Available (was: Assigned)

Comment 35 by no...@chromium.org, Jun 2 2018

Labels: -Pri-1 -Type-Bug Pri-3 Type-Feature

Comment 36 by hinoka@chromium.org, Jun 21 2018

Status: WontFix (was: Available)
We're phasing out buildbot by the end of the year, I don't think we'll need this.
►

Sign in to add a comment