New issue
Advanced search Search tips

Issue 856724 link

Starred by 1 user
Status: WontFix
Owner:
no...@chromium.org
Closed: Jun 2018
Cc:
tandrii@chromium.org
vadimsh@chromium.org
jclinton@chromium.org
no...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature



Sign in to add a comment

Optimize search queries

Project Member Reported by no...@chromium.org, Jun 26 2018 
model.Build contains large non-indexed properties e.g. parameters, result_details. Some API users don't need them. For example, Gerrit, the primary driver of Buildbucket QPS, doesn't need them. Same for Milo builder page. Those clients needs fast response because they are GUI.

Move all non-indexed properties out of model.Build to separate entities under Build and load them on demand, closed to API response.

This will improve latency of Milo builder page and GoldenEye.
Project Member

Comment 2 by bugdroid1@chromium.org, Jun 27 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/437fa860b12aec9dd43b271d560021c81d553bf1

commit 437fa860b12aec9dd43b271d560021c81d553bf1
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jun 27 00:30:52 2018

[buildbucket] Split TaskBackfillTagIndex

Split TaskBackfillTagIndex into 3 separate handles hosted on 3 different
URL paths suffixes. Remove "action" key in task payload. Exclude tag from URL
paths. Split tests in the same way.

This CL is pure refactoring.

Bug:  856724 

Change-Id: I5454e20a283109959fea8152b1f4e19230f757e7
Reviewed-on: https://chromium-review.googlesource.com/1115349
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/437fa860b12aec9dd43b271d560021c81d553bf1/appengine/cr-buildbucket/bulkproc.py
[modify] https://crrev.com/437fa860b12aec9dd43b271d560021c81d553bf1/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/437fa860b12aec9dd43b271d560021c81d553bf1/appengine/cr-buildbucket/test/bulkproc_test.py
[modify] https://crrev.com/437fa860b12aec9dd43b271d560021c81d553bf1/appengine/cr-buildbucket/api.py
Project Member

Comment 4 by bugdroid1@chromium.org, Jun 27 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6f75dc3fa65ec7026758286b8b8a5ca4758427bb

commit 6f75dc3fa65ec7026758286b8b8a5ca4758427bb
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jun 27 21:10:35 2018

[buildbucket] Store steps in v2 format

Start storing build steps in v2 format, as a binary protobuf messages contained
in a buildbucket.v2.Build, in new datastore entity "BuildSteps".

Leave BuildAnnotations entity until we migrate to the new schema.

Bug:  853450 ,  856724 
Change-Id: I4e48f71707d59d635faac544e5f6ead64bde384a
Reviewed-on: https://chromium-review.googlesource.com/1115766
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/annotations.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/v2/api.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/model.py

Comment 5 by no...@chromium.org, Jun 27 2018

Summary: Optimize search queries (was: Move not-indexed properties out of model.Build)
Project Member

Comment 6 by bugdroid1@chromium.org, Jun 28 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/63aebea725dfbef04b146b5e92659e54a2b765ad

commit 63aebea725dfbef04b146b5e92659e54a2b765ad
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Jun 28 15:17:00 2018

[buildbucket] Exclude task_result from result_details

Swarming task result is large, so it slows queries down. Exclude it.

Its bot_dimensions is used in v2/builds.py. Leave those in result_details,
but make them smaller.

Bug:  856724 
Change-Id: I3e415860a19995b4c814daf36deb1db2c3bb3813
Reviewed-on: https://chromium-review.googlesource.com/1117830
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/63aebea725dfbef04b146b5e92659e54a2b765ad/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/63aebea725dfbef04b146b5e92659e54a2b765ad/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/63aebea725dfbef04b146b5e92659e54a2b765ad/appengine/cr-buildbucket/v2/test/builds_test.py
[modify] https://crrev.com/63aebea725dfbef04b146b5e92659e54a2b765ad/appengine/cr-buildbucket/swarming/test/swarming_test.py

Comment 7 by no...@chromium.org, Jun 28 2018

Status: WontFix (was: Started)
i ended up not doing the data remodeling for now. I'd like to do some analysis first.

however, the motivation for this optimization was Legoland queries, which, apparently cannot use v2 batch search because those builds are still on LUCI (and search v2 currently does not support filtering by a buildbot builder)

when we can, we'll need to define a concrete metric to optimize. In the meantime, I will closed this bug

Comment 8 by jclinton@chromium.org, Jul 2 

> which, apparently cannot use v2 batch search because those builds are still on LUCI 

What does this mean? I'm sorry, I'm not following. You mean that because of the way that we are using Buildbot? We should be off of Buildbot in about 2 months.

Comment 9 by jclinton@chromium.org, Jul 2

Cc: jclinton@chromium.org

Comment 10 by no...@chromium.org, Jul 9 

today a bucket is identified by its name in cr-buildbucket.cfg file, e.g. "luci.chromeos.general". We want to change bucket identifier to (project_id, bucket_name) tuple, so that given a bucket id, one can tell its project id. This transition wasn't done yet. We have "luci.*" prefix in buckets to make such transition easier in the future. When it happens, the bucket names will be shorter, e.g. "general", not "luci.chromeos.general", so that bucket id is ("chromeos", "general"), not ("chromeos", "luci.chromeos.general").

v2 APIs accept new form from the start, e.g. builder id {project "chromeos", bucket: "general", builder: "Try"}. Internally, buildbucket prepends "luci.<project>." to the short bucket name, because internally all buckets are still stored with long names.
This makes it impossible to specify builder id that does not have "luci." prefix. Buildbot builders don't.

So to start searching for builds in a given builder using v2 API, we have to move to a LUCI (so that it has "luci." prefix) or I should fix  bug 851036  so that v2 API implementation stops prepending "luci.<project>." prefix.
►

Sign in to add a comment