New issue
Advanced search Search tips

Issue 851036 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 23
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Task

Blocking:
issue 875056



Sign in to add a comment

Migrate to short bucket names

Project Member Reported by no...@chromium.org, Jun 8 2018

Issue description

LUCI buckets are currently named "luci.{project}.{short_name}" so that eventually we can make bucket names unique only within the project and use the short name as a bucket name. Then bucket identifier is tuple (project, short_name).

Now that GetBuild and SearchBulids v2 APIs are available, it is time to implement this.
 

Comment 1 by no...@chromium.org, Jun 16 2018

Summary: Migrate to short bucket names (was: Migrate to short bucket names)
Talked to estaab. Maybe a little later. Implementing config push is more important. Doing this when more builds are on LUCI is easier.

However, we should design skylark high level config to use short names for LUCI buckets from day 1.

Comment 2 by no...@chromium.org, Jun 16 2018

Labels: -Pri-1 Pri-2
Blocking: 875056
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 26

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/18cebe6abd3e73a18dd8e9553406e352d5b0c150

commit 18cebe6abd3e73a18dd8e9553406e352d5b0c150
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Sep 26 23:06:53 2018

[buildbucket] Remove bucket reservation code

Short bucket name scheme allows duplicate bucket names in different projects

R=vadimsh@chromium.org

Bug:  851036 
Change-Id: Ib99ed5c11ffdc526b565b56b7c4fb97a5d964c75
Reviewed-on: https://chromium-review.googlesource.com/1244279
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17903}
[modify] https://crrev.com/18cebe6abd3e73a18dd8e9553406e352d5b0c150/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/18cebe6abd3e73a18dd8e9553406e352d5b0c150/appengine/cr-buildbucket/test/config_test.py

Status: Started (was: Assigned)
Project Member

Comment 6 by bugdroid1@chromium.org, Oct 2

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/62e56d7aa3420184445a0c8160404652c8484d59

commit 62e56d7aa3420184445a0c8160404652c8484d59
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Oct 02 17:35:17 2018

[buildbucket] add config.put_bucket

Add config.put_bucket which encapsulates how a bucket is put.
Used it in tests.
It will be extended to save two buckets.

Bug:  851036 
Change-Id: I391090297d5c855aef2e039ef90b159dbd319275
Reviewed-on: https://chromium-review.googlesource.com/c/1252021
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17992}
[modify] https://crrev.com/62e56d7aa3420184445a0c8160404652c8484d59/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/62e56d7aa3420184445a0c8160404652c8484d59/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/62e56d7aa3420184445a0c8160404652c8484d59/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/62e56d7aa3420184445a0c8160404652c8484d59/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 7 by bugdroid1@chromium.org, Oct 2

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d26a9cab655287479e806588fcf2fed020c3118c

commit d26a9cab655287479e806588fcf2fed020c3118c
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Oct 02 17:46:17 2018

[buildbucket] config.Bucket -> LegacyBucket

Rename config.Bucket to LegacyBucket. It will be deleted.

Bug:  851036 
Change-Id: I23f98d28fb9e1c06c08deda2d159bfef9cf8ff2e
Reviewed-on: https://chromium-review.googlesource.com/c/1251803
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17993}
[modify] https://crrev.com/d26a9cab655287479e806588fcf2fed020c3118c/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/d26a9cab655287479e806588fcf2fed020c3118c/appengine/cr-buildbucket/test/config_test.py
[modify] https://crrev.com/d26a9cab655287479e806588fcf2fed020c3118c/appengine/cr-buildbucket/api.py

Project Member

Comment 8 by bugdroid1@chromium.org, Oct 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c6a91751c6b8bb5d327f793532c6c0f1d4e53187

commit c6a91751c6b8bb5d327f793532c6c0f1d4e53187
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 17:51:11 2018

[buildbucket] Add config.Bucket

- add config.Bucket which is a new format of bucket configuration storage.
- config.put_bucket: put two buckets
- config.cron_update_buckets: use new bucket to check for diff

It is a simpler alternative to
https://chromium-review.googlesource.com/c/infra/infra/+/1249944

Bug:  851036 
Change-Id: I9392211ed9172ea7073da667f2e4a2e319a6664b
Reviewed-on: https://chromium-review.googlesource.com/c/1252026
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18052}
[modify] https://crrev.com/c6a91751c6b8bb5d327f793532c6c0f1d4e53187/appengine/cr-buildbucket/config.py

Project Member

Comment 9 by bugdroid1@chromium.org, Oct 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9b515518d85e4a481dd8cf6a30c343dcd11f4a7b

commit 9b515518d85e4a481dd8cf6a30c343dcd11f4a7b
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 18:06:37 2018

[buildbucket] Add model.Build.bucket_id

model.Build.bucket_id is a repalcement for model.Build.bucket.
It has format "<project_id>/<bucket_name>". The rest of the code
will be migrated to use it.
model.Build.bucket is deprecated.

Bug:  851036 
Change-Id: I7a7a006bb881c32a0f4655bd080a1cd3cbc8e409
Reviewed-on: https://chromium-review.googlesource.com/c/1252343
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18054}
[modify] https://crrev.com/9b515518d85e4a481dd8cf6a30c343dcd11f4a7b/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/9b515518d85e4a481dd8cf6a30c343dcd11f4a7b/appengine/cr-buildbucket/creation.py

Project Member

Comment 10 by bugdroid1@chromium.org, Oct 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f0b32741888233fae972af7254f01b239cd8acf0

commit f0b32741888233fae972af7254f01b239cd8acf0
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 20:35:27 2018

[buildbucket] add reput_builds.py

Add a bulkproc job that, simply, reads a build and puts it back.
As a side effect, it recomputes all of its properties.

Some completed builds in prod and dev turned out to have null complete_time.
Derive complete_time from status_changed_time, which is always set.

Some completed builds in prod and dev do not have "project" property.
Try to restore it from current bucket->project mapping, otherwise
fallback to "chrome" project.

Bug:  851036 
Change-Id: I189b3cdd6e938b9455fd88bdbcbb0299b5e4fdbc
Reviewed-on: https://chromium-review.googlesource.com/c/1252352
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18063}
[add] https://crrev.com/f0b32741888233fae972af7254f01b239cd8acf0/appengine/cr-buildbucket/reput_builds.py
[modify] https://crrev.com/f0b32741888233fae972af7254f01b239cd8acf0/appengine/cr-buildbucket/api.py

Project Member

Comment 11 by bugdroid1@chromium.org, Oct 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2f3f2c8eb24660f1e4e5beef689d55678de20858

commit 2f3f2c8eb24660f1e4e5beef689d55678de20858
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 21:00:07 2018

[buildbucket] Update config.get_bucket_async

Add support for modern bucket_id to config.get_bucket_async.
Update tests.

Bug:  851036 
Change-Id: Ic6ae6baf7e6f9a6e1a9d7d5e38eb820e39824f8a
Reviewed-on: https://chromium-review.googlesource.com/c/1252359
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18064}
[modify] https://crrev.com/2f3f2c8eb24660f1e4e5beef689d55678de20858/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/2f3f2c8eb24660f1e4e5beef689d55678de20858/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 12 by bugdroid1@chromium.org, Oct 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/65f91384968b07737e8135308f6e17eeffb0e0ee

commit 65f91384968b07737e8135308f6e17eeffb0e0ee
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 21:03:25 2018

[buildbucket] Add config.validate_bucket_id

config.validate_bucket_id validates a bucket_id string or legacy bucket name
string.

It also prohibits "/" in bucket name. No prod buckets use it.

Bug:  851036 
Change-Id: Ie11ae63a0d1f7793322abe99625a6333779ae2be
Reviewed-on: https://chromium-review.googlesource.com/c/1253401
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18065}
[modify] https://crrev.com/65f91384968b07737e8135308f6e17eeffb0e0ee/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/65f91384968b07737e8135308f6e17eeffb0e0ee/appengine/cr-buildbucket/errors.py
[modify] https://crrev.com/65f91384968b07737e8135308f6e17eeffb0e0ee/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 13 by bugdroid1@chromium.org, Oct 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/284c1fe1efd966372c58641f8f0d9c905ae10ac4

commit 284c1fe1efd966372c58641f8f0d9c905ae10ac4
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 21:12:47 2018

[buildbucket] Update user.get_role_async

Add support for modern bucket_id to user.get_role_async.
Update tests.

Bug:  851036 
Change-Id: If0b9462ce63dbe5c37558af22a5f822d4041f901
Reviewed-on: https://chromium-review.googlesource.com/c/1252733
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18066}
[modify] https://crrev.com/284c1fe1efd966372c58641f8f0d9c905ae10ac4/appengine/cr-buildbucket/user.py
[modify] https://crrev.com/284c1fe1efd966372c58641f8f0d9c905ae10ac4/appengine/cr-buildbucket/test/user_test.py

Project Member

Comment 14 by bugdroid1@chromium.org, Oct 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f9a3155494f80de8c6e28156ed3488a7c69a99ae

commit f9a3155494f80de8c6e28156ed3488a7c69a99ae
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 05 19:11:53 2018

[buildbucket] Update config.get_buckets_async

Add support for modern bucket_id to user.get_buckets_async.
Update tests.

Bug:  851036 
Change-Id: Icce68ed33294d3ea151279516394c3a6a8a6608a
Reviewed-on: https://chromium-review.googlesource.com/c/1252740
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18111}
[modify] https://crrev.com/f9a3155494f80de8c6e28156ed3488a7c69a99ae/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/f9a3155494f80de8c6e28156ed3488a7c69a99ae/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 15 by bugdroid1@chromium.org, Oct 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0bfc3af23b38c8ae6edb2531c2dd845199f578b4

commit 0bfc3af23b38c8ae6edb2531c2dd845199f578b4
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 05 19:28:32 2018

[buildbucket] Update config.get_accessible_buckets_async

Add support for modern bucket_id to user.get_accessible_buckets_async.
Update tests.

R=vadimsh@chromium.org

Bug:  851036 
Change-Id: I6a8a9fdf6e770e3efd9b33aff2253d450fec278b
Reviewed-on: https://chromium-review.googlesource.com/c/1252923
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18113}
[modify] https://crrev.com/0bfc3af23b38c8ae6edb2531c2dd845199f578b4/appengine/cr-buildbucket/user.py
[modify] https://crrev.com/0bfc3af23b38c8ae6edb2531c2dd845199f578b4/appengine/cr-buildbucket/test/user_test.py

Project Member

Comment 16 by bugdroid1@chromium.org, Oct 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a5b96aaddcb89c5820ac5898e1ed5e9e05d5c624

commit a5b96aaddcb89c5820ac5898e1ed5e9e05d5c624
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 05 19:35:04 2018

[buildbucket] Update config.can_async

Add support for modern bucket_id to user.can_async.
Update tests.

Bug:  851036 
Change-Id: If8cd43673fc77e1ff0752fb60418c4500fd6a4d9
Reviewed-on: https://chromium-review.googlesource.com/c/1252926
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18114}
[modify] https://crrev.com/a5b96aaddcb89c5820ac5898e1ed5e9e05d5c624/appengine/cr-buildbucket/user.py
[modify] https://crrev.com/a5b96aaddcb89c5820ac5898e1ed5e9e05d5c624/appengine/cr-buildbucket/test/user_test.py

Project Member

Comment 18 by bugdroid1@chromium.org, Oct 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b8593a94d3a0a3adb80a31c434df19557378c35a

commit b8593a94d3a0a3adb80a31c434df19557378c35a
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 05 22:39:51 2018

[buildbucket] Add search.TagIndexEntry.bucket_id

Add search.TagIndexEntry.bucket_id field that uses new format, same
as Build.bucket_id.

Bug:  851036 
Change-Id: Ia56ccb36e4bb4459bae26f398018e545691b9907
Reviewed-on: https://chromium-review.googlesource.com/c/1263422
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18129}
[modify] https://crrev.com/b8593a94d3a0a3adb80a31c434df19557378c35a/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/b8593a94d3a0a3adb80a31c434df19557378c35a/appengine/cr-buildbucket/backfill_tag_index.py
[modify] https://crrev.com/b8593a94d3a0a3adb80a31c434df19557378c35a/appengine/cr-buildbucket/search.py
[modify] https://crrev.com/b8593a94d3a0a3adb80a31c434df19557378c35a/appengine/cr-buildbucket/test/creation_test.py

Project Member

Comment 19 by bugdroid1@chromium.org, Oct 6

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/585321055e72ee2341d24479d44fdb9316b9c453

commit 585321055e72ee2341d24479d44fdb9316b9c453
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Oct 06 03:41:35 2018

[buildbucket] Lazily populate TagIndexEntry.bucket_id

In the search code path, if a TagIndexEntry does not have bucket_id, populate
it. This will slow down first-time searches by a given tag (e.g. CL).
Future searches by the same tag won't make extra RPCs.
The longer the interval between deployments of
https://chromium-review.googlesource.com/c/infra/infra/+/1263422
and this CL, the less likely the migration code will run in practice,
because older builds are less likely to be searched.

This is simpler than implementing and running a MR job on TagIndex.
Note that backfill_tag_index.py does not support changing format of the index.

TagIndexEntry.bucket will be deleted when it is not used by search code path.

Also make TagIndex.permanently_incomplete non-indexed.

Bug:  851036 
Change-Id: I933688661f7e651cdb7c00c404279af004353df0
Reviewed-on: https://chromium-review.googlesource.com/c/1264016
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18135}
[modify] https://crrev.com/585321055e72ee2341d24479d44fdb9316b9c453/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/585321055e72ee2341d24479d44fdb9316b9c453/appengine/cr-buildbucket/search.py

Project Member

Comment 20 by bugdroid1@chromium.org, Oct 18

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8a4788ebf479ba11655c448fedad8c2c2a8595c6

commit 8a4788ebf479ba11655c448fedad8c2c2a8595c6
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 18 23:30:17 2018

[buildbucket] Update build_to_message

api_common.build_to_message is used everwhere an internal model.Build is
converted to a client-facing build in V1 APIs: RPC and PubSub.
Use bucket_id instead of project and bucket.

Bug:  851036 
Change-Id: I9b70b1e7744e97ac889cf7deb0c1156868c600a0
Reviewed-on: https://chromium-review.googlesource.com/c/1266782
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18463}
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/events.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/reput_builds.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/user.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/swarming/swarmbucket_api.py
[modify] https://crrev.com/8a4788ebf479ba11655c448fedad8c2c2a8595c6/appengine/cr-buildbucket/test/user_test.py

Project Member

Comment 21 by bugdroid1@chromium.org, Oct 19

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c

commit 2a53f0c27f390b3e4379b3ab1b2230d82adcf83c
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 19 01:17:58 2018

[buildbucket] Update search

Switch search APIs and implementations to bucket ids.
Update tests.

Improve search tests:
- use modern test values for project an bucket strings
- do not pass bucket_ids where it is irrelevant

Bug:  851036 
Change-Id: I73f7c48aa19cd96aa9167efa61b2088171461713
Reviewed-on: https://chromium-review.googlesource.com/c/1265335
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18470}
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/test/config_test.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/v2/api.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/api.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/search.py
[modify] https://crrev.com/2a53f0c27f390b3e4379b3ab1b2230d82adcf83c/appengine/cr-buildbucket/config.py

Project Member

Comment 23 by bugdroid1@chromium.org, Oct 19

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2272740bcc8232ac7a0db0ee5c5b7bd0e2d001cd

commit 2272740bcc8232ac7a0db0ee5c5b7bd0e2d001cd
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 19 22:04:16 2018

[buildbucket] Fix resolve_bucket_name_async cache

resolve_bucket_name_async() memcache key does not include bucket name.
Incorporate bucket name.

TBR=vadimsh@chromium.org

Bug:  851036 
Change-Id: I2c2fcccb8f7c10da842703a91d965bc1cae4a366
Reviewed-on: https://chromium-review.googlesource.com/c/1292343
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18496}
[modify] https://crrev.com/2272740bcc8232ac7a0db0ee5c5b7bd0e2d001cd/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/2272740bcc8232ac7a0db0ee5c5b7bd0e2d001cd/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 24 by bugdroid1@chromium.org, Oct 19

Project Member

Comment 26 by bugdroid1@chromium.org, Oct 19

Project Member

Comment 27 by bugdroid1@chromium.org, Oct 20

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/5d21b61d8922783e1f86e64740632e2a34b9be92

commit 5d21b61d8922783e1f86e64740632e2a34b9be92
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 19 23:59:52 2018

Project Member

Comment 28 by bugdroid1@chromium.org, Oct 20

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/93e1cf16dacf85144c3bc517ff6a9435303a54ad

commit 93e1cf16dacf85144c3bc517ff6a9435303a54ad
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Oct 20 00:09:18 2018

[buildbucket] Add a composite index

metrics.set_build_latency requires a composite index that was never defined
in index.yaml, but a combination of unrelated indexes was used instead and
that's how the query worked.

Add an index suggested by error messages on cr-buildbucket-dev.

TBR=jchinlee@chromium.org

Bug:  851036 
Change-Id: I4199d17069afa5f515b59b70844ba74ca00e997e
Reviewed-on: https://chromium-review.googlesource.com/c/1292762
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18505}
[modify] https://crrev.com/93e1cf16dacf85144c3bc517ff6a9435303a54ad/appengine/cr-buildbucket/index.yaml

Project Member

Comment 30 by bugdroid1@chromium.org, Oct 23

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8c74fc6a33a2d539cf3cf3e5edf2a391193c30d4

commit 8c74fc6a33a2d539cf3cf3e5edf2a391193c30d4
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Oct 23 05:37:33 2018

[buildbucket] Update swarming.py

Update most of swarming.py to use model.Build.bucket_id, as opposed to
model.Build.bucket

Remove "bucket" task template parameter. It is not used.

Use simpler ci.chromium.org/b/<id> URL and let Milo do a proper redirect.

Update delegation tag format. Corresponding config updates:
https://chrome-internal-review.googlesource.com/c/infradata/config/+/701613

Bug:  851036 
Change-Id: Ied35116e44d38fef5dc16c9bf7299e8e16781744
Reviewed-on: https://chromium-review.googlesource.com/c/1292755
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18533}
[modify] https://crrev.com/8c74fc6a33a2d539cf3cf3e5edf2a391193c30d4/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/8c74fc6a33a2d539cf3cf3e5edf2a391193c30d4/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/8c74fc6a33a2d539cf3cf3e5edf2a391193c30d4/appengine/cr-buildbucket/doc/swarming.md
[modify] https://crrev.com/8c74fc6a33a2d539cf3cf3e5edf2a391193c30d4/appengine/cr-buildbucket/swarming/test/swarming_test.py

Project Member

Comment 32 by bugdroid1@chromium.org, Oct 24

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5f2b266af2586da1ae48905041b71b79f6b1142e

commit 5f2b266af2586da1ae48905041b71b79f6b1142e
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Oct 24 17:13:43 2018

[buildbucket] Update delete_many_builds_async

Update service.delete_many_builds_async to accept bucket_id, not bucket.

Bug:  851036 
Change-Id: Ib9a4456674766db26b2f7aa6d31b5c32243af896
Reviewed-on: https://chromium-review.googlesource.com/c/1296071
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18559}
[modify] https://crrev.com/5f2b266af2586da1ae48905041b71b79f6b1142e/appengine/cr-buildbucket/service.py
[modify] https://crrev.com/5f2b266af2586da1ae48905041b71b79f6b1142e/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/5f2b266af2586da1ae48905041b71b79f6b1142e/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/5f2b266af2586da1ae48905041b71b79f6b1142e/appengine/cr-buildbucket/api.py

Project Member

Comment 33 by bugdroid1@chromium.org, Oct 24

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/58cf208f6cd871202f955a3936b079d8b3cad00b

commit 58cf208f6cd871202f955a3936b079d8b3cad00b
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Oct 24 17:49:05 2018

[buildbucket] Update sequence.py

Update sequence.py to use bucket ids.
Old ID format: "<bucket>/<builder>", e.g. "luci.chromium.try/linux".
New ID format: "<bucket_id>/<builder>". e.g. "chromium/try/linux".

This CL also implements lazy migration from old to new format.

If this CL is rolled back, buildbucket might generate duplicate build numbers,
so ideally it is not reverted.

Bug:  851036 
Change-Id: Id933298392e2b4a61101ac8a91f41706d5eebf9c
Reviewed-on: https://chromium-review.googlesource.com/c/1296070
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18560}
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/sequence.py
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/swarming/swarmbucket_api.py
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/creation.py
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/58cf208f6cd871202f955a3936b079d8b3cad00b/appengine/cr-buildbucket/test/sequence_test.py

Project Member

Comment 34 by bugdroid1@chromium.org, Nov 1

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/90a42fe9d3042bbb629ddbce79afba561a4ef84a

commit 90a42fe9d3042bbb629ddbce79afba561a4ef84a
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Nov 01 22:54:20 2018

[buildbucket] Remove model.Build.bucket

Nothing reads model.Build.bucket. Code was updated to read model.Build.bucket_id
Remove model.Build.bucket.
Make model.Build.bucket_id non-computed
and model.Build.project computed.
When creating new builds, set bucket_id instead of project and bucket.

Most of this CL updates tests. Changes to non-test code are minimal.

Bug:  851036 
Change-Id: I70f3ad3f08d274c8e0719a7c22babc5f5fb29e38
Reviewed-on: https://chromium-review.googlesource.com/c/1296072
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18743}
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/backfill_tag_index_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/user_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/metrics_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/v2/test/builds_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/bulkproc_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/creation.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/model_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/90a42fe9d3042bbb629ddbce79afba561a4ef84a/appengine/cr-buildbucket/model.py

Project Member

Comment 35 by bugdroid1@chromium.org, Nov 6

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ef22b633cb12916d77c7aa120af6829a987a83a0

commit ef22b633cb12916d77c7aa120af6829a987a83a0
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Nov 06 17:37:00 2018

[buildbucket] Update set_next_build_number

Update set_next_build_number to use bucket ids.

Bug:  851036 
Change-Id: I267f228d2708191e751a452506c7ef6a3562c3f6
Reviewed-on: https://chromium-review.googlesource.com/c/1319974
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18811}
[modify] https://crrev.com/ef22b633cb12916d77c7aa120af6829a987a83a0/appengine/cr-buildbucket/swarming/swarmbucket_api.py

Project Member

Comment 36 by bugdroid1@chromium.org, Nov 6

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/aef87a3d8a1d847a9fc188f2308f61ac30eb3769

commit aef87a3d8a1d847a9fc188f2308f61ac30eb3769
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Nov 06 19:13:42 2018

[buildbucket] Update PermittedActions

Update PermittedActions RPC to use bucket ids.

R=vadimsh@chromium.org

Bug:  851036 
Change-Id: I8ed584c11e001140b11def151eb0ac46ecbf4df2
Reviewed-on: https://chromium-review.googlesource.com/c/1319980
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18814}
[modify] https://crrev.com/aef87a3d8a1d847a9fc188f2308f61ac30eb3769/appengine/cr-buildbucket/access/test/api_test.py
[modify] https://crrev.com/aef87a3d8a1d847a9fc188f2308f61ac30eb3769/appengine/cr-buildbucket/access/api.py

Project Member

Comment 37 by bugdroid1@chromium.org, Nov 6

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/390f24661e09953aac61a0aa46099799412bf77e

commit 390f24661e09953aac61a0aa46099799412bf77e
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Nov 06 21:08:53 2018

[buildbucket] Update validate_bucket_id

Disallow legacy bucket names in config.validate_bucket_id.
This guarantees that legacy bucket names are not floating in the code and we
can safely remove config.LegacyBucket entity.

Bug:  851036 
Change-Id: I5a62b9bc47d62f1eb13482dc2e362e73a0390d41
Reviewed-on: https://chromium-review.googlesource.com/c/1320869
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18819}
[modify] https://crrev.com/390f24661e09953aac61a0aa46099799412bf77e/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/390f24661e09953aac61a0aa46099799412bf77e/appengine/cr-buildbucket/service.py
[modify] https://crrev.com/390f24661e09953aac61a0aa46099799412bf77e/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/390f24661e09953aac61a0aa46099799412bf77e/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/390f24661e09953aac61a0aa46099799412bf77e/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 38 by bugdroid1@chromium.org, Nov 7

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8f092fa0751008ffb702770513966362fe0c4c11

commit 8f092fa0751008ffb702770513966362fe0c4c11
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Nov 07 22:24:14 2018

[buildbucket] Update ACL check in creation.py

Update access check in creation.py to use bucket_id, not bucket.
This also fixes retry API which is currently broken because of this.

Bug:  851036 
Change-Id: I5124c3bd0fb4bdc0fe8a6a5c3a5378be5466af8a
Reviewed-on: https://chromium-review.googlesource.com/c/1324769
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18846}
[modify] https://crrev.com/8f092fa0751008ffb702770513966362fe0c4c11/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/8f092fa0751008ffb702770513966362fe0c4c11/appengine/cr-buildbucket/creation.py

Project Member

Comment 39 by bugdroid1@chromium.org, Nov 14

Labels: merge-merged-config
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b3a92142c77cece83888be2acf94491d4f530104

commit b3a92142c77cece83888be2acf94491d4f530104
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Nov 14 04:10:17 2018

luci.chromium.ci: use short name

TBR=vadimsh@chromium.org
Bug:  851036 
Change-Id: Ic80430074d977a6678bb5bec8ae642aadc36a5ca
Reviewed-on: https://chromium-review.googlesource.com/c/1335170
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/b3a92142c77cece83888be2acf94491d4f530104/cr-buildbucket-dev.cfg

Project Member

Comment 41 by bugdroid1@chromium.org, Nov 14

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/91ac329db34382b431f6ed3eece1c63de463c1b3

commit 91ac329db34382b431f6ed3eece1c63de463c1b3
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Nov 14 17:25:28 2018

[buildbucket] Drop config.LegacyBucket

Drop config.LegacyBucket, nothing relies on it.
Update tests to use config.Bucket instead.
Remove a test that checks bucket reservation.
Change a test to use a short bucket name in one the configs.

R=vadimsh@chromium.org

Bug:  851036 
Change-Id: I5163987e6f35c6be205289df8dfc304e2199d87e
Reviewed-on: https://chromium-review.googlesource.com/c/1335169
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18984}
[modify] https://crrev.com/91ac329db34382b431f6ed3eece1c63de463c1b3/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/91ac329db34382b431f6ed3eece1c63de463c1b3/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/91ac329db34382b431f6ed3eece1c63de463c1b3/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 42 by bugdroid1@chromium.org, Nov 14

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9e87a843d18d6954cc171e95473e12ccf1405086

commit 9e87a843d18d6954cc171e95473e12ccf1405086
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Nov 14 17:53:15 2018

[buildbucket] Fix default pool name

Do not assume config uses long bucket name (luci.chromium.ci).
Generate the long bucket name for swarming pool ourselves.

R=vadimsh@chromium.org

Bug:  851036 
Change-Id: Iddc5834f0353187650273d2806bd06bfca8df975
Reviewed-on: https://chromium-review.googlesource.com/c/1335172
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18986}
[modify] https://crrev.com/9e87a843d18d6954cc171e95473e12ccf1405086/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/9e87a843d18d6954cc171e95473e12ccf1405086/appengine/cr-buildbucket/test/config_test.py

Project Member

Comment 43 by bugdroid1@chromium.org, Nov 14

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8f501f7ce07e48ade2b82060186690ea39b0e388

commit 8f501f7ce07e48ade2b82060186690ea39b0e388
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Nov 14 19:12:20 2018

Use short bucket names in infra project

Drop "luci.infra." prefix in bucket names.
Server drops them anyway.

Similar CL in cr-buildbucket-dev.cfg WAI.
https://chromium-review.googlesource.com/c/infra/infra/+/1335170

R=tandrii@chromium.org, vadimsh@chromium.org

Bug:  851036 
Change-Id: Ie6cd63de8efb28b5ec3b4ab81ed24a75e552cf21
Reviewed-on: https://chromium-review.googlesource.com/c/1335171
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/8f501f7ce07e48ade2b82060186690ea39b0e388/cr-buildbucket.cfg

Project Member

Comment 44 by bugdroid1@chromium.org, Nov 16

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/468d9e2dea961985edaee65928b78c057e4193e9

commit 468d9e2dea961985edaee65928b78c057e4193e9
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Nov 16 00:18:58 2018

Project Member

Comment 45 by bugdroid1@chromium.org, Nov 19

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/44d51a42c67d24df34c633cc83c4f38cc533592b

commit 44d51a42c67d24df34c633cc83c4f38cc533592b
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Nov 19 22:22:13 2018

Status: Fixed (was: Started)

Sign in to add a comment