this blocks any new progress in buildbucket v2 impl

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0500b40ddefa532af78fc86f8d784fe4cf86cb89

commit 0500b40ddefa532af78fc86f8d784fe4cf86cb89
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Jan 17 20:11:44 2019

[buildbucket] Add model.Build.proto

This is a first out of 3 CLs that migrate Build entity to a container of build proto:
1) For all new builds, model.Build.proto must be correct and corresponds to
   existing properties.
2) Run an "MR" job on all builds. By this time we will have a guarantee that
   all builds have a correct/complete "proto" property.
3) Update all code to use "proto" property and remove redundant properties.

Rearrange all model.Build properties, moving some of them into groups:
1. derived indexed properties
2. v1 properties
3. redundant with proto, to be deleted properties
No property was dropped.

Update build creation code path to initialize model.Build.proto. This code
is guaranteed to have a valid proto property. Update other code that mutates
now-redundant properties to also update the proto field if proto is initialized.

Remove build_number parameters in swarming.py because it is available
in build.proto.number now. This simplifies the code.

Update tests to assert proto fields.

R=vadimsh@chromium.org

Bug: 917851
Change-Id: I85dae5b8ec3ada49bc99749572b776adb8aabc02
Reviewed-on: https://chromium-review.googlesource.com/c/1415379
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20056}
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/sequence.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/service.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/swarming/swarmbucket_api.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/creation.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/0500b40ddefa532af78fc86f8d784fe4cf86cb89/appengine/cr-buildbucket/test/model_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9032b42c01fbcfbfeec3731641fa32fcc84ef31e

commit 9032b42c01fbcfbfeec3731641fa32fcc84ef31e
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 18 00:13:49 2019

[buildbucket] Backfill model.Build.proto

Update fix_builds.py to set model.Build.proto to whatever b2.build_to_v2
returns, minus fields that should be excluded in model.Build.proto.

Bug: 917851
Change-Id: Ie03283b400a1d6cf6cba3dceb97b318c8f5fecfb
Reviewed-on: https://chromium-review.googlesource.com/c/1415383
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20061}
[modify] https://crrev.com/9032b42c01fbcfbfeec3731641fa32fcc84ef31e/appengine/cr-buildbucket/fix_builds.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/24ce520ae6577c5693e5e689d7ea40c81efdd3a2

commit 24ce520ae6577c5693e5e689d7ea40c81efdd3a2
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 18 01:10:57 2019

[buildbucket] Fix bot_dimensions population

Fix "if build.proto" check.

Bug: 917851
Change-Id: Iaaee50177d5de06c7ed41f56a97efa5ac8001372
Reviewed-on: https://chromium-review.googlesource.com/c/1419306
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20067}
[modify] https://crrev.com/24ce520ae6577c5693e5e689d7ea40c81efdd3a2/appengine/cr-buildbucket/swarming/swarming.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/62a2384f0fd72f5ed270876043670ae6cdd78ae3

commit 62a2384f0fd72f5ed270876043670ae6cdd78ae3
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 18 06:39:25 2019

[buildbucket] Backfill proto in all completed builds

Previous implementation of proto backfill assumed no rollbacks.
A rollback happened.

Simply update all completed builds.
This will deployed some time after buildbucket starts writing proto property
in new builds.

Also fix the update code itself. It was not properly tested before.

R=vadimsh@chromium.org

Bug: 917851
Change-Id: I1e3d07fabde53b22c92a6b7575e829adc755da20
Reviewed-on: https://chromium-review.googlesource.com/c/1420020
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20071}
[modify] https://crrev.com/62a2384f0fd72f5ed270876043670ae6cdd78ae3/appengine/cr-buildbucket/fix_builds.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c7df681cc225c086ffe925fd9162bfd82c1e95b1

commit c7df681cc225c086ffe925fd9162bfd82c1e95b1
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 18 17:07:17 2019

[buildbucket] Do not clear cancel_reason

in status_to_v2, do not clear cancel_reason.
In general, do not clear fields that we do not touch.

TBR=vadimsh@chromium.org

Bug: 917851
Change-Id: I211447d0299c2c8c6f0bccf449b446865099914b
Reviewed-on: https://chromium-review.googlesource.com/c/1421858
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20073}
[modify] https://crrev.com/c7df681cc225c086ffe925fd9162bfd82c1e95b1/appengine/cr-buildbucket/v2/builds.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ba39b93b51dc36f8891fd0738c620d9c6a257506

commit ba39b93b51dc36f8891fd0738c620d9c6a257506
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 18 23:06:41 2019

[buildbucket] Update tests to use protos

Update all buildbucket tests to use Build proto instead of to-be-deprecated
properties.

Add test_util.build utility function that accepts a kwargs of build proto
fields, supplies reasonable defaults and returns a model.Build.
Update all tests to use it.
This overall improves tests because they specify only the parts of the build
that they actually care about, and deal only with build proto fields that
will stay for a long time (they are public API).
In particular, remove almost all directl model.Build() invocations
from tests.

This CL does not change the app behavior, but only updates tests.

Bug: 917851
Change-Id: I3734d1bfd3424170ba274865ca1859beea1219a8
Reviewed-on: https://chromium-review.googlesource.com/c/1422563
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20087}
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/backfill_tag_index_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/test_util.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/service.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/metrics_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/expiration_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/bulkproc_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/user_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/model_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/ba39b93b51dc36f8891fd0738c620d9c6a257506/appengine/cr-buildbucket/model.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5117412bc79f9e4f14aac2b691ea6bd71a6880bf

commit 5117412bc79f9e4f14aac2b691ea6bd71a6880bf
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jan 22 18:04:08 2019

[buildbucket] Switch to status v2

Rename model.Build.status to status_legacy and make it derived from
proto.status.

Remove cyclic dependency between v2.py and model.py and remove no longer needed
v2.py imports from tests.

R=vadimsh@chromium.org

Bug: 917851
Change-Id: I314368681b54aeea86278a6aa6d0456c8026f032
Reviewed-on: https://chromium-review.googlesource.com/c/1422564
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20113}
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/backfill_tag_index_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/metrics_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/v2/test/builds_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/model_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/events.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/expiration.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/v2/__init__.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/test_util.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/service.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/metrics.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/user_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/fix_builds.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/search.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/test/logdog_test.py
[modify] https://crrev.com/5117412bc79f9e4f14aac2b691ea6bd71a6880bf/appengine/cr-buildbucket/creation.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c6967b03f217f8d9e3121aff6b77da7c536062ca

commit c6967b03f217f8d9e3121aff6b77da7c536062ca
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jan 22 19:16:06 2019

[buildbucket] Use legacy status in peek() query

peek() requires a composite index. Switch back to legacy status for peek.

R=vadimsh@chromium.org

Bug: 917851
Change-Id: I2352a0f026a6a2765faed654fa2153093fd337fe
Reviewed-on: https://chromium-review.googlesource.com/c/1427139
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20114}
[modify] https://crrev.com/c6967b03f217f8d9e3121aff6b77da7c536062ca/appengine/cr-buildbucket/service.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0a093b4031976a4093e9aa5e0c248068548f90fd

commit 0a093b4031976a4093e9aa5e0c248068548f90fd
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jan 23 00:16:35 2019

[buildbucket][bulkproc] Reduce batch size

Some shards are failing with RequestTooLargeError because of ndb's automatic
request batching. Reduce batch size from 3k to 1k.

R=vadimsh@chromium.org

Bug: 917851
Change-Id: Icd1dbba0e9daca1dad6aabe2983799957b2bc53e
Reviewed-on: https://chromium-review.googlesource.com/c/1428279
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20126}
[modify] https://crrev.com/0a093b4031976a4093e9aa5e0c248068548f90fd/appengine/cr-buildbucket/bulkproc.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0a093b4031976a4093e9aa5e0c248068548f90fd

commit 0a093b4031976a4093e9aa5e0c248068548f90fd
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jan 23 00:16:35 2019

[buildbucket][bulkproc] Reduce batch size

Some shards are failing with RequestTooLargeError because of ndb's automatic
request batching. Reduce batch size from 3k to 1k.

R=vadimsh@chromium.org

Bug: 917851
Change-Id: Icd1dbba0e9daca1dad6aabe2983799957b2bc53e
Reviewed-on: https://chromium-review.googlesource.com/c/1428279
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20126}
[modify] https://crrev.com/0a093b4031976a4093e9aa5e0c248068548f90fd/appengine/cr-buildbucket/bulkproc.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fc452f37d97eedc4624dea0c59c4fc061d8a9dbb

commit fc452f37d97eedc4624dea0c59c4fc061d8a9dbb
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jan 23 03:18:32 2019

[buildbucket] Pop result_details['annotations']

Apparently there are still some builds with deprecated and huge "annotations"
in result_details. This interferes with the currently running MR.
Pop annotations, they should not be there anyway.

Bug: 917851
Change-Id: I04ad5417dcefe21ae564bebf2fd5aa750b17cabe
Reviewed-on: https://chromium-review.googlesource.com/c/1429421
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20134}
[modify] https://crrev.com/fc452f37d97eedc4624dea0c59c4fc061d8a9dbb/appengine/cr-buildbucket/fix_builds.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fc452f37d97eedc4624dea0c59c4fc061d8a9dbb

commit fc452f37d97eedc4624dea0c59c4fc061d8a9dbb
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jan 23 03:18:32 2019

[buildbucket] Pop result_details['annotations']

Apparently there are still some builds with deprecated and huge "annotations"
in result_details. This interferes with the currently running MR.
Pop annotations, they should not be there anyway.

Bug: 917851
Change-Id: I04ad5417dcefe21ae564bebf2fd5aa750b17cabe
Reviewed-on: https://chromium-review.googlesource.com/c/1429421
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20134}
[modify] https://crrev.com/fc452f37d97eedc4624dea0c59c4fc061d8a9dbb/appengine/cr-buildbucket/fix_builds.py