i will export completed step events there too, so that we export of these two very-related event type come from the same place.

Also this will simplify ACLs: buildbucket service account can write to 2 tables, as opposed to buildbucket can write to completed_builds and a bunch of swarming system service accounts can write to completed steps.

Also this will enable monitoring of BQ export, as a part of overall buildbucket monitoring. Related: bq exporting from kitchen broke on dec 8 and no one noticed.

+katthomas (now idea how could I forgot to add her before)

too many teams/people want this fixed, so prioritizing this

proposal how to represent properties in BQ: https://docs.google.com/document/d/1PccZ62hmF8QQHHa7GDF9NriCGS5mm-uL_NcXBp2UCDI/edit#

 Issue 791059  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fee5568300ea8873fac86ce12c9e31310f511b48

commit fee5568300ea8873fac86ce12c9e31310f511b48
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 23 19:35:51 2018

[buildbucket] do not call HasField on primitive fields

(auto_builder_dimension == YES) => HasField('auto_builder_dimension')

Bug:  788034 
Change-Id: I025acceff388cb4306cd4a2cd2283ae201cd4958
Reviewed-on: https://chromium-review.googlesource.com/934884
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/fee5568300ea8873fac86ce12c9e31310f511b48/appengine/cr-buildbucket/swarming/swarming.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/af8fc27650214dfb3385bdd354600e75ee02c35e

commit af8fc27650214dfb3385bdd354600e75ee02c35e
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 23 19:56:51 2018

[bq] add support for google.protobuf.Struct

Earlier it was decided not to support google.protobuf.Struct.
Since then this decision was reversed. Persist it as a JSONPB string.

Bug:  788034 
Change-Id: Ie651ebc7dac6715b52a98a03e8c7796451b8940a
Reviewed-on: https://chromium-review.googlesource.com/932945
Reviewed-by: Katie Thomas <katthomas@google.com>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/common/bq/eventupload.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/common/bq/eventupload_test.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/common/bq/testdata/testmessage.pb.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/common/bq/testdata/testmessage.proto
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/tools/cmd/bqschemaupdater/doc.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/tools/cmd/bqschemaupdater/main_test.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/tools/cmd/bqschemaupdater/schema.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/tools/cmd/bqschemaupdater/testdata/event.desc
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/tools/cmd/bqschemaupdater/testdata/event.pb.go
[modify] https://crrev.com/af8fc27650214dfb3385bdd354600e75ee02c35e/tools/cmd/bqschemaupdater/testdata/event.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/cfee26f9d51715fca0bd91042be8cb8992afe17d

commit cfee26f9d51715fca0bd91042be8cb8992afe17d
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 23 20:13:02 2018

[bqschemaupdater] ignore empty messages

BigQuery does not like RECORD fields with no subfields.
Ignore such fields.

Bug:  788034 
Change-Id: I4b3956e6e589631eee48c0358185ab357f09ebc2
Reviewed-on: https://chromium-review.googlesource.com/932966
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>

[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/common/bq/eventupload.go
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/common/bq/testdata/testmessage.pb.go
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/common/bq/testdata/testmessage.proto
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/tools/cmd/bqschemaupdater/doc.go
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/tools/cmd/bqschemaupdater/schema.go
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/tools/cmd/bqschemaupdater/testdata/event.desc
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/tools/cmd/bqschemaupdater/testdata/event.pb.go
[modify] https://crrev.com/cfee26f9d51715fca0bd91042be8cb8992afe17d/tools/cmd/bqschemaupdater/testdata/event.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/4d83d4f07335f14e9b6451b52f1a84455998cc8b

commit 4d83d4f07335f14e9b6451b52f1a84455998cc8b
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 23 21:48:12 2018

[bqschemaupdater] remove -dry-run flag

It is not needed with schema diff and confirmations.

Bug:  788034 
Change-Id: I7335e0479239ac4e8b4787e30ecbe900cbbea796
Reviewed-on: https://chromium-review.googlesource.com/934148
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>

[delete] https://crrev.com/934d5f0d282a9be41f0ef85bc3d5dda31cd58eb0/tools/cmd/bqschemaupdater/dry_run_table_store.go
[delete] https://crrev.com/934d5f0d282a9be41f0ef85bc3d5dda31cd58eb0/tools/cmd/bqschemaupdater/dry_run_table_store_test.go
[modify] https://crrev.com/4d83d4f07335f14e9b6451b52f1a84455998cc8b/tools/cmd/bqschemaupdater/main.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/b9c727724a42db5f6c5c0c5302feb57b1a696b87

commit b9c727724a42db5f6c5c0c5302feb57b1a696b87
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 23 23:28:28 2018

[bqschemaupdater] refine doc

Refine doc explaining proto->bq schema mapping.

TBR=katthomas@google.com
Bug:  788034 
Change-Id: I04e264ff51244bd90e236e7d43f1dcdf239b7016
Reviewed-on: https://chromium-review.googlesource.com/935663
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/b9c727724a42db5f6c5c0c5302feb57b1a696b87/tools/cmd/bqschemaupdater/doc.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8fec1e6b136952926aa60c22d7a4e4ebe1df0ef9

commit 8fec1e6b136952926aa60c22d7a4e4ebe1df0ef9
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 23 23:37:38 2018

Revert "[bqschemaupdater] make schema text more compact"

Commit 586804aac3fc3c0f0617edea9c5948ec092bf693 had a bad merged and reverted
unrelated changes. Reland those changes.

TBR=katthomas@google.com
Bug:  788034 
Change-Id: I7742014f17b9ac1c9b22b3389798913fba410481
Reviewed-on: https://chromium-review.googlesource.com/935374
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/8fec1e6b136952926aa60c22d7a4e4ebe1df0ef9/common/bq/eventupload.go
[modify] https://crrev.com/8fec1e6b136952926aa60c22d7a4e4ebe1df0ef9/common/bq/testdata/testmessage.pb.go
[modify] https://crrev.com/8fec1e6b136952926aa60c22d7a4e4ebe1df0ef9/common/bq/testdata/testmessage.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/76bd5779028e546c750b3ab6f15a69c7241c59f3

commit 76bd5779028e546c750b3ab6f15a69c7241c59f3
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Feb 26 14:08:30 2018

[bq] ignore repeated empty values

Implement ignoring of empty messages at the getFieldInfos
level, so that it works for repeated empty messages.

Bug:  788034 
Change-Id: I47eae712c51bdd6f12edf9c8e8a035cd4dcb295b
Reviewed-on: https://chromium-review.googlesource.com/935829
Reviewed-by: Katie Thomas <katthomas@google.com>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/76bd5779028e546c750b3ab6f15a69c7241c59f3/common/bq/eventupload.go
[modify] https://crrev.com/76bd5779028e546c750b3ab6f15a69c7241c59f3/common/bq/eventupload_test.go
[modify] https://crrev.com/76bd5779028e546c750b3ab6f15a69c7241c59f3/common/bq/testdata/testmessage.pb.go
[modify] https://crrev.com/76bd5779028e546c750b3ab6f15a69c7241c59f3/common/bq/testdata/testmessage.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ae6e2a81ffd4fc0dfb180bf259afa1270c4d556d

commit ae6e2a81ffd4fc0dfb180bf259afa1270c4d556d
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Feb 26 18:36:05 2018

[bigquery] Fix message_to_dict

Fix bugs / add missing logic:
- save an enum value as a string
- do not ignore zero primitive values, including enums
- ignore fields of message types that would result in RECORD without sub-fields
- save google.protobuf.Struct as JSONPB string
- ignore empty arrays

Enhance tests to cover these cases.

Bug:  788034 
Change-Id: Ibd0f2d9ddd9496f12865fb4cfc2e43e97c4268af
Reviewed-on: https://chromium-review.googlesource.com/936185
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>

[modify] https://crrev.com/ae6e2a81ffd4fc0dfb180bf259afa1270c4d556d/packages/infra_libs/infra_libs/bigquery/test/testmessage.proto
[modify] https://crrev.com/ae6e2a81ffd4fc0dfb180bf259afa1270c4d556d/packages/infra_libs/infra_libs/bigquery/helper.py
[modify] https://crrev.com/ae6e2a81ffd4fc0dfb180bf259afa1270c4d556d/packages/infra_libs/infra_libs/bigquery/test/testmessage_pb2.py
[modify] https://crrev.com/ae6e2a81ffd4fc0dfb180bf259afa1270c4d556d/packages/infra_libs/infra_libs/bigquery/test/helper_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6

commit dc69e3c1f2585115ad3f7a81fdbfeebc971515e6
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Feb 26 19:27:45 2018

[buildbucket] add a subset of v2 protobufs

Add subsets of common.proto, build.proto and step.proto, required to start
saving completed builds in BigQuery tables in v2 format.

Bug:  788034 
Change-Id: Id1860eb14288c4e0f0967a5f0367f9512f78df90
Reviewed-on: https://chromium-review.googlesource.com/934863
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[add] https://crrev.com/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6/appengine/cr-buildbucket/proto/build_pb2.py
[add] https://crrev.com/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6/appengine/cr-buildbucket/proto/common_pb2.py
[add] https://crrev.com/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6/appengine/cr-buildbucket/proto/step.proto
[add] https://crrev.com/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6/appengine/cr-buildbucket/proto/common.proto
[add] https://crrev.com/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6/appengine/cr-buildbucket/proto/build.proto
[add] https://crrev.com/dc69e3c1f2585115ad3f7a81fdbfeebc971515e6/appengine/cr-buildbucket/proto/step_pb2.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/0e767d8df4e2c5e223454d7fa1353fd55d556bfd

commit 0e767d8df4e2c5e223454d7fa1353fd55d556bfd
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Feb 27 19:56:48 2018

[config] change the root of proto compilation

Proto descriptions in pb2 files have proto file names relative to proto dir, so
just "project_config.proto". By doing that, they reserve than name and clients
using config components cannot use it.

Change the root of proto compilation, so that the file name is
"components/config/project_config.proto", consistent with the fact that other
components assume "components" is in import path.

Bug:  788034 
Change-Id: I4a4ed258abcc63c0444198e0f8148ba27b048da0
Reviewed-on: https://chromium-review.googlesource.com/938327
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/0e767d8df4e2c5e223454d7fa1353fd55d556bfd/appengine/components/components/config/proto/Makefile
[modify] https://crrev.com/0e767d8df4e2c5e223454d7fa1353fd55d556bfd/appengine/components/components/config/proto/project_config_pb2.py
[modify] https://crrev.com/0e767d8df4e2c5e223454d7fa1353fd55d556bfd/appengine/components/components/config/proto/service_config_pb2.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/5dc02ee66a966b27837ed040a2d4a2d65a08e687

commit 5dc02ee66a966b27837ed040a2d4a2d65a08e687
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Feb 28 03:01:55 2018

[gerrit] make functions async

Also pass query string params in "params" param, not url.

Bug:  788034 
Change-Id: I057f9e75eac608fe044d1d416de94322f6d49848
Reviewed-on: https://chromium-review.googlesource.com/940632
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/5dc02ee66a966b27837ed040a2d4a2d65a08e687/appengine/components/components/gerrit.py
[modify] https://crrev.com/5dc02ee66a966b27837ed040a2d4a2d65a08e687/appengine/components/components/gerrit_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/06231e7e2cc52f49f15ba734874a31e22109fdc3

commit 06231e7e2cc52f49f15ba734874a31e22109fdc3
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Feb 28 04:46:32 2018

[deps.pyl] roll protobuf

roll protobuf 3.3.0 -> 3.5.1
3.5.1 has google.protobuf.struct_pb2.Struct.update(dict)

Bug:  788034 
Change-Id: I15fe1d05aa5024fe28a3a799da0cef146dace205
Reviewed-on: https://chromium-review.googlesource.com/938755
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/06231e7e2cc52f49f15ba734874a31e22109fdc3/bootstrap/deps.pyl
[modify] https://crrev.com/06231e7e2cc52f49f15ba734874a31e22109fdc3/appengine/cr-buildbucket/test/config_test.py
[modify] https://crrev.com/06231e7e2cc52f49f15ba734874a31e22109fdc3/infra/tools/send_monitoring_event/common.py
[modify] https://crrev.com/06231e7e2cc52f49f15ba734874a31e22109fdc3/appengine/monorail/services/client_config_svc.py
[modify] https://crrev.com/06231e7e2cc52f49f15ba734874a31e22109fdc3/packages/infra_libs/infra_libs/bigquery/test/helper_test.py
[modify] https://crrev.com/06231e7e2cc52f49f15ba734874a31e22109fdc3/infra/tools/send_monitoring_event/test/common_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9fe5fbdc90918ad1bd16d0bc43d5642c53364048

commit 9fe5fbdc90918ad1bd16d0bc43d5642c53364048
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Feb 28 04:55:53 2018

[buildbucket] split events.py

split events.py to events.py and notifications.py.
Next CL will add bigquery.py, also called from events.py

Bug:  788034 
Change-Id: I946f33111800ca6a62caa10e3b58eaeac63cff7a
Reviewed-on: https://chromium-review.googlesource.com/940812
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/9fe5fbdc90918ad1bd16d0bc43d5642c53364048/appengine/cr-buildbucket/events.py
[modify] https://crrev.com/9fe5fbdc90918ad1bd16d0bc43d5642c53364048/appengine/cr-buildbucket/handlers.py
[add] https://crrev.com/9fe5fbdc90918ad1bd16d0bc43d5642c53364048/appengine/cr-buildbucket/notifications.py
[rename] https://crrev.com/9fe5fbdc90918ad1bd16d0bc43d5642c53364048/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/9fe5fbdc90918ad1bd16d0bc43d5642c53364048/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/9fe5fbdc90918ad1bd16d0bc43d5642c53364048/appengine/cr-buildbucket/test/service_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5fd15d3e1aa94daf3aca52efd02afd5ff57dd421

commit 5fd15d3e1aa94daf3aca52efd02afd5ff57dd421
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Feb 28 17:38:43 2018

[bqschemaupdater] add -partition-field flag

Bug:  788034 
Change-Id: Ie8b8b2fafecff0892c2778e60d71064efa54a019
Reviewed-on: https://chromium-review.googlesource.com/940427
Reviewed-by: Katie Thomas <katthomas@google.com>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/5fd15d3e1aa94daf3aca52efd02afd5ff57dd421/tools/cmd/bqschemaupdater/main.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/9fd58c83df398e1a3e84a3001bc7f02a260f69b1

commit 9fd58c83df398e1a3e84a3001bc7f02a260f69b1
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Feb 28 23:24:03 2018

[protobuf] roll to 3.5.1

Do not redo glyco-specific local modifications.

Bug:  788034 
Change-Id: I37cb653506b98cd40a9df2b8f7ff021aca61b9fc
Reviewed-on: https://chromium-review.googlesource.com/941554
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/README.swarming
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/__init__.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/__init__.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/api_pb2.py
[delete] https://crrev.com/886d1c5e8b02df3ed771982802dae1dca93129df/client/third_party/google/protobuf/compiler/__init__.py
[delete] https://crrev.com/886d1c5e8b02df3ed771982802dae1dca93129df/client/third_party/google/protobuf/compiler/plugin_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/descriptor.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/descriptor_database.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/descriptor_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/descriptor_pool.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/field_mask_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/any_test_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/api_implementation.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/containers.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/decoder.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/encoder.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/factory_test2_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/more_extensions_dynamic_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/python_message.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/test_util.py
[add] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/testing_refleaks.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/type_checkers.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/internal/well_known_types.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/json_format.py
[add] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/map_proto2_unittest_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/map_unittest_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/message.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/message_factory.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/reflection.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/source_context_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/symbol_database.py
[add] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/test_messages_proto2_pb2.py
[add] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/test_messages_proto3_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/text_format.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/type_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/unittest_pb2.py
[modify] https://crrev.com/9fd58c83df398e1a3e84a3001bc7f02a260f69b1/client/third_party/google/protobuf/unittest_proto3_arena_pb2.py
[delete] https://crrev.com/886d1c5e8b02df3ed771982802dae1dca93129df/client/third_party/google/protobuf/util/__init__.py
[delete] https://crrev.com/886d1c5e8b02df3ed771982802dae1dca93129df/client/third_party/google/protobuf/util/json_format_proto3_pb2.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a50e89a67036c4fa25ff39be911d1590cb45150c

commit a50e89a67036c4fa25ff39be911d1590cb45150c
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 01 01:25:32 2018

[buildbucket] update v2 protos

Update the subset of v2 protos.
Exclude logdog stream name from step log because it is relevant in BQ, or
at least not urgent.

TBR=iannucci@chromium.org
Bug:  788034 
Change-Id: I5a99b0bf4424091db746785f81b4df4c5e3ca13e
Reviewed-on: https://chromium-review.googlesource.com/942082
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/a50e89a67036c4fa25ff39be911d1590cb45150c/appengine/cr-buildbucket/proto/build_pb2.py
[modify] https://crrev.com/a50e89a67036c4fa25ff39be911d1590cb45150c/appengine/cr-buildbucket/proto/step_pb2.py
[modify] https://crrev.com/a50e89a67036c4fa25ff39be911d1590cb45150c/appengine/cr-buildbucket/proto/build.proto
[modify] https://crrev.com/a50e89a67036c4fa25ff39be911d1590cb45150c/appengine/cr-buildbucket/proto/step.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/22061e10a0f6c07fe3324c1c2a24118c56f7f26d

commit 22061e10a0f6c07fe3324c1c2a24118c56f7f26d
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 01 01:36:52 2018

[buidlbucket] add v2.py

v2.py's docstring:
"""V2-specific code.

This module contains Buildbucket v2 specific code, while the rest of the
code in this app is v1. In particular, this file implements a function that
converts a v1 Build datastore entity to buildbucket.v2.Build message.
"""

Will be used to save completed builds into BigQuery.

Bug:  788034 
Change-Id: I823ed08c29f9432f59e73906e9cd997a7bb64834
Reviewed-on: https://chromium-review.googlesource.com/942223
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/22061e10a0f6c07fe3324c1c2a24118c56f7f26d/appengine/cr-buildbucket/service.py
[add] https://crrev.com/22061e10a0f6c07fe3324c1c2a24118c56f7f26d/appengine/cr-buildbucket/test/v2_test.py
[modify] https://crrev.com/22061e10a0f6c07fe3324c1c2a24118c56f7f26d/appengine/cr-buildbucket/model.py
[add] https://crrev.com/22061e10a0f6c07fe3324c1c2a24118c56f7f26d/appengine/cr-buildbucket/v2.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6f81f2c7e5690f08e452284007d832a0ace36541

commit 6f81f2c7e5690f08e452284007d832a0ace36541
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 01 02:45:52 2018

Roll infra/luci/ 886d1c5e8..9fd58c83d (1 commit)

https://chromium.googlesource.com/infra/luci/luci-py//+log/886d1c5e8b02..9fd58c83df39

$ git log 886d1c5e8..9fd58c83d --date=short --no-merges --format='%ad %ae %s'
2018-02-28 nodir [protobuf] roll to 3.5.1

Created with:
  roll-dep infra/luci

TBR=vadimsh@chromium.org
Bug:  788034 
Change-Id: Iaaca508745022cc0516c4a2007786a4475f1c12e
NOTREECHECKS=true
Reviewed-on: https://chromium-review.googlesource.com/942487
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/6f81f2c7e5690f08e452284007d832a0ace36541/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ba848a628e35dbc3097483fb901ce0594afd5445

commit ba848a628e35dbc3097483fb901ce0594afd5445
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 01 06:49:33 2018

[buildbucket][v2] exclude more tags

Exclude tags added by swarmbucket

Bug:  788034 
Change-Id: Icead68f7682c7b54f0fb0b2877093af910e34482
Reviewed-on: https://chromium-review.googlesource.com/942469
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/ba848a628e35dbc3097483fb901ce0594afd5445/appengine/cr-buildbucket/test/v2_test.py
[modify] https://crrev.com/ba848a628e35dbc3097483fb901ce0594afd5445/appengine/cr-buildbucket/v2.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/70e68464b75f632c97db40bc6c050599d9a233db

commit 70e68464b75f632c97db40bc6c050599d9a233db
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Mar 02 05:47:49 2018

[buildbucket] vendor annotations.proto

Vendor a modified version of annotations.proto to be able to parse steps
stored in LogDog.

Bug:  788034 
Change-Id: Ic44c6753ed7cea872b73eb9fd6e0aaff001fa02e
Reviewed-on: https://chromium-review.googlesource.com/944659
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[add] https://crrev.com/70e68464b75f632c97db40bc6c050599d9a233db/appengine/cr-buildbucket/third_party/annotations.proto
[add] https://crrev.com/70e68464b75f632c97db40bc6c050599d9a233db/appengine/cr-buildbucket/third_party/README.md
[add] https://crrev.com/70e68464b75f632c97db40bc6c050599d9a233db/appengine/cr-buildbucket/third_party/annotations_pb2.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/bb36a2cea99097bd97c6c954ac3472ce0533be25

commit bb36a2cea99097bd97c6c954ac3472ce0533be25
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Mar 02 20:18:14 2018

[buildbucket] export builds to BigQuery

When completing a build, transactionally create a pull task, tagged with

whether the build is experimental or not.
Add a cron job that processes the pull tasks in batches, either all experimental
or none. For each task, loads a build, converts it to v2 format using v2.py.
Converts all v2 proto messages to BQ rows using big query helper library
and sends to BQ using its REST API. Uses build id as the BQ insert id.

Bug:  788034 
Change-Id: Ie508a0bfe0daac0f3e396e3b2b624fc59b9dddd0
Reviewed-on: https://chromium-review.googlesource.com/942304
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>

[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/events.py
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/handlers.py
[add] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/bqh.py
[add] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/cron.yaml
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/notifications.py
[add] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/packages/infra_libs/infra_libs/bigquery/helper.py
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/queue.yaml
[modify] https://crrev.com/bb36a2cea99097bd97c6c954ac3472ce0533be25/appengine/cr-buildbucket/test/service_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/59d06c1f8be907278d675b3124d8a916d9d321a4

commit 59d06c1f8be907278d675b3124d8a916d9d321a4
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Mar 02 21:53:35 2018

[buildbucket][v2] exclude more tags

Bug:  788034 
Change-Id: I1cb7f6d46613e46438376697cb2c1d079d8ba7db
Reviewed-on: https://chromium-review.googlesource.com/945568
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/59d06c1f8be907278d675b3124d8a916d9d321a4/appengine/cr-buildbucket/test/v2_test.py
[modify] https://crrev.com/59d06c1f8be907278d675b3124d8a916d9d321a4/appengine/cr-buildbucket/v2.py

PSA: http://g/luci-announce/IVdte3E-FRw