https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/1211002

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a53baf40390dbfe4a586e0cbe3092bf924bf4ed9

commit a53baf40390dbfe4a586e0cbe3092bf924bf4ed9
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Sep 06 19:03:57 2018

[buildbucket] Add Step.Log.url

Users need to fetch step log contents using LogDog API. Currently Buildbucket
does not expose logdog address, so some users have to resort to separately
fetching original annotation proto which contains logdog stream addresses.
This info should be available at the buildbucket level.

Add Step.Log.url field that is a URL of the log, as opposed to the URL of the
log viewer. This is intentionally less structured (a string, not a proto).
+ minimal design / API surface
+ agnostic of log storage
+ allows both absolute URLs (with logdog project and prefix), and relative
  (relative to Build.Infra.logdog.*). We will expose absolute form anyway, though,
  because it is simpler. Impl may decide to store relative URLs if they are relative
  to the common prefix.
- requires more parsing. It should not be a big deal because the field will be
  consumed only by programs (not humans) that need to fetch log contents. They will
  use a logdog client anyway, which typically has a parser. Otherwise, parsing
  this string isn't hard.
  It is hard in SQL/BQ, but such use case is not anticipated.

Rejected design: stream name field that is supposed to be used in conjunction
with Build.Infra.logdog.{host, project, prefix} fields. Mostly because in
go/buildbucket-api-v2 iannucci@ suggested that in the future we might want to
support lightweight steps which happen on different machines, with different
logdog prefixes. The design proposed in this CL does not have this problem.

Bug:  879242 
Change-Id: Ieb33edfac20edac0a5d9060ba6592ce9a7b40980
Reviewed-on: https://chromium-review.googlesource.com/1211002
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/a53baf40390dbfe4a586e0cbe3092bf924bf4ed9/buildbucket/proto/pb.discovery.go
[modify] https://crrev.com/a53baf40390dbfe4a586e0cbe3092bf924bf4ed9/buildbucket/proto/step.pb.go
[modify] https://crrev.com/a53baf40390dbfe4a586e0cbe3092bf924bf4ed9/buildbucket/proto/step.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ab7cd24da6100ea7a95ec49ce7656c2b081cb106

commit ab7cd24da6100ea7a95ec49ce7656c2b081cb106
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Sep 06 20:57:09 2018

[buildbucket] Populate Build.steps.logs.url

Populate Build.steps.logs.url field when converting builds from annotations.
See also https://chromium-review.googlesource.com/c/infra/infra/+/1211689

R=jchinlee@chromium.org, tandrii@chromium.org

Bug:  879242 
Change-Id: I84971597e43e16608cd87b864bcc6c323f83b882
Reviewed-on: https://chromium-review.googlesource.com/1211646
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/ab7cd24da6100ea7a95ec49ce7656c2b081cb106/buildbucket/annotations.go
[modify] https://crrev.com/ab7cd24da6100ea7a95ec49ce7656c2b081cb106/buildbucket/testdata/annotations/expected_steps.pb.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c

commit 5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Sep 06 21:15:52 2018

[buildbucket] Populate Build.steps.logs.url

Populate Build.steps.logs.url field when converting builds from annotations.
Also update compiled proto files.

Bug:  879242 
Change-Id: I0bbd6f62ab6bd64c0d054b89be00ca70207ef0d4
Reviewed-on: https://chromium-review.googlesource.com/1211689
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17428}
[modify] https://crrev.com/5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c/appengine/cr-buildbucket/annotations.py
[modify] https://crrev.com/5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c/infra/libs/buildbucket/proto/rpc_prpc_pb2.py
[modify] https://crrev.com/5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c/infra/libs/buildbucket/proto/rpc_pb2.py
[modify] https://crrev.com/5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c/infra/libs/buildbucket/proto/step_pb2.py
[modify] https://crrev.com/5d67ea11f04c0ff7d30bc5b10c93cd2ca086669c/appengine/cr-buildbucket/test/expected_steps.pb.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/071ab700e67256e3c5c71718186c59dc59117d94

commit 071ab700e67256e3c5c71718186c59dc59117d94
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Sep 07 18:41:46 2018

[buildbucket] Compress BuildSteps.step_container

Since https://chromium-review.googlesource.com/c/infra/infra/+/1211689
some builds no longer fit the build step limit.

The step URLs are highly compressible. Use ndb-provided compression and double
byte limit for uncompressed steps.

R=jchinlee@chromium.org

Bug:  879242 
Change-Id: If1c25fbe5f69c775d2cdfff5bf2167fbc244b222
Reviewed-on: https://chromium-review.googlesource.com/1213945
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17445}
[modify] https://crrev.com/071ab700e67256e3c5c71718186c59dc59117d94/appengine/cr-buildbucket/model.py

for all LUCI builds that complete starting from now, step urls are available

https://cr-buildbucket.appspot.com/rpcexplorer/services/buildbucket.v2.Builds/SearchBuilds?request={%20%20%20%20%22predicate%22:%20{%20%20%20%20%20%20%20%20%22builder%22:%20{%20%20%20%20%20%20%20%20%20%20%20%20%22project%22:%20%22chromium%22,%20%20%20%20%20%20%20%20%20%20%20%20%22bucket%22:%20%22try%22,%20%20%20%20%20%20%20%20%20%20%20%20%22builder%22:%20%22linux_chromium_rel_ng%22%20%20%20%20%20%20%20%20}%20%20%20%20},%20%20%20%20%22fields%22:%20%22builds.*.id,builds.*.steps.*.logs.*.url%22}