New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 833946 link

Starred by 11 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 842920

Blocking:
issue 695105
issue 832580



Sign in to add a comment

design higher-level LUCI config

Project Member Reported by no...@chromium.org, Apr 17 2018

Issue description

cr-buildbucket.cfg and luci-scheduler.cfg files are suffering from duplication inside each file and across the files. In particular, builder names are duplicated. In general, Buildbucket had to come up with "builder mixins" to mitigate repetition. There are other configs that also have builders: cq.cfg, and json files in
https://chromium.googlesource.com/chromium/src/+/a052727f16ac1becbd68deeb2fe5e4bb3c552b5b/testing/buildbot/chromium.mac.json

We need a higher-level config for LUCI services, where a client can define a builder once and specify all of its properties including scheduling and properties. New config format must encapsulation LUCI implementation details, such as the fact that buildbucket and luci-scheduler happen to be separate services.
 
Showing comments 4 - 103 of 103 Older
Owner: s...@google.com
Given the label 'User never visited' for 'smut@chromium.org', I am going to change the owner to 'smut@google.com' :-).

Comment 5 by no...@chromium.org, Apr 18 2018

Owner: smut@chromium.org
smut@google.com has 7 assigned bugs (1 is open (this one)). smut@chromium.org has 1203 bugs (13 open). 
Good point.
Project Member

Comment 7 by bugdroid1@chromium.org, Apr 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c0d4f2b6e03435ba2033f3f21a2231c798e7be19

commit c0d4f2b6e03435ba2033f3f21a2231c798e7be19
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Apr 20 04:13:11 2018

[go] Add skylark to deps.yaml and bump the deps.

Updated repos:
https://code.googlesource.com/gocloud.git/+log/5f3c0cc6ec87..d8617376fb72
https://chromium.googlesource.com/external/github.com/StackExchange/wmi.git/+log/5d049714c4a6..cdffdb33acae
https://chromium.googlesource.com/external/github.com/chromedp/cdproto.git/+log/ad3a5240868f..cc77834c1ac1
https://chromium.googlesource.com/external/github.com/go-sql-driver/mysql.git/+log/1a676ac6e4dc..3287d94d4c6a
https://chromium.googlesource.com/external/github.com/gopherjs/gopherjs.git/+log/558a9132744c..27662f8dae27
https://chromium.googlesource.com/external/github.com/gorilla/websocket.git/+log/eb925808374e..cd94665a654c
https://chromium.googlesource.com/external/github.com/census-instrumentation/opencensus-go.git/+log/5710257e78ab..268594c8cd3c
https://go.googlesource.com/net.git/+log/61147c48b25b..d41e8174641f
https://go.googlesource.com/oauth2.git/+log/921ae394b943..6881fee410a5
https://go.googlesource.com/tools.git/+log/dc06d3e64341..94b14834a201
https://code.googlesource.com/google-api-go-client.git/+log/9f7560f3b05b..b08f8989af48
https://chromium.googlesource.com/external/github.com/google/go-genproto.git/+log/51d0944304c3..7fd901a49ba6
https://chromium.googlesource.com/external/github.com/grpc/grpc-go.git/+log/db0f0713e3a7..4172bfc25e1c

TBR=smut@google.com, nodir@chromium.org
BUG=833946

Change-Id: Ia2c49059d7e432ad372a7f239f897b934e917423
Reviewed-on: https://chromium-review.googlesource.com/1020883
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/c0d4f2b6e03435ba2033f3f21a2231c798e7be19/go/deps.lock
[modify] https://crrev.com/c0d4f2b6e03435ba2033f3f21a2231c798e7be19/go/deps.py
[modify] https://crrev.com/c0d4f2b6e03435ba2033f3f21a2231c798e7be19/go/deps.yaml

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/c420db997df69896284ad4adff29bb1d3b36a604

commit c420db997df69896284ad4adff29bb1d3b36a604
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 25 22:22:28 2018

[skylark] Add skylarkproto package.

Has basic functionality of making proto messages available as skylark values.
Currently only int64 and message-typed fields are supported (as POC). More
features will be added later in follow up commits.

R=smut@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Ia30551bb458d34faf9f124ef260931230c33cd43
Reviewed-on: https://chromium-review.googlesource.com/1020013
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: smut <smut@google.com>

[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/conversions.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/default.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/doc.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/functions.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/loader.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/loader_test.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/message.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/skylark_test.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testdata/int64_repeated.sky
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testdata/int64_singular.sky
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testdata/misc.sky
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testdata/msg_repeated.sky
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testdata/msg_singular.sky
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testdata/to_pbtext.sky
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testprotos/another.pb.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testprotos/another.proto
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testprotos/gen.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testprotos/test.pb.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/testprotos/test.proto
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/type.go
[add] https://crrev.com/c420db997df69896284ad4adff29bb1d3b36a604/skylark/skylarkproto/type_test.go

Project Member

Comment 9 by bugdroid1@chromium.org, Apr 27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ae10a60c1d343edc9a1434cfd350cbd232ca365f

commit ae10a60c1d343edc9a1434cfd350cbd232ca365f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Apr 27 22:44:58 2018

[skylark] Add support for all scalar proto types except 'bytes'.

Namely: int32, uint32, uint64, float, double, string.

'bytes' are special because they look like a slice in the go representation
(unlike other primitive types). Will be handled separately.

BUG=833946
R=smut@google.com

Change-Id: I2b006026fbf1672e013f0e2b29dce2e5e71c616b
Reviewed-on: https://chromium-review.googlesource.com/1031126
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: smut <smut@google.com>

[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/conversions.go
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/default.go
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/doc.go
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/loader_test.go
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/skylark_test.go
[add] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/bool.sky
[add] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/float32.sky
[add] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/float64.sky
[add] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/int_ranges.sky
[rename] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/ints.sky
[rename] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/msg.sky
[rename] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/simple_repeated.sky
[add] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testdata/string.sky
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testprotos/another.pb.go
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testprotos/test.pb.go
[modify] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/skylark/skylarkproto/testprotos/test.proto

Project Member

Comment 14 by bugdroid1@chromium.org, May 10 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e811e4e67677286cf2e78a46e7556196c5686928

commit e811e4e67677286cf2e78a46e7556196c5686928
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu May 10 02:58:20 2018

[skylark] Add support for loading multiple *.proto files.

This is especially useful when protos start to import each other. Switch tests
to use this mechanism to load test.proto too, to be explicit where 'testproto'
symbol comes from.

R=smut@google.com
BUG=833946

Change-Id: I3816f98affbd025cebffe86873b2b59ea6c3dbf1
Reviewed-on: https://chromium-review.googlesource.com/1050572
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: smut <smut@google.com>

[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/doc.go
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/loader_test.go
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/skylark_test.go
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/bool.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/bytes.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/bytes_repeated.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/enums.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/float32.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/float64.sky
[add] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/imported_msg.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/int_ranges.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/ints.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/misc.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/msg.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/msg_repeated.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/nested_msg.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/oneof.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/simple_repeated.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/string.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testdata/to_pbtext.sky
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testprotos/another.pb.go
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testprotos/another.proto
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testprotos/test.pb.go
[modify] https://crrev.com/e811e4e67677286cf2e78a46e7556196c5686928/skylark/skylarkproto/testprotos/test.proto

Cc: estaab@chromium.org
Project Member

Comment 16 by bugdroid1@chromium.org, May 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/77b23ce4c9189484e14035690f439c97f7629c2e

commit 77b23ce4c9189484e14035690f439c97f7629c2e
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri May 11 23:00:07 2018

[skylark] Add awareness of proto2 messages.

For some weird reason proto lib doesn't expose any metadata regardless whether
a message was defined as proto2 or proto3. So instead we detect unsupported
proto2 fields on the fly and fail with a clean error message (instead of
panicing as was happening before).

Such late detection has a benefit of allowing to convert stuff to proto3
gradually (i.e. it is fine to have proto2 fields in proto3 message as long as
skylark code doesn't touch their scalar fields).

R=smut@google.com
BUG=833946

Change-Id: I6b5e32fd497787ff5a624851500f04c21f4ef5bd
Reviewed-on: https://chromium-review.googlesource.com/1056129
Reviewed-by: smut <smut@google.com>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/conversions.go
[modify] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/default.go
[modify] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/doc.go
[add] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/testdata/proto2.sky
[modify] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/testprotos/another.pb.go
[add] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/testprotos/proto2.pb.go
[add] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/testprotos/proto2.proto
[modify] https://crrev.com/77b23ce4c9189484e14035690f439c97f7629c2e/skylark/skylarkproto/testprotos/test.pb.go

Blockedon: 842920
Project Member

Comment 18 by bugdroid1@chromium.org, May 15 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/f68e007be6bd8370c0776fb6c3834cbefdcb199c

commit f68e007be6bd8370c0776fb6c3834cbefdcb199c
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue May 15 22:51:39 2018

[buildbucket] Import project_config.proto from infra.git.

It will be needed for the new config generator. Update 'buildbucket' tool to use
the most recent config protos.

R=nodir@chromium.org, maruel@chromium.org
BUG=833946

Change-Id: I2f02f412a4a6e3867a5bd335d101c6c248315648
Reviewed-on: https://chromium-review.googlesource.com/1060520
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/f68e007be6bd8370c0776fb6c3834cbefdcb199c/buildbucket/client/cmd/buildbucket/builders.go
[delete] https://crrev.com/300322260bf8e274f3d2d0264ffd0decc50c8fd1/buildbucket/client/cmd/buildbucket/proto/gen.go
[delete] https://crrev.com/300322260bf8e274f3d2d0264ffd0decc50c8fd1/buildbucket/client/cmd/buildbucket/proto/project_config.pb.go
[delete] https://crrev.com/300322260bf8e274f3d2d0264ffd0decc50c8fd1/buildbucket/client/cmd/buildbucket/proto/project_config.proto
[add] https://crrev.com/f68e007be6bd8370c0776fb6c3834cbefdcb199c/buildbucket/proto/config/gen.go
[add] https://crrev.com/f68e007be6bd8370c0776fb6c3834cbefdcb199c/buildbucket/proto/config/project_config.pb.go
[add] https://crrev.com/f68e007be6bd8370c0776fb6c3834cbefdcb199c/buildbucket/proto/config/project_config.proto

Project Member

Comment 19 by bugdroid1@chromium.org, May 16 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9

commit 8d4aeead7205b2e87307d741aa67ec4bdf5aeba9
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed May 16 00:28:29 2018

[config] Update luci-config protos imported from luci-py.

They are now proto3.

R=nodir@chromium.org, maruel@chromium.org
BUG=833946

Change-Id: Ib1ccebb3a01bbaf6ae771087577fda902086cac7
Reviewed-on: https://chromium-review.googlesource.com/1060078
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[delete] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/common/proto/config/README.md
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/common/proto/config/generate.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/common/proto/config/project_config.pb.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/common/proto/config/project_config.proto
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/common/proto/config/service_config.pb.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/common/proto/config/service_config.proto
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/config/appengine/backend/datastore/ds_test.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/config/appengine/gaeconfig/default_test.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/config/server/cfgclient/textproto/resolver_test.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/config/validation/handler.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/config/validation/handler_test.go
[modify] https://crrev.com/8d4aeead7205b2e87307d741aa67ec4bdf5aeba9/logdog/appengine/coordinator/coordinatorTest/context.go

Project Member

Comment 20 by bugdroid1@chromium.org, May 19 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ff473fa1fdb40d47884436438fdd17629a4cc064

commit ff473fa1fdb40d47884436438fdd17629a4cc064
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat May 19 02:49:42 2018

[skylark] Implement an interpreter that can load modular scripts.

Raw skylark.ExecFile and skylark.Thread facilities are limited to working with
only one file. This is not sufficient for any non-trivial scripts.

This CL introduces Interpreter struct that knows how work with a tree of skylark
scripts that import each other, as well as protobuf definition files.

It also has a support for a separate tree of script that constitute a "standard
library", with an entry point being in init.sky. All scripts executed by the
interpreter have access to symbols exported by init.sky. The standard library
can be compiled into the interpreter executable, to make it completely
standalone (this will be used in next CLs).

R=smut@google.com
BUG=833946

Change-Id: I1d2b1a8d622abcf14c0f6eaf82111c7058f29ec4
Reviewed-on: https://chromium-review.googlesource.com/1060698
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: smut <smut@google.com>

[add] https://crrev.com/ff473fa1fdb40d47884436438fdd17629a4cc064/skylark/interpreter/fail.go
[add] https://crrev.com/ff473fa1fdb40d47884436438fdd17629a4cc064/skylark/interpreter/fail_test.go
[add] https://crrev.com/ff473fa1fdb40d47884436438fdd17629a4cc064/skylark/interpreter/helpers_test.go
[add] https://crrev.com/ff473fa1fdb40d47884436438fdd17629a4cc064/skylark/interpreter/interpreter.go
[add] https://crrev.com/ff473fa1fdb40d47884436438fdd17629a4cc064/skylark/interpreter/interpreter_test.go

Project Member

Comment 21 by bugdroid1@chromium.org, May 19 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/19b64372d320d5ddb8441b20a0efdbad0842bb93

commit 19b64372d320d5ddb8441b20a0efdbad0842bb93
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat May 19 02:54:52 2018

[skylark] Add 'mutable' helper function.

It allows modules to keep a mutable internal state. Unlike Python (where module
dict is always mutable), Skylark freezes module dicts after module loading. With
'mutable' a module can get a handle to a value that doesn't get frozen.

This would allow to prototype config generation mostly entirely in Skylark. Once
we get to a final state we can move all mutable state into the Go part of the
interpreter (this is how other Skylark embedders deal with it).

R=smut@google.com
BUG=833946

Change-Id: Iff2ee6eb8ca6c8977841f13e90b94f8b9b112309
Reviewed-on: https://chromium-review.googlesource.com/1062909
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: smut <smut@google.com>

[modify] https://crrev.com/19b64372d320d5ddb8441b20a0efdbad0842bb93/skylark/interpreter/interpreter.go
[add] https://crrev.com/19b64372d320d5ddb8441b20a0efdbad0842bb93/skylark/interpreter/mutable.go
[add] https://crrev.com/19b64372d320d5ddb8441b20a0efdbad0842bb93/skylark/interpreter/mutable_test.go

Comment 24 by no...@chromium.org, Jun 17 2018

Issue 828330 has been merged into this issue.

Comment 25 by no...@chromium.org, Jun 17 2018

Blocking: 695105
Owner: s...@google.com
Owner: smut@chromium.org
oh... I see the @google.com vs @chromium.org thing has happened already...
Cc: s...@google.com
Owner: vadimsh@chromium.org
Vadim has taken the OKR about this.
Cc: -s...@google.com smut@chromium.org
Project Member

Comment 30 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e17bf10249cdc4cca9cca44bc178a6a5580568e2

commit e17bf10249cdc4cca9cca44bc178a6a5580568e2
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Nov 21 17:24:17 2018

[starlark] Add 'proto.to_jsonpb' builtin.

Serializes a proto message to JSONPB string.

R=iannucci@chromium.org
BUG=833946

Change-Id: I2fae846ea211be824ce5cc3fd0475f37be7e7862
Reviewed-on: https://chromium-review.googlesource.com/c/1344631
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/e17bf10249cdc4cca9cca44bc178a6a5580568e2/starlark/starlarkproto/functions.go
[add] https://crrev.com/e17bf10249cdc4cca9cca44bc178a6a5580568e2/starlark/starlarkproto/testdata/to_jsonpb.star

Project Member

Comment 31 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/583ce3194f76c7b78cc7f9d595d63fb30de5021a

commit 583ce3194f76c7b78cc7f9d595d63fb30de5021a
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Nov 21 18:01:03 2018

[starlark] Refactor Interpreter to be more structured.

This introduces a notion of a package, which is a bunch of starlark modules
under a single root that can import each other and which are distributed and
versioned as a single unit.

This is roughly similar to how Bazel structures *.bzl code as far as my research
shows.

There are two special packages for now:
  "__main__": an alias for top-level user supplied code, since all code have
  to live in some package.
  "stdlib": a starlark code that implements various global functions, it is
  auto-executed by the interpreter before doing anything else.

Presumably in the (far) future we'll be able to do something like:

  external_cipd_package(alias="stuff", pkg="infra...", version="latest")
  load("@stuff//some/starlark.star", "say_hi")
  say_hi()

(Again, roughly similar to Bazel's http_archive).

Also protobuf handling is now less special. All protobuf messages embedded in
the binary are now represented by a starlark package (conventionally named
"proto"):

  load("@proto//go.chromium.org/luci/.../config.proto", 'buildbucket')
  buildbucket.Bucket(...)

The interpreter itself is oblivious of proto stuff.

R=iannucci@chromium.org
BUG=833946

Change-Id: Ia0608a9ab26fdf025b61a65377722d587ee3fc80
Reviewed-on: https://chromium-review.googlesource.com/c/1344645
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/helpers_test.go
[modify] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/interpreter.go
[modify] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/interpreter_test.go
[add] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/loaders.go
[add] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/loaders_test.go
[modify] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/mutable_test.go
[modify] https://crrev.com/583ce3194f76c7b78cc7f9d595d63fb30de5021a/starlark/interpreter/to_json_test.go

Project Member

Comment 32 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b

commit d1479a9f64f37bd74ff9cab9880069e8a5a47e7b
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Nov 21 18:56:49 2018

[lucicfg] Framework for the config generator.

This CL just implements the barebone structure without any meat on top of it.

High-level configs are defined in LUCI.star file (which may load other Starlark
files via '//<path>').

There's a standard library (presently embedded into the interpreter) that
exposes a bunch of global functions that become our public API. There are
also portions of the standard library that can be explicitly loaded via
load("@stdlib//...") or some other TBD package alias (e.g. "@lowlevel") to
access more advanced public APIs which we tolerate but discourage (and for that
reason hide deeper).

Thus API surfaces are expressed in Starlark. This would allow us, for example,
to consistently generate API docs based on Starlark code ala bazelbuild/skydoc.
Or eventually move them to an external package versioned separately from the
interpreter binary.

To interface with the interpreter guts, stdlib functions use '__native__' object
which is a bundle of all private functions implemented in Go. They may observe
and mutate interpreter state (for example, create config sets and populate them
with protos, though not in this CL). At the end of execution interpreter's State
defines what ends up in generated configs.

R=iannucci@chromium.org
BUG=833946

Change-Id: I004278cf4ad0e8337903f9cd897c87715ec2e753
Reviewed-on: https://chromium-review.googlesource.com/c/1345849
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/cmd/main.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/generator.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/generator_test.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/greetings.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/native.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/protos.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/starlark/assets.gen.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/starlark/assets_test.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/starlark/gen.go
[add] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/lucicfg/starlark/stdlib/builtins.star

Project Member

Comment 33 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d7265271465b7c0a68fd7228f5d26cfa2583f1cc

commit d7265271465b7c0a68fd7228f5d26cfa2583f1cc
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Nov 21 22:37:33 2018

[starlark] Remove 'mutable' builtin.

Mutable state in Starlark module dicts is a pathway to many abilities some
consider to be unnatural.

We'll try to manage without it by explicitly keeping all mutable state on the
Go side. Perhaps in an overly generic way initially, ala TLS store. Still better
than having a mix of mutable and immutable stuff inside deeply nested
starlark.StringDicts.

R=iannucci@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I5ad9d12c6b22e014559d3f34b214f9012971e4a8
Reviewed-on: https://chromium-review.googlesource.com/c/1347193
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/d7265271465b7c0a68fd7228f5d26cfa2583f1cc/starlark/interpreter/interpreter.go
[delete] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/starlark/interpreter/mutable.go
[delete] https://crrev.com/d1479a9f64f37bd74ff9cab9880069e8a5a47e7b/starlark/interpreter/mutable_test.go

Project Member

Comment 34 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/49a1898440c427b3d9eae255faa370c77798fbff

commit 49a1898440c427b3d9eae255faa370c77798fbff
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Nov 21 22:44:13 2018

[starlark] Allow propagating context.Context to builtins.

In lucicfg, don't bind *State to builtins, but pass it through the context. This
allows to load starlark code only once, but execute it multiple times
manipulating different state.

We don't really need it now, but this is cleaner and appears to be a use case
intended by Starlark go library.

R=iannucci@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I28cb8ead5513abd3654052610d72df07a48fc5c0
Reviewed-on: https://chromium-review.googlesource.com/c/1347489
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/49a1898440c427b3d9eae255faa370c77798fbff/lucicfg/generator.go
[modify] https://crrev.com/49a1898440c427b3d9eae255faa370c77798fbff/lucicfg/native.go
[modify] https://crrev.com/49a1898440c427b3d9eae255faa370c77798fbff/starlark/interpreter/helpers_test.go
[modify] https://crrev.com/49a1898440c427b3d9eae255faa370c77798fbff/starlark/interpreter/interpreter.go
[modify] https://crrev.com/49a1898440c427b3d9eae255faa370c77798fbff/starlark/interpreter/interpreter_test.go

Project Member

Comment 35 by bugdroid1@chromium.org, Nov 27

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/651ddd7b25471a93e306020f4038bb8e0063fad5

commit 651ddd7b25471a93e306020f4038bb8e0063fad5
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Nov 27 01:08:43 2018

[starlark] Move default builtins out of Interpreter to its callers.

This allows to declare ALL builtins available to Starlark high-level config
scripts in a single place.

R=iannucci@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: Icb6ec2524f686ae37bc0976f0d6241433e0241b1
Reviewed-on: https://chromium-review.googlesource.com/c/1347206
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/lucicfg/generator.go
[add] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/builtins/doc.go
[rename] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/builtins/fail.go
[rename] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/builtins/fail_test.go
[add] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/builtins/struct.go
[rename] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/builtins/to_json.go
[rename] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/builtins/to_json_test.go
[modify] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/interpreter/helpers_test.go
[modify] https://crrev.com/651ddd7b25471a93e306020f4038bb8e0063fad5/starlark/interpreter/interpreter.go

Project Member

Comment 36 by bugdroid1@chromium.org, Nov 27

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/033b9bca166b0a540cfc47b63937a8d9585e49a5

commit 033b9bca166b0a540cfc47b63937a8d9585e49a5
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Nov 27 01:48:31 2018

[lucicfg] Decouple proto paths as seen from Starlark from luci-go paths.

Protos are lowest layer of the lucicfg public API. Thus their location (among
other things) is part of the API. Loading protos in Starlark using location
within luci-go repository looks a bit weird.

Instead they all are now hosted under more stable load path:
  load("@proto//luci/<service>/project_config.proto", ...)

This would allow us to refactor Go code without breaking Starlark API contract.

Unfortunately, proto package names are also important, and they are still pretty
inconsistent across services.

R=iannucci@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I23eb59550a3fe8680387e77a51dc303a02f4a777
Reviewed-on: https://chromium-review.googlesource.com/c/1347791
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/033b9bca166b0a540cfc47b63937a8d9585e49a5/lucicfg/generator.go
[modify] https://crrev.com/033b9bca166b0a540cfc47b63937a8d9585e49a5/lucicfg/protos.go
[add] https://crrev.com/033b9bca166b0a540cfc47b63937a8d9585e49a5/lucicfg/protos_test.go
[modify] https://crrev.com/033b9bca166b0a540cfc47b63937a8d9585e49a5/starlark/interpreter/loaders.go
[modify] https://crrev.com/033b9bca166b0a540cfc47b63937a8d9585e49a5/starlark/interpreter/loaders_test.go

Project Member

Comment 38 by bugdroid1@chromium.org, Nov 27

Project Member

Comment 39 by bugdroid1@chromium.org, Nov 27

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/36fbecc328f6db9eec0047c4ad5bd6a8e69419c1

commit 36fbecc328f6db9eec0047c4ad5bd6a8e69419c1
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Nov 27 16:50:24 2018

[lucicfg] Move 'State' into its own file, add 'clear_state' for tests.

State will grow noticeably in the future. It is a bit more readable to keep it
in a dedicated file.

R=iannucci@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: Ib60dd21143bd18407c35c99e35e762bd99787f89
Reviewed-on: https://chromium-review.googlesource.com/c/1351554
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/36fbecc328f6db9eec0047c4ad5bd6a8e69419c1/lucicfg/generator.go
[add] https://crrev.com/36fbecc328f6db9eec0047c4ad5bd6a8e69419c1/lucicfg/state.go
[modify] https://crrev.com/36fbecc328f6db9eec0047c4ad5bd6a8e69419c1/lucicfg/testdata/hi.star

Project Member

Comment 41 by bugdroid1@chromium.org, Nov 29

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2f37f58c54c9d3ba76f8c5f13b7e890a72e74f24

commit 2f37f58c54c9d3ba76f8c5f13b7e890a72e74f24
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Nov 29 18:12:34 2018

[starlark] Add 'stacktrace(skip=0)' builtin.

It captures a stack trace at the moment it was called and returns it as a value,
so it can be rendered at some later time.

Will be used to "remember" where various lucicfg rules were defined, so that
at a later time we can emit useful error messages.

R=iannucci@chromium.org
BUG=833946

Change-Id: I52d6cf694fc4a454c1534695a8f9ecc53e154c5c
Reviewed-on: https://chromium-review.googlesource.com/c/1354521
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/2f37f58c54c9d3ba76f8c5f13b7e890a72e74f24/starlark/builtins/stacktrace.go
[add] https://crrev.com/2f37f58c54c9d3ba76f8c5f13b7e890a72e74f24/starlark/builtins/stacktrace_test.go

Project Member

Comment 42 by bugdroid1@chromium.org, Nov 29

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8dbbd471c4ba648397b7b44423a33503ec463b4d

commit 8dbbd471c4ba648397b7b44423a33503ec463b4d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Nov 29 22:21:55 2018

[lucicfg] Add an internal API for emitting errors with captured stack traces.

Also teach Starlark tests to declare they expect an error to happen.

R=iannucci@chromium.org
BUG=833946

Change-Id: If397c472600a973d2c3cf0b3db88f291c94cc7cb
Reviewed-on: https://chromium-review.googlesource.com/c/1354524
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/cmd/main.go
[add] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/errors.go
[modify] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/generator.go
[delete] https://crrev.com/e929ae24be82178d719cb82da40b7123c761d19d/lucicfg/generator_test.go
[delete] https://crrev.com/e929ae24be82178d719cb82da40b7123c761d19d/lucicfg/greetings.go
[modify] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/starlark/stdlib/builtins.star
[add] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/starlark/stdlib/internal/error.star
[modify] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/starlark_test.go
[modify] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/state.go
[add] https://crrev.com/8dbbd471c4ba648397b7b44423a33503ec463b4d/lucicfg/testdata/errors.star
[delete] https://crrev.com/e929ae24be82178d719cb82da40b7123c761d19d/lucicfg/testdata/hi.star

Project Member

Comment 43 by bugdroid1@chromium.org, Nov 29

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/583d93c146cb0cce6785f40be89f33d7a7cb875a

commit 583d93c146cb0cce6785f40be89f33d7a7cb875a
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Nov 29 22:49:46 2018

[lucicfg] Low-level API for generating arbitrary configs.

This is an escape hatch for cases when a high-level declarative API is not
sufficient. It will also be used internally by the high-level generator itself.

Config generating functions are registered as callbacks to allow high-level
configs to be defined in a declarative way, where order of definitions is not
important, as long as the entire high-level config is consistent. To allow this,
we should execute all top-level Starlark code before doing any evaluation of
the state, thus a necessity for delaying generator calls via callbacks.

R=iannucci@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: Ib62640a316fc1255de873defc521be7cf67665bd
Reviewed-on: https://chromium-review.googlesource.com/c/1352371
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/cmd/main.go
[add] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/configset.go
[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/errors.go
[add] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/genctx.go
[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/generator.go
[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/protos_test.go
[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/starlark/stdlib/builtins.star
[modify] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/state.go
[add] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/testdata/config_set.star
[add] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/testproto/gen.go
[add] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/testproto/test.pb.go
[add] https://crrev.com/583d93c146cb0cce6785f40be89f33d7a7cb875a/lucicfg/testproto/test.proto

Project Member

Comment 45 by bugdroid1@chromium.org, Dec 3

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/4b0a053fadfba9ae294733dff376682956d6a369

commit 4b0a053fadfba9ae294733dff376682956d6a369
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Dec 03 19:14:56 2018

[starlark] Expose CaptureStacktrace as public Go function.

This would allow to capture stack traces in Go implementation of various builtin
methods.

R=nodir@chromium.org
BUG=833946

Change-Id: I7ccc5d6604697d046696d6cc7d464f9fccb9edf4
Reviewed-on: https://chromium-review.googlesource.com/c/1357256
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/4b0a053fadfba9ae294733dff376682956d6a369/starlark/builtins/stacktrace.go

Project Member

Comment 46 by bugdroid1@chromium.org, Dec 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/7838e91a61d4bedab52cc2c4539d9f3273507e98

commit 7838e91a61d4bedab52cc2c4539d9f3273507e98
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Dec 04 01:35:22 2018

[lucicfg] Implement 'graph.key(typ1, id1, typ2, id2, ...)' internal API.

Config entities will be defined as named nodes in a DAG. graph.key(...) returns
an opaque label that identifies such a node.

We intern keys to make their identities (~address) match their values 1-to-1.
That allows them to be used as keys in Starlark dicts and Go maps right away,
without any additional code.

R=nodir@chromium.org
BUG=833946

Change-Id: I4db550081e331b2e4c49761ac581f0543814e213
Reviewed-on: https://chromium-review.googlesource.com/c/1357257
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/graph/doc.go
[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/graph/graph.go
[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/graph/key.go
[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/graph/starlark_test.go
[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/graph/testdata/key.star
[modify] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/starlark/assets.gen.go
[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/starlark/stdlib/internal/graph.star
[modify] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/state.go
[add] https://crrev.com/7838e91a61d4bedab52cc2c4539d9f3273507e98/lucicfg/testdata/graph.star

Project Member

Comment 47 by bugdroid1@chromium.org, Dec 4

Project Member

Comment 48 by bugdroid1@chromium.org, Dec 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/fa7cff5e3a65d834aafbc1228149bd1bdade0f90

commit fa7cff5e3a65d834aafbc1228149bd1bdade0f90
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Dec 04 21:52:22 2018

[lucicfg] Implement 'graph.add_edge' API.

Checks for cycles. Doesn't yet check for dangling edges.

R=nodir@chromium.org
BUG=833946

Change-Id: Ia1642181d97632ebaa96c87ae6b564e34471dd2f
Reviewed-on: https://chromium-review.googlesource.com/c/1361566
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/fa7cff5e3a65d834aafbc1228149bd1bdade0f90/lucicfg/graph/graph.go
[modify] https://crrev.com/fa7cff5e3a65d834aafbc1228149bd1bdade0f90/lucicfg/graph/node.go
[add] https://crrev.com/fa7cff5e3a65d834aafbc1228149bd1bdade0f90/lucicfg/graph/testdata/edge.star

Project Member

Comment 49 by bugdroid1@chromium.org, Dec 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/08b8fa0527ebe6b2cc31f03b8720e745858e6bda

commit 08b8fa0527ebe6b2cc31f03b8720e745858e6bda
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Dec 05 00:47:14 2018

[lucicfg] Check that the graph has no dangling edges when finalizing it.

Also:
  1. Make all graph errors implement BacktracableError.
  2. Forbid querying non-finalized graphs.

R=nodir@chromium.org
BUG=833946

Change-Id: Ic912f5379a09cf12490fe3b16d547ad695e01639
Reviewed-on: https://chromium-review.googlesource.com/c/1361576
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/errors.go
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/generator.go
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/graph/graph.go
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/graph/node.go
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/graph/testdata/edge.star
[add] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/graph/testdata/finalization.star
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/graph/testdata/node.star
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/starlark/stdlib/internal/graph.star
[modify] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/state.go
[add] https://crrev.com/08b8fa0527ebe6b2cc31f03b8720e745858e6bda/lucicfg/testdata/graph_dangling_edge.star

Project Member

Comment 50 by bugdroid1@chromium.org, Dec 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/0cc8989a28833d473520233ae2ffff4e6561f4fa

commit 0cc8989a28833d473520233ae2ffff4e6561f4fa
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Dec 05 20:47:28 2018

[lucicfg] Slightly improve error messages by removing redundant info.

Instead of reporting the full key (a list of (kind, id) pairs), we report only
the last pair. The stack trace should provide enough context to understand
what exactly the error refers to.

This reduces how many internal implementation details (e.g. composite keys) we
leak to the outside.

R=nodir@chromium.org
BUG=833946

Change-Id: I4227d5c015b037c498d63f4d4a3104d2aff5b91f
Reviewed-on: https://chromium-review.googlesource.com/c/1362398
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/0cc8989a28833d473520233ae2ffff4e6561f4fa/lucicfg/graph/graph.go
[modify] https://crrev.com/0cc8989a28833d473520233ae2ffff4e6561f4fa/lucicfg/graph/key.go
[modify] https://crrev.com/0cc8989a28833d473520233ae2ffff4e6561f4fa/lucicfg/graph/node.go
[modify] https://crrev.com/0cc8989a28833d473520233ae2ffff4e6561f4fa/lucicfg/graph/testdata/edge.star
[modify] https://crrev.com/0cc8989a28833d473520233ae2ffff4e6561f4fa/lucicfg/graph/testdata/node.star
[modify] https://crrev.com/0cc8989a28833d473520233ae2ffff4e6561f4fa/lucicfg/testdata/graph_dangling_edge.star

Project Member

Comment 51 by bugdroid1@chromium.org, Dec 6

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/0e40450f0980d1ae6aa1ab2150a065f4dd25a798

commit 0e40450f0980d1ae6aa1ab2150a065f4dd25a798
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Dec 06 21:28:50 2018

[lucicfg] Pattern matching for expected errors messages in tests.

Now we can declare that a test is expected to emit an error like:

Traceback (most recent call last):
  ...
  //testdata/errors_like.star:???: in func2
  //testdata/errors_like.star:???: in capture_stack
Error: ??? err

And '...' will be matched to zero or more arbitrary lines and '???' to a word.

This should make tests that examine errors much less brittle.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I2d978ff7df55d1291f92cee36bb6a7f7cc6bf4aa
Reviewed-on: https://chromium-review.googlesource.com/c/1365130
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/0e40450f0980d1ae6aa1ab2150a065f4dd25a798/lucicfg/starlark_test.go
[add] https://crrev.com/0e40450f0980d1ae6aa1ab2150a065f4dd25a798/lucicfg/testdata/errors_like.star

Project Member

Comment 52 by bugdroid1@chromium.org, Dec 6

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e10def5af7048a119f1c9b096564de4d4510fddd

commit e10def5af7048a119f1c9b096564de4d4510fddd
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Dec 06 21:30:40 2018

[starlark] Make starlarktest.RunTests discover tests recursively.

This will allow to group tests by package etc. instead of having a ton of files
in a single directory.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I814b65850b422f48d0040a159317dcb239e065cd
Reviewed-on: https://chromium-review.googlesource.com/c/1365131
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/e10def5af7048a119f1c9b096564de4d4510fddd/starlark/starlarktest/runner.go
[add] https://crrev.com/e10def5af7048a119f1c9b096564de4d4510fddd/starlark/starlarktest/testdata/d/works_too.star

Project Member

Comment 53 by bugdroid1@chromium.org, Dec 7

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/004d0b87580c34886825603c346097c48a7229d1

commit 004d0b87580c34886825603c346097c48a7229d1
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Dec 07 00:07:30 2018

[lucicfg] Implement 'graph.children(...)' query.

Returns direct children of a node with given kind, sorted either by their key
(default), or the order of how they were linked to the parent node (will be
useful for Milo configs, where an order of builders in a builder group matters
for UI).

R=nodir@chromium.org
BUG=833946

Change-Id: I1da12a8174790268a97bc0018faf2e3c5c9e695b
Reviewed-on: https://chromium-review.googlesource.com/c/1363877
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/graph/graph.go
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/graph/key.go
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/graph/node.go
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/graph/starlark_test.go
[add] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/graph/testdata/children.star
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/graph/testdata/finalization.star
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/starlark/stdlib/internal/graph.star
[modify] https://crrev.com/004d0b87580c34886825603c346097c48a7229d1/lucicfg/testdata/graph.star

Project Member

Comment 54 by bugdroid1@chromium.org, Dec 7

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/df9b4818ec08df51f50efe1b61e34d00c644667d

commit df9b4818ec08df51f50efe1b61e34d00c644667d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Dec 07 04:10:02 2018

[lucicfg] Setup basic structure of LUCI rules and generators.

Implement very simple subsets of core.project(), core.bucket() and
core.logdog() as an example.

In particular:
  * All rules (node definition statements) live in //internal/luci/rules/*.star,
    one per file for easier search.
  * //internal/luci/common.star defines keys for all LUCI nodes.
  * //internal/luci/generators.star implements all config generators.
  * //internal/luci/lib/*.star are various utilities.

R=nodir@chromium.org
BUG=833946

Change-Id: I05ca65676508cdafb99241dad05dbf9d00bb27a4
Reviewed-on: https://chromium-review.googlesource.com/c/1364750
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/cmd/main.go
[modify] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/graph/graph.go
[modify] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/protos.go
[modify] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/builtins.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/generator.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/common.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/generators.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/lib/service.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/rules/bucket.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/rules/logdog.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark/stdlib/internal/luci/rules/project.star
[modify] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/starlark_test.go
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/errors/bucket_without_project.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/errors/bucket_without_service.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/errors/logdog_without_project.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/errors/project_redefinition.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/errors/project_without_name.star
[add] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/full_example.star
[rename] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/misc/config_set.star
[rename] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/misc/errors.star
[rename] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/misc/errors_like.star
[rename] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/misc/generator_api.star
[rename] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/misc/graph.star
[rename] https://crrev.com/df9b4818ec08df51f50efe1b61e34d00c644667d/lucicfg/testdata/misc/graph_dangling_edge.star

Project Member

Comment 55 by bugdroid1@chromium.org, Dec 10

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a55c8a7168eac483e5405111adb47b5a70c595f7

commit a55c8a7168eac483e5405111adb47b5a70c595f7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Dec 10 23:50:40 2018

[lucicfg] Boilerplate for command line interface.

Follows current "best practices" of CLIs in luci-go:
  * All subcommands have -log-level and -json-output.
  * Subcommand that do RPCs consistently expose auth and service host options.
  * All Chrome Infra hardcoded stuff is replaceable (if only anyone wanted to
    replace it).
  * Has version string that also ends up in User-Agent strings.
  * Supports required positional arguments and required flags.

R=iannucci@chromium.org
BUG=833946

Change-Id: I792d416e21fe82220eda9df8eed49e729c70d40b
Reviewed-on: https://chromium-review.googlesource.com/c/1368813
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/hardcoded/chromeinfra/chromeinfra.go
[add] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/lucicfg/cli/app.go
[add] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/lucicfg/cli/base.go
[add] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/lucicfg/cli/generate.go
[add] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/lucicfg/cli/validate.go
[add] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/lucicfg/cmd/lucicfg/main.go
[delete] https://crrev.com/e0291a833256290b8551792e6750ec1e02a861c1/lucicfg/cmd/main.go
[add] https://crrev.com/a55c8a7168eac483e5405111adb47b5a70c595f7/lucicfg/version.go

Project Member

Comment 56 by bugdroid1@chromium.org, Dec 12

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2e46e5dee4bb0e3ce4367435865925348fde82bd

commit 2e46e5dee4bb0e3ce4367435865925348fde82bd
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Dec 12 00:02:01 2018

[starlark] Add genstruct(...) and ctor(...) builtins.

They can be used to create structs "tagged" with a symbol. It's not quite
like a typed struct (each individual object still can have arbitrary fields),
but pretty close:

   mystruct = genstruct("mystruct")
   s = mystruct(a=1, b=2)
   assert.eq(ctor(s), mystruct)
   assert.eq(ctor(1), None)
   assert.eq(ctor(struct(a=1, b=2)), "struct")

Can be used to add some sort of type safety to Starlark libraries. Note that
using type(strct) is insufficient, since it always returns "struct".

R=nodir@chromium.org, iannucci@chromium.org
BUG=833946

Change-Id: I65abc0594441367a7ecd3c0faa05c06499d61c30
Reviewed-on: https://chromium-review.googlesource.com/c/1370693
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/2e46e5dee4bb0e3ce4367435865925348fde82bd/starlark/builtins/struct.go
[add] https://crrev.com/2e46e5dee4bb0e3ce4367435865925348fde82bd/starlark/builtins/struct_test.go

Project Member

Comment 57 by bugdroid1@chromium.org, Dec 12

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/6c166b858d7d555ba7474f1ceac79a8a5a060ce4

commit 6c166b858d7d555ba7474f1ceac79a8a5a060ce4
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Dec 12 02:20:50 2018

[lucicfg] Add @stdlib//internal/luci/lib/acl.star library.

It defines public API for acls:
  * A function that constructs an ACL entry: acl.entry(...).
  * Various initial LUCI roles, e.g acl.BUILDBUCKET_READER.

Also contains internal API used by the rules implementation:
  * aclimpl.validate_acls: checks format of acl.entry(...) definitions.
  * aclimpl.normalize_acls: expands, dedups and sorts ACLs.

More roles will be added when we add support for more services.

R=nodir@chromium.org, iannucci@chromium.org
BUG=833946

Change-Id: I9a41b3ffdc745b66138f73ede984aeff27984b17
Reviewed-on: https://chromium-review.googlesource.com/c/1370611
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/generator.go
[modify] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/starlark/stdlib/builtins.star
[add] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/starlark/stdlib/internal/luci/lib/acl.star
[modify] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[add] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/testdata/acl/entry.star
[add] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/testdata/acl/normalize_acls.star
[add] https://crrev.com/6c166b858d7d555ba7474f1ceac79a8a5a060ce4/lucicfg/testdata/acl/validate_acls.star

Project Member

Comment 58 by bugdroid1@chromium.org, Dec 12

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/71109ce07933a26aa8ec91eda73856d3350c5e46

commit 71109ce07933a26aa8ec91eda73856d3350c5e46
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Dec 12 02:33:00 2018

[lucicfg] Add support for luci-config, logdog and buildbucket ACLs.

ACLs can be defined either on a project or on a bucket levels. ACLs defined on
the project level are inherited by all buckets.

Additionally, some ACLs (such as PROJECT_CONFIGS_READER, LOGDOG_*) can be
defined only on the project level, since LUCI doesn't support defining them on
a bucket level yet or they make no sense when defined on a bucket level.

R=nodir@chromium.org, iannucci@chromium.org
BUG=833946

Change-Id: Ib6175cb2cdc83aae7300c5e2a731882f6ecc4430
Reviewed-on: https://chromium-review.googlesource.com/c/1370612
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/71109ce07933a26aa8ec91eda73856d3350c5e46/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/71109ce07933a26aa8ec91eda73856d3350c5e46/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/71109ce07933a26aa8ec91eda73856d3350c5e46/lucicfg/starlark/stdlib/internal/luci/rules/bucket.star
[modify] https://crrev.com/71109ce07933a26aa8ec91eda73856d3350c5e46/lucicfg/starlark/stdlib/internal/luci/rules/project.star
[modify] https://crrev.com/71109ce07933a26aa8ec91eda73856d3350c5e46/lucicfg/testdata/full_example.star

Project Member

Comment 59 by bugdroid1@chromium.org, Dec 20

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/82607619879e0e403f666fea49c137a2b8611f5c

commit 82607619879e0e403f666fea49c137a2b8611f5c
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Dec 20 01:15:48 2018

[lucicfg] Make graph.key objects introspectable through Starlark.

They now have attributes 'container', 'kind', 'id'. This would become useful
for implementing rules that behave differently depending on a kind of an object
they are related to.

For the key [(k1, i1), (k2, i2), (k3, i3)]:
  'container' is key [(k1, i1), (k2, i2)]
  'kind' is string 'k3'
  'id' is string 'i3'

Term "container" was chosen to avoid confusion with term "parent", already used
in the graph API to denote a different sort of relation.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I16ea3709b6a2adbff9a9d7f810d1080cd5c48f24
Reviewed-on: https://chromium-review.googlesource.com/c/1385666
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/82607619879e0e403f666fea49c137a2b8611f5c/lucicfg/graph/graph.go
[modify] https://crrev.com/82607619879e0e403f666fea49c137a2b8611f5c/lucicfg/graph/key.go
[add] https://crrev.com/82607619879e0e403f666fea49c137a2b8611f5c/lucicfg/graph/keyset.go
[modify] https://crrev.com/82607619879e0e403f666fea49c137a2b8611f5c/lucicfg/graph/node.go
[modify] https://crrev.com/82607619879e0e403f666fea49c137a2b8611f5c/lucicfg/graph/testdata/key.star

Project Member

Comment 60 by bugdroid1@chromium.org, Dec 20

Project Member

Comment 61 by bugdroid1@chromium.org, Dec 20

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/4749df1b10b6d6fb810a1d88021d6c84ff081b2d

commit 4749df1b10b6d6fb810a1d88021d6c84ff081b2d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Dec 20 22:03:09 2018

[lucicfg] Show full key path in node.String().

This will be helpful for printing error messages involving ambiguities when
builder with a same name is defined in multiple buckets.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I91bde848d8a45a4fd745f1a040d7d153d6b69b73
Reviewed-on: https://chromium-review.googlesource.com/c/1387628
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/4749df1b10b6d6fb810a1d88021d6c84ff081b2d/lucicfg/graph/node.go
[modify] https://crrev.com/4749df1b10b6d6fb810a1d88021d6c84ff081b2d/lucicfg/graph/testdata/node.star

FYI, I'm starting a parallel effort for chromium/src side configs in issue 912367. Design doc is in the works, I expect to release it in January 2019.

There is a chance the two efforts will roll into a single larger project, since separating LUCI-only configs and chromium-specific configs still results in unnecessary duplication.
Project Member

Comment 63 by bugdroid1@chromium.org, Dec 20

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/b889c4bdc18d45184d8a88a6eb02fc653f5c68d3

commit b889c4bdc18d45184d8a88a6eb02fc653f5c68d3
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Dec 20 23:19:04 2018

[lucicfg] Allow nodes to be marked as idempotent.

Idempotent nodes can be redeclared, but only if previous declaration was also
marked as idempotent and has same 'props'. This is useful to "auto-declare"
nodes of first usage.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I4d86c0681974d980cab8a39d3f5c8bccf35b9347
Reviewed-on: https://chromium-review.googlesource.com/c/1385673
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/b889c4bdc18d45184d8a88a6eb02fc653f5c68d3/lucicfg/graph/graph.go
[modify] https://crrev.com/b889c4bdc18d45184d8a88a6eb02fc653f5c68d3/lucicfg/graph/node.go
[modify] https://crrev.com/b889c4bdc18d45184d8a88a6eb02fc653f5c68d3/lucicfg/graph/testdata/node.star
[modify] https://crrev.com/b889c4bdc18d45184d8a88a6eb02fc653f5c68d3/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/b889c4bdc18d45184d8a88a6eb02fc653f5c68d3/lucicfg/starlark/stdlib/internal/graph.star

Project Member

Comment 64 by bugdroid1@chromium.org, Dec 20

Project Member

Comment 65 by bugdroid1@chromium.org, Dec 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d4f2cd56eee65b842128834e760da7632cf2ede0

commit d4f2cd56eee65b842128834e760da7632cf2ede0
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Dec 21 02:50:48 2018

[starlark] Teach 'fail' builtin to accept custom stack trace for errors.

This is useful in lucicfg to display stack traces of where nodes are
defined when discovering errors during the generation phase.

Unfortunately Starlark stringifies errors from builtins, so we have to pass
structured information in a roundabout way though thread locals.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I5d6da3b9765679df456fb599a604bc3424879599
Reviewed-on: https://chromium-review.googlesource.com/c/1387635
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/d4f2cd56eee65b842128834e760da7632cf2ede0/starlark/builtins/fail.go
[modify] https://crrev.com/d4f2cd56eee65b842128834e760da7632cf2ede0/starlark/builtins/fail_test.go

Project Member

Comment 66 by bugdroid1@chromium.org, Dec 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/577e508664290fabf83a1a3093d8a4a51258aab7

commit 577e508664290fabf83a1a3093d8a4a51258aab7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Dec 21 03:04:45 2018

[lucicfg] Display traces to badly defined nodes on errors in generators.

Also don't stop on the first error, keep running generator callbacks to collect
all errors. Generators should not depend on each other.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I40d57948465b0a58b9db8630ad3ae1f55b7b4698
Reviewed-on: https://chromium-review.googlesource.com/c/1387636
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/configset.go
[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/errors.go
[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/generator.go
[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/starlark_test.go
[modify] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/testdata/errors/bucket_without_service.star
[add] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/testdata/misc/fail_with_stack.star
[add] https://crrev.com/577e508664290fabf83a1a3093d8a4a51258aab7/lucicfg/testdata/misc/generator_errors.star

Project Member

Comment 67 by bugdroid1@chromium.org, Dec 21

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2c6bc8a7ee925d1b79982a177097237d0e57d6c2

commit 2c6bc8a7ee925d1b79982a177097237d0e57d6c2
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Dec 21 03:12:45 2018

[lucicfg] Deduplicate errors with identical backtraces.

May happen if different generators all discover exact same erroneous node in
the graph.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I6421e721d7b98406c62b1783e226d6315b2f4d84
Reviewed-on: https://chromium-review.googlesource.com/c/1387638
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/2c6bc8a7ee925d1b79982a177097237d0e57d6c2/lucicfg/generator.go
[modify] https://crrev.com/2c6bc8a7ee925d1b79982a177097237d0e57d6c2/lucicfg/state.go
[modify] https://crrev.com/2c6bc8a7ee925d1b79982a177097237d0e57d6c2/lucicfg/testdata/misc/generator_errors.star

Project Member

Comment 71 by bugdroid1@chromium.org, Jan 2

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/0b841cfedf2076c9fb6e20af92562f53fc003ea9

commit 0b841cfedf2076c9fb6e20af92562f53fc003ea9
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 02 21:07:33 2019

[starlarkproto] Add support for map<k,v> fields.

Also improve error messages when assigning unexpected types to a proto-valued
fields.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Ia648549d33dc74842fff2e0be063ee9802bdbd59
Reviewed-on: https://chromium-review.googlesource.com/c/1392589
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/conversions.go
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/default.go
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/doc.go
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/loader.go
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/loader_test.go
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testdata/ints.star
[add] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testdata/map_int.star
[add] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testdata/map_msg.star
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testdata/misc.star
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testdata/msg.star
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testdata/msg_repeated.star
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testprotos/test.pb.go
[modify] https://crrev.com/0b841cfedf2076c9fb6e20af92562f53fc003ea9/starlark/starlarkproto/testprotos/test.proto

Project Member

Comment 72 by bugdroid1@chromium.org, Jan 2

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/aa907e8fbcd3aeb840ba9d5f9a2f810829ad1041

commit aa907e8fbcd3aeb840ba9d5f9a2f810829ad1041
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 02 21:15:33 2019

[lucicfg] Expose well-known protobuf types to Starlark code.

Some configs use them, e.g. Buildbucket uses google.protobuf.UInt32Value.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Ia578386c4d128e35bfa393ba1446a9217e6312e3
Reviewed-on: https://chromium-review.googlesource.com/c/1392588
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/aa907e8fbcd3aeb840ba9d5f9a2f810829ad1041/lucicfg/protos.go

Project Member

Comment 73 by bugdroid1@chromium.org, Jan 3

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8f20b70edb7d57e1fb618b36fc51e5add64cb6fc

commit 8f20b70edb7d57e1fb618b36fc51e5add64cb6fc
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 03 22:35:30 2019

[lucicfg] Add a mode for tests in which error(...) immediately aborts execution.

Note that by default error(...) just collects the error without aborting the
script, all collected errors are reported at the end.

fail_on_errors() mode is useful for writing smallish unit tests that check error
detection logic.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I0ca06978871ec1733ba12d9104fb6675eec99467
Reviewed-on: https://chromium-review.googlesource.com/c/1394190
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/8f20b70edb7d57e1fb618b36fc51e5add64cb6fc/lucicfg/errors.go
[modify] https://crrev.com/8f20b70edb7d57e1fb618b36fc51e5add64cb6fc/lucicfg/state.go
[add] https://crrev.com/8f20b70edb7d57e1fb618b36fc51e5add64cb6fc/lucicfg/testdata/misc/fail_on_errors_mode.star

Project Member

Comment 74 by bugdroid1@chromium.org, Jan 3

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0

commit 0ed1f105d7c6c668d1072cce1e63d6c6db8147a0
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 03 22:47:50 2019

[lucicfg] Add graph.sorted_nodes function, rename 'exec' order to 'def'.

Also clarify that graph.children(..., 'def') and graph.parents(..., 'def')
sort nodes according to the order edges where defined, regardless of the order
of definition of nodes themselves.

On the other hand, sorted_nodes(..., 'def') sorts nodes by the order they were
declared in the graph.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: If3ce2580c72ca3ccf99e0c55130c699a7b03abdc
Reviewed-on: https://chromium-review.googlesource.com/c/1394247
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/graph/graph.go
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/graph/node.go
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/graph/testdata/children.star
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/graph/testdata/descendants.star
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/graph/testdata/parents.star
[add] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/graph/testdata/sorting.star
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/starlark/stdlib/internal/graph.star
[modify] https://crrev.com/0ed1f105d7c6c668d1072cce1e63d6c6db8147a0/lucicfg/testdata/misc/graph.star

Project Member

Comment 75 by bugdroid1@chromium.org, Jan 3

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/dd8514e8c3a0e491ee942b3552b32653bec08b5d

commit dd8514e8c3a0e491ee942b3552b32653bec08b5d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 03 23:44:31 2019

[lucicfg] Initial empty definition of core.builder rule.

There is an aspect that makes the implementation somewhat tricky: we want to
allow to refer to builders using their names alone (without specifying the
bucket name), if this is unambiguous. For cases when there are more than one
builder with a given name, the ambiguity can be resolved by using the full
builder name (e.g. "ci/linux builder").

This is solved by introduction of intermediate 'builder_ref' node:
  * Nodes that refer to builders have builder_refs as children.
  * A builder_ref has a builder as a child.
  * A single builder has two parent builder_ref's: one has bucket-scoped
    name (e.g. "ci/linux builder") and another is global (e.g. "linux builder").
  * Ambiguities manifest as builder_ref nodes with more than one child.

There's an internal API to follow such refs.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I1c126dd3f387a5915f4c49079b05acb1d95b45c8
Reviewed-on: https://chromium-review.googlesource.com/c/1390984
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/starlark/stdlib/builtins.star
[modify] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/starlark/stdlib/internal/luci/common.star
[modify] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/starlark/stdlib/internal/luci/generators.star
[add] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[modify] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/testdata/full_example.star
[add] https://crrev.com/dd8514e8c3a0e491ee942b3552b32653bec08b5d/lucicfg/testdata/misc/builder_refs.star

Project Member

Comment 76 by bugdroid1@chromium.org, Jan 4

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/6a1d1ce6842335fe93e01e332664e9d60130ccb7

commit 6a1d1ce6842335fe93e01e332664e9d60130ccb7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 04 20:54:46 2019

[lucicfg] Module to define durations in a way similar to go's time.Duration.

Will be used to get rid of awkward '_sec' in various '<something>_sec' fields
and also to transform stuff like '1800 # 30 min' into '30 * time.minute'.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I32abe93bbf68bff4d3dc95c31b55d4f19e93b9a9
Reviewed-on: https://chromium-review.googlesource.com/c/1395742
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/6a1d1ce6842335fe93e01e332664e9d60130ccb7/lucicfg/duration.go
[modify] https://crrev.com/6a1d1ce6842335fe93e01e332664e9d60130ccb7/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/6a1d1ce6842335fe93e01e332664e9d60130ccb7/lucicfg/starlark/stdlib/builtins.star
[add] https://crrev.com/6a1d1ce6842335fe93e01e332664e9d60130ccb7/lucicfg/starlark/stdlib/internal/time.star
[add] https://crrev.com/6a1d1ce6842335fe93e01e332664e9d60130ccb7/lucicfg/testdata/misc/time.star

Project Member

Comment 77 by bugdroid1@chromium.org, Jan 4

Project Member

Comment 79 by bugdroid1@chromium.org, Jan 8

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/87686f6c1da41f45a3143646ac0f8f2b9d5f8964

commit 87686f6c1da41f45a3143646ac0f8f2b9d5f8964
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 08 01:46:59 2019

[lucicfg] Add validate.int(), validate.bool() and validate.str_dict().

validate.int will be used to validate e.g. swarming priority
validate.bool will be used to validate Toggle-typed buildbucket fields.
validate.str_dict will be used to validate e.g. dimensions and properties.

Adjust the interface of the validators slightly to allow None to be used as
a value for required=False fields. It is useful to propagate None as "hasn't
been specified" flag (or equivalently to represent Toggle.UNSET), instead of
replacing it with some default value right away.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: Ifbdc9a740972803e7f6d4bb707b07bcd418fb74b
Reviewed-on: https://chromium-review.googlesource.com/c/1399142
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/87686f6c1da41f45a3143646ac0f8f2b9d5f8964/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/87686f6c1da41f45a3143646ac0f8f2b9d5f8964/lucicfg/starlark/stdlib/internal/luci/lib/service.star
[modify] https://crrev.com/87686f6c1da41f45a3143646ac0f8f2b9d5f8964/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[modify] https://crrev.com/87686f6c1da41f45a3143646ac0f8f2b9d5f8964/lucicfg/starlark/stdlib/internal/luci/rules/logdog.star
[modify] https://crrev.com/87686f6c1da41f45a3143646ac0f8f2b9d5f8964/lucicfg/testdata/misc/validators.star

Project Member

Comment 80 by bugdroid1@chromium.org, Jan 8

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/4c84998eba52eb3a7efb27112a07adab7215bda2

commit 4c84998eba52eb3a7efb27112a07adab7215bda2
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 08 02:48:24 2019

[cipd] Add infra/tools/luci/lucicfg/${platform} package.

It will probably become part of depot_tools, so build it for all platforms
supported by depot_tools.

R=nodir@chromium.org
CC=garymm@google.com
BUG=833946

Change-Id: Iaf57032eddb514930799045230582fb53b5f9715
Reviewed-on: https://chromium-review.googlesource.com/c/1399321
Auto-Submit: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19819}
[add] https://crrev.com/4c84998eba52eb3a7efb27112a07adab7215bda2/build/packages/lucicfg.yaml

Project Member

Comment 81 by bugdroid1@chromium.org, Jan 8

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c

commit cb94fd9523b01c1b26c29b6408cb5262da0fbe7c
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 08 19:29:22 2019

[lucicfg] Add empty core.gitiles_poller, define 'triggers/triggered_by' rels.

It's a first instance of polymorphism in lucicfg: 'triggered_by' can refer to
either a builder or a gitiles_poller. As additional complication, we allow to
refer to builders via either their full names ("<bucket>/<name>") or short names
("<name>"), if the latter is non-ambiguous. Same for gitiles_poller names.

This is resolved by introduction of a new internal 'triggerer' node: it is an
entity that can be referred to in 'triggered_by' relation. Both 'builder' and
'gitiles_poller' (each) have two 'triggerer' nodes as children: a bucket-scoped
one and a global one.

This is very similar to 'builder_ref', except the direction of the relation is
reversed: a 'builder_ref' has associated builders as children, but 'triggerer'
has them as parents. This allows to catch cycles in triggering graphs.

For example, a builder that triggers itself produces the following cycle:
  core.builder_ref('b') ->
  core.builder('ci/b') ->
  core.triggerer('ci/b') ->
  core.builder_ref('b')

While such cycle doesn't necessarily mean an invalid configuration, it does
have a potential to introduce a positive feedback loop in runtime that can blow
everything up, so we forbid them.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I25b21a51fc2b355a8688182ccf55820c768ee46b
Reviewed-on: https://chromium-review.googlesource.com/c/1394188
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/starlark/stdlib/builtins.star
[modify] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/starlark/stdlib/internal/luci/common.star
[modify] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[add] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/starlark/stdlib/internal/luci/rules/gitiles_poller.star
[add] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/testdata/errors/ambiguous_references.star
[add] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/testdata/errors/poller_builder_clash.star
[modify] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/testdata/full_example.star
[add] https://crrev.com/cb94fd9523b01c1b26c29b6408cb5262da0fbe7c/lucicfg/testdata/misc/triggerer.star

Project Member

Comment 82 by bugdroid1@chromium.org, Jan 8

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/80712f8b382bbdae855ec0d1d01f6d6170a94981

commit 80712f8b382bbdae855ec0d1d01f6d6170a94981
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 08 20:14:56 2019

[lucicfg] Make validate.duration() return duration instead of number of units.

Turns out it is more useful to keep 'duration' type until the very end.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I815b32813a752db9273091cc98bb4bc5cb7ed91f
Reviewed-on: https://chromium-review.googlesource.com/c/1401327
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/80712f8b382bbdae855ec0d1d01f6d6170a94981/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/80712f8b382bbdae855ec0d1d01f6d6170a94981/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[modify] https://crrev.com/80712f8b382bbdae855ec0d1d01f6d6170a94981/lucicfg/testdata/misc/validators.star

Project Member

Comment 83 by bugdroid1@chromium.org, Jan 8

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/78b43ec434fc1fc1ae67023923cacd7c9884c7d2

commit 78b43ec434fc1fc1ae67023923cacd7c9884c7d2
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 08 20:29:37 2019

[lucicfg] Be careful when mixing assert.fails(...) and fail(...) builtin.

fail(...) builtin uses a thread local state (FailureCollector) to capture
structured information about an error (because Starlark stringifies all errors
returned by a builtin).

assert.fails(...) "catches" starlark errors, but it doesn't clear the captured
error in the thread local state. Fixing this would require forking assert
library, which I'd like to avoid.

So instead just don't use failure collector in tests that don't care about
extra information (like custom stack trace) attached to errors. In lucicfg only
tests that use "Expect errors" blocks care about custom stack traces, and they
generally do not use 'assert' library.

R=tandrii@chromium.org
BUG=833946

Change-Id: I06d9f8fe48c6720070141a4d2e2bcd3d9d88584a
Reviewed-on: https://chromium-review.googlesource.com/c/1397345
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/78b43ec434fc1fc1ae67023923cacd7c9884c7d2/lucicfg/configset.go
[modify] https://crrev.com/78b43ec434fc1fc1ae67023923cacd7c9884c7d2/lucicfg/generator.go
[modify] https://crrev.com/78b43ec434fc1fc1ae67023923cacd7c9884c7d2/lucicfg/starlark_test.go
[modify] https://crrev.com/78b43ec434fc1fc1ae67023923cacd7c9884c7d2/starlark/builtins/fail.go

Project Member

Comment 84 by bugdroid1@chromium.org, Jan 9

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/73abe063c75c7926c68e9520154a169df4c2ad7d

commit 73abe063c75c7926c68e9520154a169df4c2ad7d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 09 01:14:07 2019

[lucicfg] Add swarming helper module.

It defines swarming.cache(...) and swarming.dimension(...) structs that will be
used when defining builders.

  core.builder(
      ...
      caches = [
          swarming.cache('path', name='cache_name'),
          swarming.cache('name_same_as_path'),
          swarming.cache('warmish', wait_for_warm_cache=5*time.minute),
      ],
      ...
      dimensions = {
          'os': 'Linux',
          'expiring': swarming.dimension('value', expiration=5*time.minute),
          'many': ['val1', swarming.dimension('val2')],
      },
      ...
  )

Also defines a bunch of swarmingimpl.validate_X functions that will be used
by the builder rule implementation to validate parameters.

The documentation is mostly copied from Buildbucket's project_config.proto.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Ie55c56af785ed0538fc7674c607bed0f699ec3a6
Reviewed-on: https://chromium-review.googlesource.com/c/1399992
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/73abe063c75c7926c68e9520154a169df4c2ad7d/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/73abe063c75c7926c68e9520154a169df4c2ad7d/lucicfg/starlark/stdlib/builtins.star
[add] https://crrev.com/73abe063c75c7926c68e9520154a169df4c2ad7d/lucicfg/starlark/stdlib/internal/luci/lib/swarming.star
[add] https://crrev.com/73abe063c75c7926c68e9520154a169df4c2ad7d/lucicfg/testdata/swarming/cache.star
[add] https://crrev.com/73abe063c75c7926c68e9520154a169df4c2ad7d/lucicfg/testdata/swarming/dimension.star
[add] https://crrev.com/73abe063c75c7926c68e9520154a169df4c2ad7d/lucicfg/testdata/swarming/tags.star

Project Member

Comment 85 by bugdroid1@chromium.org, Jan 9

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/82c27dc23ccd959a4cdd974e26b0237f63337704

commit 82c27dc23ccd959a4cdd974e26b0237f63337704
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 09 22:54:37 2019

[lucicfg] Implement all fields of core.builder(...) except 'recipe'.

'recipe' field needs a definition of core.recipe(...) rule. It will be done
later.

The following proto fields we explicitly do not support:
  * All mixin related stuff.
  * 'category' (was never used)
  * 'auto_builder_dimension' (shouldn't be necessary with Starlark)
  * recipe.properties (we always use properties_j)

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Iac635f6f28a946e2fb133f42c377f8b901c4a3e5
Reviewed-on: https://chromium-review.googlesource.com/c/1399994
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/82c27dc23ccd959a4cdd974e26b0237f63337704/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/82c27dc23ccd959a4cdd974e26b0237f63337704/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/82c27dc23ccd959a4cdd974e26b0237f63337704/lucicfg/starlark/stdlib/internal/luci/lib/swarming.star
[modify] https://crrev.com/82c27dc23ccd959a4cdd974e26b0237f63337704/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[modify] https://crrev.com/82c27dc23ccd959a4cdd974e26b0237f63337704/lucicfg/testdata/full_example.star

Project Member

Comment 86 by bugdroid1@chromium.org, Jan 9

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b

commit ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 09 23:18:17 2019

[lucicfg] Add basic core.recipe(...) rule.

Make 'recipe=...' be required when defining a builder.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Iff91bd0d8a134036f9722b376d72b3749357b1a9
Reviewed-on: https://chromium-review.googlesource.com/c/1401474
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/starlark/stdlib/builtins.star
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/starlark/stdlib/internal/luci/common.star
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[add] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/starlark/stdlib/internal/luci/rules/recipe.star
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/testdata/errors/ambiguous_references.star
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/testdata/errors/poller_builder_clash.star
[modify] https://crrev.com/ef45b74c00894b9c47e5b1df40b5cfc0f8b58f4b/lucicfg/testdata/full_example.star

Project Member

Comment 87 by bugdroid1@chromium.org, Jan 10

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/51509a72b83b5c5167b213a69dda6f30701b03d9

commit 51509a72b83b5c5167b213a69dda6f30701b03d9
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 10 01:25:55 2019

[lucicfg] Add acl.SCHEDULER_* roles, rename Buildbucket roles for consistency.

Scheduler and Buildbucket use different names for the same kind of roles:
  Buildbucket's SCHEDULER ~= Scheduler's TRIGGERER
  Buildbucket's WRITER ~= Scheduler's OWNER

This is confusing. So we'll need to make a compromise somewhere: either use
Buildbucket-inspired roles or Scheduler-inspired ones (or pick something
completely unrelated).

Arguably, scheduler's ones (TRIGGERER, OWNER) are clearer than buildbucket's
ones (SCHEDULER, WRITER). For one, 'schedule' is a pretty overloaded term, and
SCHEDULER_SCHEDULER will look odd. Also Buildbucket's WRITER role doesn't convey
the powers it grants (e.g. ability to cancel all builds in the bucket).

So we now have:
  BUILDBUCKET_READER - can list buckets, read build properties
  BUILDBUCKET_TRIGGERER - can launch buildbucket tasks
  BUILDBUCKET_OWNER - full access to the bucket (e.g. clearing it)
  SCHEDULER_READER - can list scheduler job invocations
  SCHEDULER_TRIGGERER - can launch scheduler jobs by submitting triggers
  SCHEDULER_OWNER - can abort jobs

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: I494a9af90714bf1739c91fdb6fa25167681be1e6
Reviewed-on: https://chromium-review.googlesource.com/c/1400850
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/starlark/stdlib/internal/luci/lib/acl.star
[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/testdata/acl/entry.star
[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/testdata/acl/normalize_acls.star
[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/testdata/acl/validate_acls.star
[modify] https://crrev.com/51509a72b83b5c5167b213a69dda6f30701b03d9/lucicfg/testdata/full_example.star

Project Member

Comment 89 by bugdroid1@chromium.org, Jan 11

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/aa7c87510bda051d0f4b8d52d1ad08308ae7105e

commit aa7c87510bda051d0f4b8d52d1ad08308ae7105e
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 11 05:22:06 2019

[lucicfg] Implement (most of) scheduler.cfg generation.

Also slightly adjust codestyle in the buildbucket generator for consistency.

R=nodir@chromium.org, tandrii@chromium.org
BUG=833946

Change-Id: Ie2db248c91421c85631527f63395723d43e395e2
Reviewed-on: https://chromium-review.googlesource.com/c/1404297
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/starlark/stdlib/internal/luci/rules/project.star
[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/errors/ambiguous_references.star
[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/errors/bucket_without_service.star
[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/errors/poller_builder_clash.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/errors/scheduler_disambiguation_fail.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/errors/triggering_builder_no_account.star
[modify] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/full_example.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/scheduler/ambiguity_resolution.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/scheduler/builder_builder.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/scheduler/dedups_triggerer_acl.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/scheduler/many_to_many.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/scheduler/poller_builder.star
[add] https://crrev.com/aa7c87510bda051d0f4b8d52d1ad08308ae7105e/lucicfg/testdata/scheduler/single_poller.star

Project Member

Comment 90 by bugdroid1@chromium.org, Jan 11

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/4f9ab59fed2146929cd085c7946651bd2126e709

commit 4f9ab59fed2146929cd085c7946651bd2126e709
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 11 19:34:51 2019

[lucicfg] Make rule's arguments kwargs only and None by default.

Using positional args leads to some very confusing error messages like:

    core.builder(
        name = 'b',
        bucket = 'ci',
        service_account = 'noop@example.com',
    )

    Traceback (most recent call last):
      ...
    Error: function builder takes at least 3 positional arguments (3 given)

Instead we now get:

    Traceback (most recent call last):
      ...
    Error: missing required field "recipe"

Hopefully https://github.com/google/starlark-go/issues/61 will land someday
and we'll be able to say def f(*, a=None, b=None) to force all arguments to
be passed via keywords.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I96336bd6b849e55db0b0a2fa508786212c5652d6
Reviewed-on: https://chromium-review.googlesource.com/c/1406418
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/stdlib/internal/luci/rules/bucket.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/stdlib/internal/luci/rules/gitiles_poller.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/stdlib/internal/luci/rules/project.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/starlark/stdlib/internal/luci/rules/recipe.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/testdata/acl/entry.star
[add] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/testdata/errors/builder_without_recipe.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/testdata/errors/project_without_name.star
[modify] https://crrev.com/4f9ab59fed2146929cd085c7946651bd2126e709/lucicfg/testdata/misc/validators.star

Project Member

Comment 91 by bugdroid1@chromium.org, Jan 11

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/3000bf43edfc8b220ba822311d24cbdfe46a0d72

commit 3000bf43edfc8b220ba822311d24cbdfe46a0d72
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 11 19:59:15 2019

[lucicfg] Finish gitiles_poller(...) definition.

I decided to omit 'triggering_policy' for it, since I've never seen custom
polices used with pollers. Can be added later if needed.

Also the validation rules are quite involved, so let the server do the thorough
validation and only validate types in lucicfg.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: Icc56c5a3f2073dce51b3ad859e454ea36bef5382
Reviewed-on: https://chromium-review.googlesource.com/c/1406423
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/starlark/stdlib/internal/luci/generators.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/starlark/stdlib/internal/luci/rules/gitiles_poller.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/errors/ambiguous_references.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/errors/poller_builder_clash.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/errors/scheduler_disambiguation_fail.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/full_example.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/scheduler/ambiguity_resolution.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/scheduler/dedups_triggerer_acl.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/scheduler/many_to_many.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/scheduler/poller_builder.star
[modify] https://crrev.com/3000bf43edfc8b220ba822311d24cbdfe46a0d72/lucicfg/testdata/scheduler/single_poller.star

Project Member

Comment 92 by bugdroid1@chromium.org, Jan 15

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e22dfba782fbedc56bca4ed0818f2113cc3f914f

commit e22dfba782fbedc56bca4ed0818f2113cc3f914f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 15 04:45:48 2019

[lucicfg] docgen: Add AST parser for the documentation generator.

It takes a Starlark's AST and reduces it only to structures important for the
documentation generation purposes, preserving their docstring and comments.

It works on a single file only. It will be used by a higher level parser that
understands relations between modules in a package, e.g. if a symbol is defined
in an internal module, but then publicly exposed through an alias in another
module, the documentation generator will be able to trace the references back
to the original definition, to grab a documentation from it.

R=tandrii@chromium.org
BUG=833946

Change-Id: Ic5ae1ac25978cb3964f20d3ae43bc3265fc7078c
Reviewed-on: https://chromium-review.googlesource.com/c/1407728
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/e22dfba782fbedc56bca4ed0818f2113cc3f914f/lucicfg/docgen/ast/parser.go
[add] https://crrev.com/e22dfba782fbedc56bca4ed0818f2113cc3f914f/lucicfg/docgen/ast/parser_test.go

Project Member

Comment 93 by bugdroid1@chromium.org, Jan 15

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/7f11c87315815a78eb3554e4d5c76e059c350f60

commit 7f11c87315815a78eb3554e4d5c76e059c350f60
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 15 05:10:42 2019

[lucicfg] docgen: Add docstring parser.

It parses a docstring into a main body, a list of arguments and a list of
additional sections, like "Returns". All extracted strings are properly
deindented and cleaned up, ready to be inserted into e.g. markdown.

This is surprisingly non-trivial, complicated by careful handling of
indentation, since it is syntactically significant in this case.

R=tandrii@chromium.org
BUG=833946

Change-Id: I38c1f99cb71331e2f6617196415b4e914e15614e
Reviewed-on: https://chromium-review.googlesource.com/c/1408602
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/7f11c87315815a78eb3554e4d5c76e059c350f60/lucicfg/docgen/docstring/docstring.go
[add] https://crrev.com/7f11c87315815a78eb3554e4d5c76e059c350f60/lucicfg/docgen/docstring/docstring_test.go

Project Member

Comment 94 by bugdroid1@chromium.org, Jan 15

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/44f46fb06a244d4a352071774427cbb03214216e

commit 44f46fb06a244d4a352071774427cbb03214216e
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 15 23:50:02 2019

[lucicfg] docgen: Understand 'load(...)' statements and aliasing.

This allows to discover AST nodes with definitions (and documentation!) of all
symbols exported by a module by following the chain of assignments, load(...)'s
and struct lookups.

R=tandrii@chromium.org
BUG=833946

Change-Id: Icea3dc4cffb8cc1cdcdfa13fe416d175d1d90849
Reviewed-on: https://chromium-review.googlesource.com/c/1407993
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/44f46fb06a244d4a352071774427cbb03214216e/lucicfg/docgen/symbols/loader.go
[add] https://crrev.com/44f46fb06a244d4a352071774427cbb03214216e/lucicfg/docgen/symbols/loader_test.go
[add] https://crrev.com/44f46fb06a244d4a352071774427cbb03214216e/lucicfg/docgen/symbols/symbols.go

Project Member

Comment 95 by bugdroid1@chromium.org, Jan 17 (5 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/786e5243fb201d489d44b0750adb1fac3975726e

commit 786e5243fb201d489d44b0750adb1fac3975726e
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 17 19:28:14 2019

[lucicfg] Add validate.float and validate.type.

Will be used to validate scheduler's TriggeringPolicy definitions.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I252daebcd5d4a624474601f2a72aae1520e6f589
Reviewed-on: https://chromium-review.googlesource.com/c/1417292
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/786e5243fb201d489d44b0750adb1fac3975726e/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/786e5243fb201d489d44b0750adb1fac3975726e/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[modify] https://crrev.com/786e5243fb201d489d44b0750adb1fac3975726e/lucicfg/testdata/misc/validators.star

Project Member

Comment 96 by bugdroid1@chromium.org, Jan 17 (5 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/1a0d955d7361e3c127006559f935fed350f3d6c1

commit 1a0d955d7361e3c127006559f935fed350f3d6c1
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 17 20:04:58 2019

[lucicfg] docgen: Add the actual generator.

It is a text generator based on text/template that has access to functions that
return structured information about symbols defined in the starlark code. This
allows templates to insert docs extracted from the source code at appropriate
places.

This CL has only very basic implementation, as a proof of concept. It needs
a lot more polish before it generates acceptably looking and useful docs.

R=tandrii@chromium.org
BUG=833946

Change-Id: I849ace37b62782f25ac2603d968016a709728f5a
Reviewed-on: https://chromium-review.googlesource.com/c/1408603
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/1a0d955d7361e3c127006559f935fed350f3d6c1/lucicfg/cmd/docgen/main.go
[add] https://crrev.com/1a0d955d7361e3c127006559f935fed350f3d6c1/lucicfg/doc/README.md
[add] https://crrev.com/1a0d955d7361e3c127006559f935fed350f3d6c1/lucicfg/doc/generate.go
[add] https://crrev.com/1a0d955d7361e3c127006559f935fed350f3d6c1/lucicfg/doc/templates/README.mdt
[modify] https://crrev.com/1a0d955d7361e3c127006559f935fed350f3d6c1/lucicfg/docgen/docstring/docstring.go
[add] https://crrev.com/1a0d955d7361e3c127006559f935fed350f3d6c1/lucicfg/docgen/generator.go

Project Member

Comment 98 by bugdroid1@chromium.org, Jan 18 (5 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/fc59bd114abd60c52ccad825779d9e1f2406fd5d

commit fc59bd114abd60c52ccad825779d9e1f2406fd5d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 18 05:25:24 2019

[lucicfg] Cleanup docs a bit.

This also moves 'ctor' and 'genstruct' from the global namespace to __native__,
since they are a bit weird and I don't want to advertise them as public API.

R=tandrii@chromium.org
BUG=833946

Change-Id: If856001d4856f41096a1c04d3b89ceb7f904fa1d
Reviewed-on: https://chromium-review.googlesource.com/c/1413997
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/doc/README.md
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/doc/templates/README.mdt
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/generator.go
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/builtins.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/generator.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/lib/acl.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/lib/swarming.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/lib/validate.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/rules/bucket.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/rules/gitiles_poller.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/rules/logdog.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/rules/project.star
[modify] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/internal/luci/rules/recipe.star
[add] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/native_doc.star
[add] https://crrev.com/fc59bd114abd60c52ccad825779d9e1f2406fd5d/lucicfg/starlark/stdlib/proto_doc.star

Project Member

Comment 99 by bugdroid1@chromium.org, Jan 18 (4 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/37b4f2fcc80cf8075860412039614792ba2e0d93

commit 37b4f2fcc80cf8075860412039614792ba2e0d93
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 18 21:31:58 2019

[lucicfg] Add 'schedule' and 'triggering_policy' to core.builder(...).

Also improve the documentation regarding how batching happens, i.e. how git
commits transform into builds.

This concludes cr-buildbucket.cfg and luci-scheduler.cfg portion of lucicfg.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I4d58104565ca04ad2e9aeb779d269f81bdf37a36
Reviewed-on: https://chromium-review.googlesource.com/c/1419286
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/doc/README.md
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/doc/templates/README.mdt
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/starlark/stdlib/builtins.star
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/starlark/stdlib/internal/luci/generators.star
[add] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/starlark/stdlib/internal/luci/lib/scheduler.star
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/starlark/stdlib/internal/luci/rules/builder.star
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/starlark/stdlib/internal/luci/rules/gitiles_poller.star
[modify] https://crrev.com/37b4f2fcc80cf8075860412039614792ba2e0d93/lucicfg/testdata/full_example.star

Project Member

Comment 100 by bugdroid1@chromium.org, Jan 19 (4 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f

commit 40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Jan 19 03:13:53 2019

[lucicfg] Add a way to provide "meta" configuration.

This is configuration for the config generator itself. Stuff like where to
place files and how to call LUCI Config.

It can be set via core.meta(...) in *.star, or overriden through command line
flags.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I109d5be20b73b8e1f72fe7293fba94e9b7b5d2e9
Reviewed-on: https://chromium-review.googlesource.com/c/1420019
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/doc/README.md
[modify] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/doc/templates/README.mdt
[add] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/meta.go
[add] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/meta_test.go
[modify] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/starlark/assets.gen.go
[modify] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/starlark/stdlib/builtins.star
[add] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/starlark/stdlib/internal/meta.star
[modify] https://crrev.com/40266bdca3e0c5da0bfb4b4ffe2d6649c2c1f85f/lucicfg/state.go

Project Member

Comment 101 by bugdroid1@chromium.org, Today (17 hours ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/42221c6b4351b26da2f4ad6910335d8391642192

commit 42221c6b4351b26da2f4ad6910335d8391642192
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 22 20:08:21 2019

[lucicfg] Add ConfigSet Go type, extract validation logic into it.

This code will be shared by multiple lucicfg subcommands.

Also fix printing of errors. Apparently errors.WalkLeaves completely skips
errors produced with errors.Reason(...).Err().

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: Ie196e92af88f7f2d16680f23d9041f1772c54da8
Reviewed-on: https://chromium-review.googlesource.com/c/1423325
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/cli/base/base.go
[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/cli/cmds/validate/validate.go
[delete] https://crrev.com/73453766172cf2ba3a100ba6e54e1b735be70109/lucicfg/cli/cmds/validate/validate_test.go
[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/configset.go
[add] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/configset_test.go
[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/genctx.go
[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/generator.go
[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/starlark_test.go
[modify] https://crrev.com/42221c6b4351b26da2f4ad6910335d8391642192/lucicfg/state.go

Project Member

Comment 102 by bugdroid, Today (7 hours ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5256b7785e98410664e2696df9d4f5b52ae4dfb7

commit 5256b7785e98410664e2696df9d4f5b52ae4dfb7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 23 05:21:15 2019

[lucicfg] Implement FindTrackedFiles function.

It will be used to figure out what existing files on disk are supposed to be
generated from *.star. See 'tracked_files' in meta.config(...) doc for more
information.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I74930c7ce871993b682a9dd4d0a4822a324f99fd
Reviewed-on: https://chromium-review.googlesource.com/c/1429189
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/5256b7785e98410664e2696df9d4f5b52ae4dfb7/lucicfg/tracked.go
[add] https://crrev.com/5256b7785e98410664e2696df9d4f5b52ae4dfb7/lucicfg/tracked_test.go

Project Member

Comment 103 by bugdroid, Today (7 hours ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5256b7785e98410664e2696df9d4f5b52ae4dfb7

commit 5256b7785e98410664e2696df9d4f5b52ae4dfb7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 23 05:21:15 2019

[lucicfg] Implement FindTrackedFiles function.

It will be used to figure out what existing files on disk are supposed to be
generated from *.star. See 'tracked_files' in meta.config(...) doc for more
information.

R=tandrii@chromium.org, nodir@chromium.org
BUG=833946

Change-Id: I74930c7ce871993b682a9dd4d0a4822a324f99fd
Reviewed-on: https://chromium-review.googlesource.com/c/1429189
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/5256b7785e98410664e2696df9d4f5b52ae4dfb7/lucicfg/tracked.go
[add] https://crrev.com/5256b7785e98410664e2696df9d4f5b52ae4dfb7/lucicfg/tracked_test.go

Showing comments 4 - 103 of 103 Older

Sign in to add a comment