New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 742478 link

Starred by 5 users
Status: Fixed
Owner:
no...@chromium.org
Closed: Mar 2018
Cc:
█ phajdan.jr@chromium.org
d...@chromium.org
█ katthomas@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 757060


Sign in to add a comment

build events in LUCI

Project Member Reported by no...@chromium.org, Jul 13 2017 
Events of buildbot build and step completion is available in BigQuery as chrome_infra.completed_builds from chrome_infra.completed_steps:

SELECT * from chrome_infra.completed_builds LIMIT 100;
SELECT * from chrome_infra.completed_steps LIMIT 100;

We need similar functionality for LUCI so that users can do offline analysis of builds/steps, e.g. to reason about step timing. It will also be used to determine number of consecutive infra failure on a builder or bot.

Comment 1 by no...@chromium.org, Jul 13 2017 

Proposal

Given katthomas@'s effort to convert the eventlog pipeline to BigQuery and the fact that these two tables do not need any aggregation, kitchen can stream build/step completion events to BigQuery tables directly, using native BigQuery lib in Go.

The tables chrome_infra.completed_builds and chrome_infra.completed_steps are not pure BigQuery tables, so they are left intact for this bug.

Fixing this bug includes:

1. creation similar pure-BigQuery tables with identical schemas and names, using https://chromium.git.corp.google.com/infra/infra/+/HEAD/go/src/infra/tools/bqschemaupdater# tool. Katie, in which cloud project should we create the tables. 

2. modify kitchen to start streaming events to BigQuery using native BigQuery lib

At this time, recipe engine does not stream notification of step completion back to the caller, ATM so we cannot stream step completion events at the time of step completion. This is not a concern because it is not worse than Buildbot. Buildbot master uses eventlog which has 20min delay.

After recipe engine exits, Kitchen can use the final annotation proto to create events for step completion.

3. a-nice-to-have: update https://cs.chromium.org/chromium/build/scripts/master/status_logger.py to stream events to the new BigQuery tables in addition to eventlog. Later we will turn down eventlog seamlessly.

--------

Event without (3), (1) and (2) will allow us to compare steps/builds from Buildbot and LUCI in the BigQuery interface because events from both sources are available there. With (3), however, they will be in one table instead of two.

Comment 2 by no...@chromium.org, Jul 13 2017 

These new BigQuery tables must be internal. When we are doing with items 1-3, we can define a new public table and write a dataflow pipeline to filter+anonymize+copy data from the internal table to the public one.

Comment 3 by katthomas@google.com, Jul 13 2017 

bqschemaupdater will automatically create the table in the "chrome-infra-events" project. I'll make that explicit in the README.

Comment 4 by tandrii@chromium.org, Jul 17 2017

Status: Available (was: Untriaged)

Comment 5 by efoo@chromium.org, Jul 18 2017

Labels: LUCI-M2-Dev2 LUCI-M2-S6 REQBY-LUCI-M3-ClosedBeta
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 15 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6253a12bdcf055618c699bbd136e00d29bbeadab

commit 6253a12bdcf055618c699bbd136e00d29bbeadab
Author: Dan Jacques <dnj@google.com>
Date: Tue Aug 15 23:12:09 2017

[bqexport] Add bqexport tool.

Add the "bqexport" tool, a Go generator which extracts a BigQuery
struct from Go source and exports it as a table definition schema that
"infra/tools/bqschemaupdater" can use.

BUG= chromium:742478 
TEST=local

Change-Id: Ib808b9ff0e4c227b0cdebe02c8847d51686a6ea6
Reviewed-on: https://chromium-review.googlesource.com/614563
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/testing/golden.json
[modify] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/tools/bqschemaupdater/README.md
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/main.go
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/bqexport.infra_testing
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/util/util.infra_testing
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/testing/schema.go
[modify] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/bootstrap.py
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/README.md
[modify] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/doc/users/event_pipeline.md
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/util/util.go
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/testing/testing.infra_testing
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/export.go
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/util/proto.go
[add] https://crrev.com/6253a12bdcf055618c699bbd136e00d29bbeadab/go/src/infra/cmd/bqexport/export_test.go

Comment 7 by no...@chromium.org, Aug 16 2017 

I was using completed_steps table right now and needed to filter by whether a step was infra or not. Dan, when you define a bq table for completed steps, please add a bool field for that. Thanks

Comment 8 by d...@chromium.org, Aug 16 2017 

Sounds like non-legacy schema?
Project Member

Comment 9 by bugdroid1@chromium.org, Aug 16 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e5b5b48eab11c02cd21e4cc301766aa6527db275

commit e5b5b48eab11c02cd21e4cc301766aa6527db275
Author: Dan Jacques <dnj@google.com>
Date: Wed Aug 16 03:57:25 2017

[bqexport] Fix flag parsing.

Fix a bug in flag parsing that was introduced when the package was
refactored for testing.

TBR=nodir@chromium.org
BUG= chromium:742478 
TEST=None

Change-Id: Icb752087b7928006c536916f96e341380a230c16
Reviewed-on: https://chromium-review.googlesource.com/616185
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/e5b5b48eab11c02cd21e4cc301766aa6527db275/go/src/infra/cmd/bqexport/export.go
[modify] https://crrev.com/e5b5b48eab11c02cd21e4cc301766aa6527db275/go/src/infra/cmd/bqexport/main.go

Comment 10 by no...@chromium.org, Aug 16 2017 

I think an extra field won't hurt the legacy schema.

However, I spoke to estaab@ offline yesterday and now I tend to think that we shouldn't be marking steps as infra, in general. I can provide the rationale, not this is outside of the scope of this bug.
Project Member

Comment 11 by bugdroid1@chromium.org, Aug 16 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1a2e174f4b93b52addf32438b50bae7f811e01bd

commit 1a2e174f4b93b52addf32438b50bae7f811e01bd
Author: Dan Jacques <dnj@google.com>
Date: Wed Aug 16 19:18:03 2017

[bqexport] Apply optional default to records.

Currently, "bqexport" doesn't apply the optional-by-default field state
to record fields.

BUG= chromium:742478 
TEST=unit
R=katthomas@chromium.org, nodir@chromium.org

Change-Id: I57b6552be16efb05804fe2b347ba00acd3d6b4f5
Reviewed-on: https://chromium-review.googlesource.com/617643
Reviewed-by: Katie Thomas <katthomas@google.com>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/1a2e174f4b93b52addf32438b50bae7f811e01bd/go/src/infra/cmd/bqexport/testing/golden.json
[modify] https://crrev.com/1a2e174f4b93b52addf32438b50bae7f811e01bd/go/src/infra/cmd/bqexport/util/util.go
[modify] https://crrev.com/1a2e174f4b93b52addf32438b50bae7f811e01bd/go/src/infra/cmd/bqexport/testing/schema.go
Project Member

Comment 12 by bugdroid1@chromium.org, Aug 18 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/377b01a84d075d6250fd2e26c1a70222063e673f

commit 377b01a84d075d6250fd2e26c1a70222063e673f
Author: Dan Jacques <dnj@google.com>
Date: Fri Aug 18 22:22:41 2017

[bq] BigQuery TableDef proto uses dataset enum.

Currently, the "DatasetId" proto field is implicitly constrainted to a
handful of supported dataset values. Formally encode this as an
enumeration in the protobuf, and update associated tooling and libraries
to support this.

Update "bqexport" to output to a base directory rather than a full
output path. It will then use the dataset ID as a subdirectory so that
it maintains the layout currently in place by "bqschemaupdater".

BUG= chromium:742478 
TEST=None

Change-Id: I5f64b55eaf94beffe3d7bdb64e84390d052f4843
Reviewed-on: https://chromium-review.googlesource.com/619281
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>

[add] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/libs/bqschema/tabledef/generate.go
[rename] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/tools/bqschemaupdater/raw_events/cq.json
[rename] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/cmd/bqexport/test_data/raw_events/golden.json
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/libs/bqschema/tabledef/table_def.proto
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/cmd/bqexport/main.go
[add] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/libs/bqschema/tabledef/util.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/tools/bqschemaupdater/bqschemaupdater.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/tools/bqschemaupdater/bqschemaupdater_test.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/cmd/bqexport/testing/schema.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/libs/bqschema/tabledef/table_def.pb.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/cmd/bqexport/util/util.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/tools/bqschemaupdater/aggregated/cq_attempts.json
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/cmd/bqexport/export.go
[modify] https://crrev.com/377b01a84d075d6250fd2e26c1a70222063e673f/go/src/infra/cmd/bqexport/export_test.go

Comment 13 by d...@chromium.org, Aug 18 2017

Blockedon: 757060
Project Member

Comment 14 by bugdroid1@chromium.org, Aug 19 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f55e98920c87cf328f077c56a3200ecef76d8c24

commit f55e98920c87cf328f077c56a3200ecef76d8c24
Author: Dan Jacques <dnj@google.com>
Date: Sat Aug 19 00:55:31 2017

Add LegacyCompletedBuild/Steps schema.

This is a modified version of the schemas used by BuildBot builds.
Those schemas are full of unused or useless fields. This schema trims
those fields and adds some additional fields.

The schemas used here are deemed "legacy". They are propagated
into BigQuery event monitoring because we want to have general
compatibility with the current BuildBot build events, but the intent
is to revise them significantly and export a new table that takes better
advantage of BigQuery's schema layout and is more suited to our expanding
infrastructure, notably LUCI.

BUG= chromium:742478 
TEST=None

Change-Id: I779da54f9aa59ab1954e2866906c48d0b3d837a9
Reviewed-on: https://chromium-review.googlesource.com/614565
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/libs/bqschema/buildevent/completedstep.go
[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/libs/bqschema/buildevent/completedbuild.go
[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/libs/bqschema/buildevent/generate.go
[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/libs/bqschema/buildevent/doc.go
[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_step_legacy_table.json
[modify] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/cmd/bqexport/util/util.go
[modify] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/cmd/bqexport/test_data/raw_events/golden.json
[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_build_legacy_table.json
[add] https://crrev.com/f55e98920c87cf328f077c56a3200ecef76d8c24/go/src/infra/libs/bqschema/buildevent/buildevent.infra_testing
Project Member

Comment 15 by bugdroid1@chromium.org, Aug 22 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5d0125abe5737fcc8b4a0563d8d7ee847cf3319a

commit 5d0125abe5737fcc8b4a0563d8d7ee847cf3319a
Author: Dan Jacques <dnj@google.com>
Date: Tue Aug 22 20:36:25 2017

[kitchen] Use LUCI context on Swarming.

Update Kitchen to use a default authenticator for LogDog on Swarming.

Provide a flag, "-logdog-service-account", so BuildBot invocations of
Kitchen can still specify the local service account.

BUG= chromium:742478 
TEST=None
R=nodir@chromium.org, vadimsh@chromium.org

Change-Id: Idf3aeead2d091c6a68ce1c8ba4f96a25847bdb53
Reviewed-on: https://chromium-review.googlesource.com/622079
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/5d0125abe5737fcc8b4a0563d8d7ee847cf3319a/go/src/infra/tools/kitchen/cook.go
[modify] https://crrev.com/5d0125abe5737fcc8b4a0563d8d7ee847cf3319a/go/src/infra/tools/kitchen/cookflags/flags.go
[modify] https://crrev.com/5d0125abe5737fcc8b4a0563d8d7ee847cf3319a/go/src/infra/tools/kitchen/cookflags/logdog_flags.go
[modify] https://crrev.com/5d0125abe5737fcc8b4a0563d8d7ee847cf3319a/go/src/infra/tools/kitchen/cook_logdog.go

Comment 16 by efoo@chromium.org, Aug 22 2017

Cc: no...@chromium.org
Owner: d...@chromium.org
Assigning to dnj for tracking for BigQuery work.
Project Member

Comment 17 by bugdroid1@chromium.org, Aug 24 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7afb4ae133678e2d02c5d20cd61762ce9649365a

commit 7afb4ae133678e2d02c5d20cd61762ce9649365a
Author: Dan Jacques <dnj@google.com>
Date: Thu Aug 24 00:56:29 2017

[bqexport] Invert, generate Go from schema.

Invert "bqexport"; now it generates Go structs from an input schema
protobuf. This is a more natural translation and, notably, enables
descriptions in the BigQuery schema.

Add comments to build events tables.

TBR=nodir@chromium.org
BUG= chromium:742478 
TEST=local

Change-Id: Id12fefb55316630ef0a5e99011940d81e6eef75f
Reviewed-on: https://chromium-review.googlesource.com/630499
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>

[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/libs/bqschema/buildevent/completed_builds_legacy.gen.go
[delete] https://crrev.com/7397dbbc200f1fcc44f7ffc1adc18a98437e3a09/go/src/infra/cmd/bqexport/util/util.go
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_build_legacy.json
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/PRESUBMIT.py
[delete] https://crrev.com/7397dbbc200f1fcc44f7ffc1adc18a98437e3a09/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_build_legacy_table.json
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/export.go
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/test_data/main.go
[delete] https://crrev.com/7397dbbc200f1fcc44f7ffc1adc18a98437e3a09/go/src/infra/cmd/bqexport/test_data/raw_events/golden.json
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/libs/bqschema/buildevent/generate.go
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_step_legacy.json
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/generate.go
[delete] https://crrev.com/7397dbbc200f1fcc44f7ffc1adc18a98437e3a09/go/src/infra/cmd/bqexport/util/proto.go
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/export_test.go
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/libs/bqschema/buildevent/completedbuild.go
[rename] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/test_data/test_data.infra_testing
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/libs/bqschema/buildevent/completed_step_legacy.gen.go
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/testing/schema.go
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/test_data/golden.gen.go
[delete] https://crrev.com/7397dbbc200f1fcc44f7ffc1adc18a98437e3a09/go/src/infra/libs/bqschema/buildevent/completedstep.go
[modify] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/cmd/bqexport/main.go
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/libs/bqschema/buildevent/completed_builds_legacy_legacy_completed_builds.gen.go
[add] https://crrev.com/7afb4ae133678e2d02c5d20cd61762ce9649365a/go/src/infra/libs/bqschema/buildevent/completed_step_legacy_legacy_build_events.gen.go
[delete] https://crrev.com/7397dbbc200f1fcc44f7ffc1adc18a98437e3a09/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_step_legacy_table.json
Project Member

Comment 18 by bugdroid1@chromium.org, Aug 24 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b9d7c9aa8559a47392aa5734388efd05f8fd3858

commit b9d7c9aa8559a47392aa5734388efd05f8fd3858
Author: Dan Jacques <dnj@google.com>
Date: Thu Aug 24 04:18:52 2017

[bqschemaupdater] Auth and better logging.

Add user authentication to the BigQuery schema updater. This will
perform an OAuth2 handshake for a user prior to updating and retain
their login token for future uses.

Add some better logging and minor bigfixes.

BUG= chromium:742478 
TEST=None
R=katthomas@chromium.org, nodir@chromium.org

Change-Id: I0413cb311f77f5a26a42712010d894fb95bb2e97
Reviewed-on: https://chromium-review.googlesource.com/628959
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[add] https://crrev.com/b9d7c9aa8559a47392aa5734388efd05f8fd3858/go/src/infra/libs/infraenv/monitoring.go
[modify] https://crrev.com/b9d7c9aa8559a47392aa5734388efd05f8fd3858/go/src/infra/tools/bqschemaupdater/bqschemaupdater.go
[modify] https://crrev.com/b9d7c9aa8559a47392aa5734388efd05f8fd3858/go/src/infra/tools/bqschemaupdater/bqschemaupdater_test.go
Project Member

Comment 19 by bugdroid1@chromium.org, Aug 24 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5bf26eaa091136370bb51a1c38f35da135f24da7

commit 5bf26eaa091136370bb51a1c38f35da135f24da7
Author: Dan Jacques <dnj@google.com>
Date: Thu Aug 24 18:46:51 2017

[bqschemaupdater] Text protobuf instead of JSON.

BUG= chromium:742478 
TEST=None

Change-Id: Ib1aeea4684d8470bc0fc57091d86139ef064586e
Reviewed-on: https://chromium-review.googlesource.com/630559
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Katie Thomas <katthomas@google.com>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[delete] https://crrev.com/3f5b554f496e027ddc9b67c9f3a64424032a6558/go/src/infra/tools/bqschemaupdater/raw_events/cq.json
[delete] https://crrev.com/3f5b554f496e027ddc9b67c9f3a64424032a6558/go/src/infra/tools/bqschemaupdater/aggregated/cq_attempts.json
[add] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/tools/bqschemaupdater/raw_events/cq.pb.txt
[delete] https://crrev.com/3f5b554f496e027ddc9b67c9f3a64424032a6558/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_step_legacy.json
[modify] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/libs/bqschema/buildevent/generate.go
[modify] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/tools/bqschemaupdater/bqschemaupdater.go
[modify] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/libs/bqschema/buildevent/completed_step_legacy.gen.go
[add] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_build_legacy.pb.txt
[delete] https://crrev.com/3f5b554f496e027ddc9b67c9f3a64424032a6558/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_build_legacy.json
[add] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/tools/bqschemaupdater/raw_events/buildevent_completed_step_legacy.pb.txt
[modify] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/cmd/bqexport/export.go
[add] https://crrev.com/5bf26eaa091136370bb51a1c38f35da135f24da7/go/src/infra/tools/bqschemaupdater/aggregated/cq_attempts.pb.txt
Project Member

Comment 20 by bugdroid1@chromium.org, Aug 25 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fa64a533e5b32a2f7aa751256f26c0779d6e66f6

commit fa64a533e5b32a2f7aa751256f26c0779d6e66f6
Author: Dan Jacques <dnj@google.com>
Date: Fri Aug 25 02:37:34 2017

[kitchen] Add event monitoring support.

Add event monitoring support to Kitchen.

Kitchen now tracks its execution parameters. At the end of execution, it
exports those parameters, along with the annotation protobuf, to a
monitoring object which, in turn, derives a build event protobuf from
the parameters and uploads it to BigQuery.

BUG= chromium:742478 
TEST=None

Change-Id: I4ce267d887b1a72c814a81521304902042ba788c
Reviewed-on: https://chromium-review.googlesource.com/616187
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/git.go
[modify] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/build/result.pb.go
[modify] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/cook_test.go
[add] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/monitoring_test.go
[modify] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/build/result.proto
[modify] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/cook.go
[modify] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/git_test.go
[add] https://crrev.com/fa64a533e5b32a2f7aa751256f26c0779d6e66f6/go/src/infra/tools/kitchen/monitoring.go
Project Member

Comment 21 by bugdroid1@chromium.org, Aug 25 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e383f17866005c94da6152e329f8585dad416850

commit e383f17866005c94da6152e329f8585dad416850
Author: Dan Jacques <dnj@google.com>
Date: Fri Aug 25 03:49:54 2017

[kitchen] Log event Put errors.

BUG= chromium:742478 
TEST=None

Change-Id: I4ad8266fc04e4a6a740d1c1e39c7fe426b33d9b4
Reviewed-on: https://chromium-review.googlesource.com/634636
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/e383f17866005c94da6152e329f8585dad416850/go/src/infra/tools/kitchen/monitoring.go
Project Member

Comment 22 by bugdroid1@chromium.org, Aug 25 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/95d23194b7acee732144c0c0c3444f13afa728bc

commit 95d23194b7acee732144c0c0c3444f13afa728bc
Author: Dan Jacques <dnj@google.com>
Date: Fri Aug 25 06:11:05 2017

[eventupload] Enable Schema inference.

Currently, "eventupload" requires the schema to be populated from the
TableDef. "bqexport"-generated TableDefs are partial, and do not include
a full schema. Rather than generate the schema, fall back on the
"bigquery" package's automatic schema inference when no explicit schema
is provided.

BUG= chromium:742478 
TEST=local
  - Ran Kitchen locally, seems to fix upload error issue.

Change-Id: I46caab535bc8fd30360bfe8c3f3b1ae00d6e8ac1
Reviewed-on: https://chromium-review.googlesource.com/634605
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/95d23194b7acee732144c0c0c3444f13afa728bc/go/src/infra/libs/bqschema/tabledef/bqschema.go
[modify] https://crrev.com/95d23194b7acee732144c0c0c3444f13afa728bc/go/src/infra/libs/eventupload/eventupload.go
Project Member

Comment 23 by bugdroid1@chromium.org, Aug 25 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/3fbafdcecee683907fc814c4a132775d1ccf956d

commit 3fbafdcecee683907fc814c4a132775d1ccf956d
Author: Dan Jacques <dnj@google.com>
Date: Fri Aug 25 07:11:16 2017

[kitchen] Print monitoring upload count.

Print the monitoring upload count via "logging" so it shows up in
Swarming console.

TBR=nodir@chromium.org
BUG= chromium:742478 
TEST=None

Change-Id: I0efc102774075c397655f1b96f397620559facd6
Reviewed-on: https://chromium-review.googlesource.com/634926
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/3fbafdcecee683907fc814c4a132775d1ccf956d/go/src/infra/tools/kitchen/monitoring.go
Project Member

Comment 24 by bugdroid1@chromium.org, Aug 26 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e66f8de786f2a6b77a6b14989ff755f5b4e480c7

commit e66f8de786f2a6b77a6b14989ff755f5b4e480c7
Author: Dan Jacques <dnj@google.com>
Date: Sat Aug 26 00:03:19 2017

[kitchen] Fix "patch_url" parameter.

Fix the "patch_url" generation parameters. This value is a JSON number,
not a JSON string.

Update testing code to not attempt (and fail) to send monitoring events.

BUG= chromium:742478 
TEST=local
R=iannucci@chromium.org, nodir@chromium.org

Change-Id: If0bb4f636fd5edc2f677d2906d1db76c476ebeb5
Reviewed-on: https://chromium-review.googlesource.com/636744
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/e66f8de786f2a6b77a6b14989ff755f5b4e480c7/go/src/infra/tools/kitchen/cook.go
[modify] https://crrev.com/e66f8de786f2a6b77a6b14989ff755f5b4e480c7/go/src/infra/tools/kitchen/monitoring.go
[modify] https://crrev.com/e66f8de786f2a6b77a6b14989ff755f5b4e480c7/go/src/infra/tools/kitchen/monitoring_test.go
[modify] https://crrev.com/e66f8de786f2a6b77a6b14989ff755f5b4e480c7/go/src/infra/tools/kitchen/cook_test.go
Project Member

Comment 25 by bugdroid1@chromium.org, Aug 26 2017 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/957ad6ef2a0a28640075f4fde2bd6180bca3fd6f

commit 957ad6ef2a0a28640075f4fde2bd6180bca3fd6f
Author: Dan Jacques <dnj@google.com>
Date: Sat Aug 26 03:33:57 2017

Comment 26 by d...@chromium.org, Aug 26 2017

Status: Fixed (was: Available)
Kitchen is now sending complete events to BigQuery in prod. I think this can be closed.

Comment 27 by efoo@chromium.org, Sep 8 2017

Labels: LUCI-M2-S7

Comment 28 by efoo@chromium.org, Sep 8 2017

Labels: -LUCI-M2-S7 LUCI-M3-S7

Comment 29 by katthomas@google.com, Sep 15 2017

Cc: -no...@chromium.org d...@chromium.org
Owner: no...@chromium.org
Status: Assigned (was: Fixed)
Are there any queries that demonstrate that the data in BQ matches what we have in Dremel? I think until we have that, and we've migrated the Viceroy graphs that use completed_builds and completed_steps to MonApp we should keep this open.

Comment 30 by no...@chromium.org, Sep 15 2017 

This bug is scoped to LUCI. There are no LUCI build events in Dremel, therefore there is nothing to match. A build runs either on Buildbot or LUCI.

Comment 31 by efoo@chromium.org, Nov 7 2017

Labels: -REQBY-LUCI-M3-ClosedBeta REQBY-LUCI-M3-Beta
Nodir, shouldn't this be closed? What's the remaining work now that events are sent to BQ?

Comment 32 by no...@chromium.org, Nov 8 2017

Status: Started (was: Assigned)
it occurred to me that builds should be sent from buildbucket, not kitchen, because only bulidbucket (and its clients) know the final decided status of a build. Also can and should use idiomatic non-legacy table schema. I've actually started working on this and BotUpdateSlow today

Comment 33 by efoo@chromium.org, Jan 31 2018

Labels: LUCI-Dev2

Comment 34 by efoo@chromium.org, Jan 31 2018

Labels: -LUCI-M2-Dev2

Comment 35 by efoo@chromium.org, Feb 13 2018

Labels: -REQBY-LUCI-M3-Beta REQBY-LUCI-Beta

Comment 36 by efoo@chromium.org, Feb 15 2018

Labels: -REQBY-LUCI-Beta LUCI-Backlog
Project Member

Comment 37 by bugdroid1@chromium.org, Mar 3 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/recipes-py/+/b10224f3b916ec7aac1e6edbfc0fefa9635d15df

commit b10224f3b916ec7aac1e6edbfc0fefa9635d15df
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Mar 03 01:25:12 2018

Reserve pipe in step names.

Bug:  742478 
Change-Id: I2d15a1d78f887f7ba9916601874e2afe3537f2e8
Reviewed-on: https://chromium-review.googlesource.com/947051
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/b10224f3b916ec7aac1e6edbfc0fefa9635d15df/recipe_engine/run.py
Project Member

Comment 38 by bugdroid1@chromium.org, Mar 3 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/cf71b22754bcaf4ce181d20bd53ff599552538fb

commit cf71b22754bcaf4ce181d20bd53ff599552538fb
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Mar 03 14:47:34 2018

[buildbucket][v2] add annotation step parser

Will be used for export of steps to BQ.

Bug:  742478 
Change-Id: Icc0f8851ac4bba151b809b177a7aacc6754cd58d
Reviewed-on: https://chromium-review.googlesource.com/947397
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/proto/step.proto
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/test/annotations_test.py
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/__init__.py
[rename] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/builds.py
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/test/__init__.py
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/third_party/__init__.py
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/annotations.py
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/test/annotations.pb.txt
[rename] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/test/builds_test.py
[add] https://crrev.com/cf71b22754bcaf4ce181d20bd53ff599552538fb/appengine/cr-buildbucket/v2/test/expected_steps.pb.txt
Project Member

Comment 39 by bugdroid1@chromium.org, Mar 5 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/81b9f9e4730f109c028844ffdb27822d4234b940

commit 81b9f9e4730f109c028844ffdb27822d4234b940
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Mar 05 19:26:24 2018

[buidlbucket][bq] simplify and make more robust

An upcoming CL will add fetching of steps from LogDog which will introduce new
kinds of failures. Refactor code to prepare for them, and simplify too.

R=vadimsh@chromium.org

Bug:  742478 
Change-Id: I50f9d785f466230c5fd15be7b0d3cd43e7d0f39f
Reviewed-on: https://chromium-review.googlesource.com/949310
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/81b9f9e4730f109c028844ffdb27822d4234b940/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/81b9f9e4730f109c028844ffdb27822d4234b940/appengine/cr-buildbucket/bq.py
Project Member

Comment 40 by bugdroid1@chromium.org, Mar 5 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5c06cae89bf9479f85a90aab2c26c5857d2c2c31

commit 5c06cae89bf9479f85a90aab2c26c5857d2c2c31
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Mar 05 20:58:34 2018

[buildbucket] v2-related refactorings

Preparation for fetching steps from LogDog.

Bug:  742478 
Change-Id: Id8e91d0dc6fe02040f9800232492ee942c5aedc4
Reviewed-on: https://chromium-review.googlesource.com/949669
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/v2/test/annotations_test.py
[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/v2/__init__.py
[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/test/test_util.py
[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/v2/annotations.py
[modify] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/v2/test/builds_test.py
[add] https://crrev.com/5c06cae89bf9479f85a90aab2c26c5857d2c2c31/appengine/cr-buildbucket/v2/errors.py
Project Member

Comment 41 by bugdroid1@chromium.org, Mar 5 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/ec541674f3a929569bdab3c840dc177c43362708

commit ec541674f3a929569bdab3c840dc177c43362708
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Mar 05 23:01:24 2018

[net] json_request: specify Accept header

Specify "Accept: application/json; charset=utf-8" header when
making a JSON request.

Bug:  742478 
Change-Id: I65e88e7ab32ff24af8edd0d42476b835373bbd69
Reviewed-on: https://chromium-review.googlesource.com/949775
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/ec541674f3a929569bdab3c840dc177c43362708/appengine/components/components/net.py
[modify] https://crrev.com/ec541674f3a929569bdab3c840dc177c43362708/appengine/components/components/net_test.py
Project Member

Comment 42 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1a5fe66f53b38cbe363681a799b80d8eac625865

commit 1a5fe66f53b38cbe363681a799b80d8eac625865
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 02:43:18 2018
Project Member

Comment 43 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/f0f735c59fa0bd0abe51033c9fc81d876b2e0025

commit f0f735c59fa0bd0abe51033c9fc81d876b2e0025
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 05:40:28 2018
Project Member

Comment 44 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/15977d0cac82f302e0f30b3cacfd1aa2b2ec3bfb

commit 15977d0cac82f302e0f30b3cacfd1aa2b2ec3bfb
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 05:42:57 2018

[buildbucket] add config.get_settings_async

Move the common code from swarming.py

Bug:  742478 
Change-Id: I372f3803cd165e448050dad1157eb8a3f5804a4b
Reviewed-on: https://chromium-review.googlesource.com/950470
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/15977d0cac82f302e0f30b3cacfd1aa2b2ec3bfb/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/15977d0cac82f302e0f30b3cacfd1aa2b2ec3bfb/appengine/cr-buildbucket/config.py
Project Member

Comment 45 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/513e03d6a73fae0cc2e90ef186b7d643f2023611

commit 513e03d6a73fae0cc2e90ef186b7d643f2023611
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 12:41:25 2018
Project Member

Comment 46 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/356508eca10772b65ff6eecd8d63fb7658689cbc

commit 356508eca10772b65ff6eecd8d63fb7658689cbc
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 19:19:18 2018

[config] get remote provide config outside of txn

config/remote.py uses datastore_utils/config.py which requires current
context to be not in an ndb transcation. Create a new transaction-less context.

Bug:  742478 
Change-Id: Ifd8f5f981de821faec85295eef5ca9def1dfb538
Reviewed-on: https://chromium-review.googlesource.com/951893
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/356508eca10772b65ff6eecd8d63fb7658689cbc/appengine/components/components/config/remote.py
Project Member

Comment 47 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/062a6a398a52acf8338293664da3eeb9e3ccd8a0

commit 062a6a398a52acf8338293664da3eeb9e3ccd8a0
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 19:34:07 2018

Roll infra/luci/ 9fd58c83d..356508eca (5 commits)

https://chromium.googlesource.com/infra/luci/luci-py//+log/9fd58c83df39..356508eca107

$ git log 9fd58c83d..356508eca --date=short --no-merges --format='%ad %ae %s'
2018-03-06 nodir [config] get remote provide config outside of txn
2018-03-05 maruel swarming_bot: android state 'host_dimensions' to use the default dimensions
2018-03-05 nodir [net] json_request: specify Accept header
2018-03-02 flowblok Add a health check tool for use by the deployment tools.
2018-02-28 iannucci [isolateserver] Do not correct python command in isolateserver.

Created with:
  roll-dep infra/luci

R=vadimsh@chromium.org

Bug:  742478 
Change-Id: I2bc54fad2d875f1a61cd379f5eeb460ae79ff9f4
Reviewed-on: https://chromium-review.googlesource.com/951634
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/062a6a398a52acf8338293664da3eeb9e3ccd8a0/DEPS
Project Member

Comment 48 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8f19c83177e03b713b4f9182a4c48a6f1ce1a947

commit 8f19c83177e03b713b4f9182a4c48a6f1ce1a947
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 19:38:07 2018

[buildbucket][bq-export] add bucket whitelist

BigQuery export implementation assumes a certain format of builds and won't
work for builds that do not confirm that format.
Add a whitelist for buckets in a service config.

R=vadimsh@chromium.org

Bug:  742478 
Change-Id: Ib117219e145916ebc86212d3a7782b5b5a6fc108
Reviewed-on: https://chromium-review.googlesource.com/950682
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/test/config_test.py
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/proto/service_config_pb2.py
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/proto/service_config.proto
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/config.py
[modify] https://crrev.com/8f19c83177e03b713b4f9182a4c48a6f1ce1a947/appengine/cr-buildbucket/test/service_test.py
Project Member

Comment 49 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/87fd1e8b858132aa6589531906df12c33ae1b4a0

commit 87fd1e8b858132aa6589531906df12c33ae1b4a0
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 19:45:55 2018

Comment 50 by efoo@chromium.org, Mar 6 2018

Labels: LUCI-Chromium-CQSets LUCI-Blocker-Chromium-CQSets

Comment 51 by efoo@chromium.org, Mar 6 2018

Labels: -LUCI-Backlog
Project Member

Comment 52 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/eefefff7d81f76edb9c07e49cf2c282ff8ed7c58

commit eefefff7d81f76edb9c07e49cf2c282ff8ed7c58
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 22:05:58 2018

[config] retrieve last good config outside of transaction

Call LastGoodConfig.get_by_id_async non-transactionally.

R=vadimsh@chromium.org

Bug:  742478 
Change-Id: I5d003458afda8a1c4cf4cf58b7e4584a09cde491
Reviewed-on: https://chromium-review.googlesource.com/951857
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/eefefff7d81f76edb9c07e49cf2c282ff8ed7c58/appengine/components/components/config/remote.py
Project Member

Comment 53 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1bf6b06fbd5a49d3979e37b1ef945f45b5bab8d4

commit 1bf6b06fbd5a49d3979e37b1ef945f45b5bab8d4
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 22:18:39 2018

Roll infra/luci/ 356508eca..eefefff7d (1 commit)

https://chromium.googlesource.com/infra/luci/luci-py//+log/356508eca107..eefefff7d81f

$ git log 356508eca..eefefff7d --date=short --no-merges --format='%ad %ae %s'
2018-03-06 nodir [config] retrieve last good config outside of transaction

Created with:
  roll-dep infra/luci

TBR=vadimsh@chromium.org

Bug:  742478 
Change-Id: I6535f5c28f1ee5781d055c028a9a270582c68097
Reviewed-on: https://chromium-review.googlesource.com/952162
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/1bf6b06fbd5a49d3979e37b1ef945f45b5bab8d4/DEPS
Project Member

Comment 54 by bugdroid1@chromium.org, Mar 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/99f713eef832a47fdcc5bc6c5d3320a3e08a4730

commit 99f713eef832a47fdcc5bc6c5d3320a3e08a4730
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Mar 06 23:29:32 2018

[buildbucket] reserve swarming tags

Reject build creation requests with tag keys "swarming_tag" or
"swarming_dimensions".

Bug:  742478 
Change-Id: Ide2b1f994b2bf21a347a52b533d8de2760dc3045
Reviewed-on: https://chromium-review.googlesource.com/952373
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/99f713eef832a47fdcc5bc6c5d3320a3e08a4730/appengine/cr-buildbucket/service.py
Project Member

Comment 55 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/bf784a4adc7f643c5f7e5cf730ef776c8f48f865

commit bf784a4adc7f643c5f7e5cf730ef776c8f48f865
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 02:03:48 2018

[buildbucket][bq-export] export steps

For each build being exported to BigQuery, when converting to v2, fetch
steps from LogDog in annotation format.

Do not mark a build as processed until the logdog stream is terminated and we
have the latest version of the annotation proto.

Bug:  742478 
Change-Id: If7d0e547fd57a9d4b7f80eeea82a566528363213
Reviewed-on: https://chromium-review.googlesource.com/949572
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/proto/service_config_pb2.py
[add] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/v2/test/steps_test.py
[modify] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/proto/service_config.proto
[modify] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/v2/__init__.py
[modify] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/v2/errors.py
[add] https://crrev.com/bf784a4adc7f643c5f7e5cf730ef776c8f48f865/appengine/cr-buildbucket/v2/steps.py
Project Member

Comment 56 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/5708a6fa7e2081b96a80be0f536788da0b01831c

commit 5708a6fa7e2081b96a80be0f536788da0b01831c
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 02:15:08 2018
Project Member

Comment 57 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6eb4c99a77b5da041ae5ba433e4aee4d334a3217

commit 6eb4c99a77b5da041ae5ba433e4aee4d334a3217
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 02:38:48 2018

[buildbucket] make authenticated requests to LogDog

When fetching annotation proto from LogDog, make an authenticated request.

TBR=vadimsh@chromium.org

Bug:  742478 
Change-Id: I4c4d66d6e28ef4c1774116500c400e3b1268e202
Reviewed-on: https://chromium-review.googlesource.com/952667
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/6eb4c99a77b5da041ae5ba433e4aee4d334a3217/appengine/cr-buildbucket/v2/steps.py
Project Member

Comment 58 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/2b520545feda34895e4250ba895e2eb166220d91

commit 2b520545feda34895e4250ba895e2eb166220d91
Author: Michael Achenbach <machenbach@chromium.org>
Date: Wed Mar 07 11:48:38 2018
Project Member

Comment 59 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/8e9c5764de0d3c30c716e7fde3332728e8f9e304

commit 8e9c5764de0d3c30c716e7fde3332728e8f9e304
Author: Michael Achenbach <machenbach@google.com>
Date: Wed Mar 07 13:07:42 2018
Project Member

Comment 60 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/413ed03baac1df87ea0251655bb87817055d4c61

commit 413ed03baac1df87ea0251655bb87817055d4c61
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 19:56:07 2018

[buildbucket][bq-export] improve non-finalized build handling

bq.py reads chains of properties in a nested dicts. Some of them may be
present and be None. Handle that.

Treat TIMED_OUT same way as BOT_DIED: in both cases, it is OK if steps
are not finalized.

Reduce waiting threshold from 1h to 20m.

R=tandrii@chromium.org

Bug:  742478 
Change-Id: I5479f7f6d7cb1e2ab2468cbba19fcdff5e4eb376
Reviewed-on: https://chromium-review.googlesource.com/952780
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/413ed03baac1df87ea0251655bb87817055d4c61/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/413ed03baac1df87ea0251655bb87817055d4c61/appengine/cr-buildbucket/bq.py
Project Member

Comment 61 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/4e3e9f800951c63319c93edca4c5f9e6ca031550

commit 4e3e9f800951c63319c93edca4c5f9e6ca031550
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 20:51:56 2018

[buildbucket][bq-export] allow losing Buildbot steps

It is hard to find all legacy/Buildbot cases where annotation URL can be
not present.

Require annotation URL only in Swarmbucket builds.
In non-Swarmbucket builds, if annotation URL is not detected, leave steps empty.

R=vadimsh@chromium.org

Bug:  742478 
Change-Id: Ife14b8a281613096df49ec9c514a7689298be87b
Reviewed-on: https://chromium-review.googlesource.com/953066
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/4e3e9f800951c63319c93edca4c5f9e6ca031550/appengine/cr-buildbucket/v2/steps.py
[modify] https://crrev.com/4e3e9f800951c63319c93edca4c5f9e6ca031550/appengine/cr-buildbucket/v2/test/steps_test.py
Project Member

Comment 62 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7e905a0016169955a6a7f990b7265edfd5ab0866

commit 7e905a0016169955a6a7f990b7265edfd5ab0866
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 21:38:57 2018

[buildbucket][bq-export] fix "will retry" logging

bq.py logged all build ids as the ones that it will retry instead of the ones
it will retry.

TBR=vadimsh@chromium..org

Bug:  742478 
Change-Id: I60e60bb9b7c38b3215a735af6ad469f325da6a02
Reviewed-on: https://chromium-review.googlesource.com/953645
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/7e905a0016169955a6a7f990b7265edfd5ab0866/appengine/cr-buildbucket/bq.py
Project Member

Comment 63 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/251344861357dfc5b913e27a374fd81fbe9e771a

commit 251344861357dfc5b913e27a374fd81fbe9e771a
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 22:53:07 2018

[buildbucket][bq-export] skip non-swarmbucket steps

Not all Buildbot builds that have annotation URL have valid annotation proto,
i.e. some of them are invalid annotation protos, not parsable.

Skip Buildbot steps altogether.

R=hinoka@chromium.org, vadimsh@chromium.org

Bug:  742478 
Change-Id: Ib4a67d9177a89a27639630b9499ac09bea99d37e
Reviewed-on: https://chromium-review.googlesource.com/953657
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/251344861357dfc5b913e27a374fd81fbe9e771a/appengine/cr-buildbucket/v2/steps.py
[modify] https://crrev.com/251344861357dfc5b913e27a374fd81fbe9e771a/appengine/cr-buildbucket/v2/test/steps_test.py
Project Member

Comment 65 by bugdroid1@chromium.org, Mar 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7344c75c464360aff712bf99af275e296b7c7578

commit 7344c75c464360aff712bf99af275e296b7c7578
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Mar 07 23:29:17 2018

[buildbucket] use F2 instance class on backend

Loading a lot of annotation logs from LogDog causes OOMs in backend.

F1 has 128 MB memory limit and costs $0.05/h.
F2 has 256 MB memory limit and costs $0.10/h.
We have 6 instances on backend module.

We use F2 for backend.

TBR=vadimsh@chromium.org

Bug:  742478 
Change-Id: I6ce83eefe9aca782f7de5394d0e3c622e6787479
Reviewed-on: https://chromium-review.googlesource.com/954008
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/7344c75c464360aff712bf99af275e296b7c7578/appengine/cr-buildbucket/module-backend.yaml
Project Member

Comment 66 by bugdroid1@chromium.org, Mar 8 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe

commit 246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 08 01:18:37 2018

[buildbucket][bq-export] delete logdog code

Loading steps from LogDog proved to be overcomplicated.
Delete this implementation.

Next CL will implement steps in a much simpler way, using annotations
fetched from isolates (already done) on build completion.

TBR=vadimsh@chromium.org
Bug:  742478 
Change-Id: Iec46f422c03759da2f7b7d207ec11d045afbef0f
Reviewed-on: https://chromium-review.googlesource.com/954340
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe/appengine/cr-buildbucket/proto/service_config_pb2.py
[delete] https://crrev.com/8d6758cd272d9bd8707922bf98f1c69ff3a315b6/appengine/cr-buildbucket/v2/test/steps_test.py
[modify] https://crrev.com/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe/appengine/cr-buildbucket/proto/service_config.proto
[modify] https://crrev.com/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe/appengine/cr-buildbucket/v2/__init__.py
[modify] https://crrev.com/246f28cb1be6a48af2d1ecf2332b3cffb0e5cbfe/appengine/cr-buildbucket/v2/errors.py
[delete] https://crrev.com/8d6758cd272d9bd8707922bf98f1c69ff3a315b6/appengine/cr-buildbucket/v2/steps.py
Project Member

Comment 67 by bugdroid1@chromium.org, Mar 8 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/bd7add1ff31aaa6177a53bf07f2700c62c8dcbb9

commit bd7add1ff31aaa6177a53bf07f2700c62c8dcbb9
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 08 01:58:17 2018

[buildbucket][bq-export] rename annotations.py

annotations.py's purpose is to convert steps from annotation format to v2
format. Its file name is not great. It is really about steps.
Rename to steps.py

Bug:  742478 
Change-Id: I0446bf3073ead8064d13b07d564868f78835a3cf
Reviewed-on: https://chromium-review.googlesource.com/954357
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[rename] https://crrev.com/bd7add1ff31aaa6177a53bf07f2700c62c8dcbb9/appengine/cr-buildbucket/v2/steps.py
[rename] https://crrev.com/bd7add1ff31aaa6177a53bf07f2700c62c8dcbb9/appengine/cr-buildbucket/v2/test/steps_test.py
Project Member

Comment 68 by bugdroid1@chromium.org, Mar 8 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/bd684ba6871502db9cd8e0113db522d0cf3f395b

commit bd684ba6871502db9cd8e0113db522d0cf3f395b
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 08 03:25:32 2018
Project Member

Comment 69 by bugdroid1@chromium.org, Mar 8 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1c9bc242354927d0fc63093d90c7828aec23b41d

commit 1c9bc242354927d0fc63093d90c7828aec23b41d
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 08 03:41:51 2018
Project Member

Comment 70 by bugdroid1@chromium.org, Mar 8 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/bb77050746af6b5ca34f84dad2ee0e068ca6be59

commit bb77050746af6b5ca34f84dad2ee0e068ca6be59
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Mar 08 22:10:04 2018

[buildbucket][bq-export] export steps

When loading kitchen result and finalizing a Swarmbucket build, persist
annotation proto from kitchen result to the datastore.

When exporting a build to BigQuery, load the annotation proto from datastore,
extract steps and include them in the BigQuery row.

Bug:  742478 
Change-Id: Ic5f70c0609392b1c3251c71b03e3e1a83b1dfabb
Reviewed-on: https://chromium-review.googlesource.com/954604
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/bb77050746af6b5ca34f84dad2ee0e068ca6be59/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/bb77050746af6b5ca34f84dad2ee0e068ca6be59/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/bb77050746af6b5ca34f84dad2ee0e068ca6be59/appengine/cr-buildbucket/v2/__init__.py
[modify] https://crrev.com/bb77050746af6b5ca34f84dad2ee0e068ca6be59/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/bb77050746af6b5ca34f84dad2ee0e068ca6be59/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/bb77050746af6b5ca34f84dad2ee0e068ca6be59/appengine/cr-buildbucket/model.py

Comment 71 by no...@chromium.org, Mar 8 2018

Status: Fixed (was: Started)
announcement: g/luci-announce/IVdte3E-FRw/56A9NhHzAwAJ
►

Sign in to add a comment