New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 782489 link

Starred by 1 user
Status: Fixed
Owner:
█ iannucci@chromium.org
Last visit > 30 days ago
Closed: Nov 2017
Cc:
tandrii@chromium.org
no...@chromium.org
hinoka@chromium.org
efoo@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature



Sign in to add a comment

Milo needs to double-index LUCI tasks so that console can switch between them w.r.t. LUCI-active or not

Project Member Reported by iannucci@chromium.org, Nov 8 2017 
In order to make the milo call actually accurately switch the console column (along with switching prod/non-prod behavior in the recipes), the console needs to be populated correctly w.r.t. which builds were active at any given revision.

Proposal:
  * Add "IsProd" to BuildSummary (populated by consulting milo's internal emulation apis)
  * Index both buildbot and LUCI builds by their alter ego.
  * Console filters out non-IsProd summaries.

Comment 1 by iannucci@chromium.org, Nov 8 2017 

wrinkle: we need a way to set the bucket for an emulated master/builder without setting the emulation number.

Comment 2 by no...@chromium.org, Nov 8 2017

Labels: -Type-Bug -Pri-3 Pri-1 Type-Feature
Is IsProd a field? Whether a build is prod or not is not a constant. I think it should be computed on demand (render time)

why do we need to index both buildbot and LUCI twice? I think just one is sufficient to solve the problem. If it does not matter which one we index twice, I'd prefer buildbot builds to be indexed twice because eventually we will have no buildbot builds, so we won't have to disable double indexing

Pri-1 because it is needed for migration
Project Member

Comment 3 by bugdroid1@chromium.org, Nov 8 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/1b8a190c83a1a450c05f13fe5278028f38cfc821

commit 1b8a190c83a1a450c05f13fe5278028f38cfc821
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Nov 08 07:37:28 2017

[milo] Allow setting buildnumber to < 0 to disable emulation.

This will allow the emulation options to still contain a bucket, even
before the builder needs emulation (so that way console indexing can
double index both buildbot and LUCI builds).

R=hinoka@chromium.org, jchinlee@chromium.org, nodir@chromium.org

Bug:  782489 
Change-Id: I9b2b1529ca937d6f1e696824be0ab39811b1fd39
Reviewed-on: https://chromium-review.googlesource.com/758052
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/api/proto/buildbot.pb.go
[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/api/proto/buildbot.proto
[add] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/api/proto/buildbot_extra.go
[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/api/proto/pb.discovery.go
[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/buildsource/buildbot/builder.go
[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/buildsource/buildbot/buildstore/build.go
[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/buildsource/buildbot/buildstore/emulation.go
[modify] https://crrev.com/1b8a190c83a1a450c05f13fe5278028f38cfc821/milo/buildsource/buildbot/buildstore/master.go
Project Member

Comment 4 by bugdroid1@chromium.org, Nov 9 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/803248fb69927c89eeabf4eca697db2447185df3

commit 803248fb69927c89eeabf4eca697db2447185df3
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Nov 09 08:20:59 2017

[milo] Console caching cleanup.

  * Adds per-request caching for common.GetAllConsoles()
  * Removes ad-hoc `BuildSummary.consoles` cache thingy
  * Simplifies manifest key interface
  * Makes BuilderSummary extract console IDs from ManifestKeys
  * Parallelizes all tests
  * Fix test inter-dependency by moving context initialization inside
    Convey (previously tests were persisting state between tests).

R=hinoka@chromium.org, jchinlee@chromium.org, nodir@chromium.org, tandrii@chromium.org

Bug:  782489 ,  782874 
Change-Id: I8c84482a7b0f6f8584c86349872cc26546432527
Reviewed-on: https://chromium-review.googlesource.com/759917
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbot/build_test.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbot/builder_test.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbot/buildinfo_test.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbot/buildstore/build.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbot/grpc_test.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbot/pubsub_test.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbucket/pubsub.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/buildsource/buildbucket/pubsub_test.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/common/config.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/common/model/build_summary.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/common/model/builder_summary.go
[modify] https://crrev.com/803248fb69927c89eeabf4eca697db2447185df3/milo/common/model/builder_summary_test.go
Project Member

Comment 6 by bugdroid1@chromium.org, Nov 9 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d3dd886b430818b4eadfa2f0690953b1465ddba3

commit d3dd886b430818b4eadfa2f0690953b1465ddba3
Author: Ryan Tseng <hinoka@chromium.org>
Date: Thu Nov 09 20:58:14 2017

Revert "[milo] Index BuildSummary.Experimental."

This reverts commit 9eafef0833ea35c0b5756d87c223500e62be60ef.

Reason for revert: Speculative revert: Console broken because
manifest keys aren't getting populated, and BuildSummaries aren't working

Original change's description:
> [milo] Index BuildSummary.Experimental.
> 
> R=​hinoka@chromium.org, jchinlee@chromium.org, nodir@chromium.org, tandrii@chromium.org
> 
> Bug:  782489 
> Change-Id: Ib23bd32b8ee4588998d381c649aa19a48d139171
> Reviewed-on: https://chromium-review.googlesource.com/759625
> Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,nodir@chromium.org,jchinlee@chromium.org,tandrii@chromium.org

Change-Id: I234afed2f4e3bb3e663dc08d9b2477ca76a52c3c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  782489 
Reviewed-on: https://chromium-review.googlesource.com/761440
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/d3dd886b430818b4eadfa2f0690953b1465ddba3/buildbucket/build.go
[modify] https://crrev.com/d3dd886b430818b4eadfa2f0690953b1465ddba3/milo/buildsource/buildbot/buildstore/build.go
[modify] https://crrev.com/d3dd886b430818b4eadfa2f0690953b1465ddba3/milo/buildsource/buildbucket/pubsub.go
[modify] https://crrev.com/d3dd886b430818b4eadfa2f0690953b1465ddba3/milo/buildsource/buildbucket/pubsub_test.go
[modify] https://crrev.com/d3dd886b430818b4eadfa2f0690953b1465ddba3/milo/common/model/build_summary.go
[modify] https://crrev.com/d3dd886b430818b4eadfa2f0690953b1465ddba3/milo/common/model/builder_summary_test.go
Project Member

Comment 7 by bugdroid1@chromium.org, Nov 9 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e0239b3d14161869fddccf55fe8df294d5dd69f8

commit e0239b3d14161869fddccf55fe8df294d5dd69f8
Author: Ryan Tseng <hinoka@chromium.org>
Date: Thu Nov 09 20:58:44 2017

Revert "[milo] Console caching cleanup."

This reverts commit 803248fb69927c89eeabf4eca697db2447185df3.

Reason for revert: Speculative revert: Console broken because manifest keys aren't getting populated, and BuildSummaries aren't working

Original change's description:
> [milo] Console caching cleanup.
> 
>   * Adds per-request caching for common.GetAllConsoles()
>   * Removes ad-hoc `BuildSummary.consoles` cache thingy
>   * Simplifies manifest key interface
>   * Makes BuilderSummary extract console IDs from ManifestKeys
>   * Parallelizes all tests
>   * Fix test inter-dependency by moving context initialization inside
>     Convey (previously tests were persisting state between tests).
> 
> R=​hinoka@chromium.org, jchinlee@chromium.org, nodir@chromium.org, tandrii@chromium.org
> 
> Bug:  782489 ,  782874 
> Change-Id: I8c84482a7b0f6f8584c86349872cc26546432527
> Reviewed-on: https://chromium-review.googlesource.com/759917
> Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,nodir@chromium.org,jchinlee@chromium.org,tandrii@chromium.org

Change-Id: I1e24cc7e6c0cc714ed9d1f1f3dba11a3a27d6767
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  782489 ,  782874 
Reviewed-on: https://chromium-review.googlesource.com/761438
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbot/build_test.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbot/builder_test.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbot/buildinfo_test.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbot/buildstore/build.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbot/grpc_test.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbot/pubsub_test.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbucket/pubsub.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/buildsource/buildbucket/pubsub_test.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/common/config.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/common/model/build_summary.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/common/model/builder_summary.go
[modify] https://crrev.com/e0239b3d14161869fddccf55fe8df294d5dd69f8/milo/common/model/builder_summary_test.go

Comment 8 by iannucci@chromium.org, Nov 11 2017

Owner: iannucci@chromium.org
Status: Started (was: Untriaged)
(I'm working on this btw)
Project Member

Comment 9 by bugdroid1@chromium.org, Nov 11 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a0365e74dbc871619c8e31f041f9acafbafb7a86

commit a0365e74dbc871619c8e31f041f9acafbafb7a86
Author: Robert Iannucci <iannucci@chromium.org>
Date: Sat Nov 11 01:28:47 2017

Reland [milo] Console caching cleanup.

  * Adds per-request caching for common.GetAllConsoles()
  * Removes ad-hoc `BuildSummary.consoles` cache thingy
  * Simplifies manifest key interface
  * Makes BuilderSummary extract console IDs from ManifestKeys
  * Parallelizes all tests
  * Fix test inter-dependency by moving context initialization inside
    Convey (previously tests were persisting state between tests).

R=hinoka@chromium.org, jchinlee@chromium.org, nodir@chromium.org, tandrii@chromium.org

Bug:  782489 
Change-Id: Ia769a74a1600de1dcf2ea8131efc78eefd2f7698
Reviewed-on: https://chromium-review.googlesource.com/764382
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbot/build_test.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbot/builder_test.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbot/buildinfo_test.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbot/buildstore/build.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbot/grpc_test.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbot/pubsub_test.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbucket/pubsub.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/buildsource/buildbucket/pubsub_test.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/common/config.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/common/model/build_summary.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/common/model/builder_summary.go
[modify] https://crrev.com/a0365e74dbc871619c8e31f041f9acafbafb7a86/milo/common/model/builder_summary_test.go
Project Member

Comment 11 by bugdroid1@chromium.org, Nov 11 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/de8fcc919c73c87ebcd33856080d61d2f7435601

commit de8fcc919c73c87ebcd33856080d61d2f7435601
Author: Robert Iannucci <iannucci@chromium.org>
Date: Sat Nov 11 05:40:09 2017

[milo] Make Console datastore entity store full Console message.

Will allow config import to preserve full proto message fidelity and
avoids an extra step of indirection.

The attached bug will need to make `Builder.Name` a repeated field, which
will be unnecessarially difficult to represent in the current Console
datastore scheme.

R=hinoka@chromium.org, jchinlee@chromium.org, nodir@chromium.org

Bug:  782489 
Change-Id: I160289bd7b37ab7a4551a2aa87172ec09aef9044
Reviewed-on: https://chromium-review.googlesource.com/765050
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/api/config/generate.go
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/api/config/proto_gae.gen.go
[add] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/api/config/util.go
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/buildsource/console.go
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/common/config.go
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/common/config_test.go
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/frontend/appengine/templates/pages/consoles.html
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/frontend/routes_test.go
[modify] https://crrev.com/de8fcc919c73c87ebcd33856080d61d2f7435601/milo/frontend/view_console.go

Comment 13 by iannucci@chromium.org, Nov 15 2017

Status: Fixed (was: Started)
Ok, I think this part is done now

Comment 14 by efoo@chromium.org, Nov 16 2017

Labels: REQBY-LUCI-M4a-MacWinPOC LUCI-M3-S11 LUCI-M3-Beta
Adding labels for tracking

Comment 15 by efoo@chromium.org, Jan 31 2018

Labels: LUCI-Beta

Comment 16 by efoo@chromium.org, Jan 31 2018

Labels: -LUCI-Blocker-M4 -LUCI-M3-Beta luci-blocker-migration

Comment 17 by efoo@chromium.org, Feb 13 2018

Labels: -LUCI-blocker-migration LUCI-Chromium-CQSets LUCI-Blocker-Chromium-CQSets
►

Sign in to add a comment