New issue
Advanced search Search tips

Issue 783337 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug

Blocking:
issue 731553



Sign in to add a comment

LUCI migration app is not WAI

Project Member Reported by efoo@chromium.org, Nov 9 2017

Issue description

Due to a bug (likely an issue in buildbucket), we have not seen any parity reminder emails sent from migration app. 

Reviewing the migration app, "Not enough data to analyze" appears for all builders and both LUCI correctness and speed show 0%. 

Percentage values are also empty on the master list pages as well. 
i.e. https://luci-migration.appspot.com/masters/tryserver.chromium.linux/

Bonus task: is there some alerting we can add to let us know when migration app failed? We will be relying on this heavily during migrations and would be good to get a heads up when data is unreliable. 

Filing this bug for tracking. 
 
The bug filing, which I previous thought was too spammy :P, was a pretty good signal, but if there's an easy way to add actual alerting that would be nice to have.

You could probably make this pri-1 instead of 0 but if you're working on it right now either seems fine.

Comment 2 by no...@chromium.org, Nov 10 2017

luci linux_chromium_rel_ng broke 3 times in 2 days for 3 different reasons. I think we cannot afford re-enabling dogfooding without this monitoring

Comment 3 by no...@chromium.org, Nov 10 2017

Status: Started (was: Assigned)

Comment 4 by efoo@chromium.org, Nov 10 2017

>> I think we cannot afford re-enabling dogfooding without this monitoring
+1. This is definitely a P0 ATM. 


Project Member

Comment 5 by bugdroid1@chromium.org, Nov 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/18f5ffa7eabf17c06cf290e64476e911f40af34e

commit 18f5ffa7eabf17c06cf290e64476e911f40af34e
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Nov 11 00:31:27 2017

[buildbucket] fix search

Search did not ensure that "next_cursor" field is included in a partial
response

Bug:  783337 
Change-Id: Idceaaa9de340b5bda3bdeab553fb35d6050f323f
Reviewed-on: https://chromium-review.googlesource.com/764586
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/18f5ffa7eabf17c06cf290e64476e911f40af34e/common/api/buildbucket/buildbucket/v1/search.go
[modify] https://crrev.com/18f5ffa7eabf17c06cf290e64476e911f40af34e/common/api/buildbucket/buildbucket/v1/search_test.go

Project Member

Comment 6 by bugdroid1@chromium.org, Nov 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fb6e95fa34305099a1a69afe4848dff47cefd639

commit fb6e95fa34305099a1a69afe4848dff47cefd639
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Nov 11 01:15:53 2017

[luci-migration] run analysis more often

linux_chromium_rel_ng analysis take ~10s.
Run analysis more often to discovery breakage sooner

Also remove randomization. It is not really useful with a task queue

Bug:  783337 
Change-Id: Iae67b17e847055720ab3e9c6f0c6c2d180bf622a
Reviewed-on: https://chromium-review.googlesource.com/765107
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/fb6e95fa34305099a1a69afe4848dff47cefd639/go/src/infra/appengine/luci-migration/app/analysis.go
[modify] https://crrev.com/fb6e95fa34305099a1a69afe4848dff47cefd639/go/src/infra/appengine/luci-migration/app/cron.yaml

Comment 7 by no...@chromium.org, Nov 11 2017

the migration app is fixed and correctly reports that up to 7h ago, there were 14 inconsistent build groups.

I still need to make analysis algorithm deterministic so that we can re-enable emails
Project Member

Comment 8 by bugdroid1@chromium.org, Nov 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/bb1ef541452d43a5b130f807804ee9f1fd1af1d7

commit bb1ef541452d43a5b130f807804ee9f1fd1af1d7
Author: Nodir Turakulov <nodir@chromium.org>
Date: Sun Nov 12 03:10:18 2017

Revert "[luci-migration] run analysis more often"

This reverts commit fb6e95fa34305099a1a69afe4848dff47cefd639.

Reason for revert: cause buildbucket to hit frontend time quota

Original change's description:
> [luci-migration] run analysis more often
> 
> linux_chromium_rel_ng analysis take ~10s.
> Run analysis more often to discovery breakage sooner
> 
> Also remove randomization. It is not really useful with a task queue
> 
> Bug:  783337 
> Change-Id: Iae67b17e847055720ab3e9c6f0c6c2d180bf622a
> Reviewed-on: https://chromium-review.googlesource.com/765107
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Erik Staab <estaab@chromium.org>
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>

TBR=nodir@chromium.org,estaab@chromium.org,tandrii@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  783337 
Change-Id: I24e3a38ad7f6219eb19d88948e254f5181d9a865
Reviewed-on: https://chromium-review.googlesource.com/764633
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/bb1ef541452d43a5b130f807804ee9f1fd1af1d7/go/src/infra/appengine/luci-migration/app/analysis.go
[modify] https://crrev.com/bb1ef541452d43a5b130f807804ee9f1fd1af1d7/go/src/infra/appengine/luci-migration/app/cron.yaml

Comment 9 by efoo@chromium.org, Nov 16 2017

Labels: LUCI-M3-Beta REQBY-LUCI-M4a-MacWinPOC LUCI-M3-S12 LUCI-M3-S11
Assign to S11 and S12 sprints

Comment 10 by no...@chromium.org, Nov 16 2017

Blocking: 731553
Project Member

Comment 11 by bugdroid1@chromium.org, Nov 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/eaaedf26de4c1da5e17f5acf4691ca25fe95a7fb

commit eaaedf26de4c1da5e17f5acf4691ca25fe95a7fb
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Nov 16 19:21:51 2017

[luci-migration] run analysis once in 4h

running analysis too often exhausts buildbucket quota. Long term we will
switch to BQ, but for now do analysis less frequently

TBR=estaab@chromium.org
Bug:  783337 
Change-Id: I31929733134d995cc3b58121b7ca06dc95e896f9
Reviewed-on: https://chromium-review.googlesource.com/775037
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/eaaedf26de4c1da5e17f5acf4691ca25fe95a7fb/go/src/infra/appengine/luci-migration/app/cron.yaml

Comment 12 by efoo@chromium.org, Nov 21 2017

Labels: LUCI-KnownIssues-UI

Comment 13 by efoo@chromium.org, Nov 21 2017

Labels: -LUCI-KnownIssues-UI
Cc: efoo@chromium.org
Labels: LUCI-Blocker-TaskForce
Status: Fixed (was: Started)

Comment 16 by efoo@chromium.org, Jan 31 2018

Labels: LUCI-Beta

Comment 17 by efoo@chromium.org, Jan 31 2018

Labels: -LUCI-Blocker-M4 -LUCI-M3-Beta luci-blocker-migration

Comment 18 by efoo@chromium.org, Feb 13 2018

Labels: -LUCI-blocker-migration LUCI-Chromium-CQSets LUCI-Blocker-Chromium-CQSets

Sign in to add a comment