New issue
Advanced search Search tips

Issue 634014 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: ----



Sign in to add a comment

master.chromium.perf.fyi console is down

Project Member Reported by no...@chromium.org, Aug 3 2016

Issue description

https://build.chromium.org/p/chromium.perf.fyi/console
responds with HTTP 500.
From logs:

     exceptions.KeyError: u'14a3cda4337f498d01ccbf897884f343ee099805'

root cause: buggy gitiles poller
trigger: ?

related issue:  issue 626019 
I've verified that the fix in that bug works correctly, so the trigger here is different
 

Comment 1 by no...@chromium.org, Aug 3 2016

proof that buildbucket does not create duplicate changes table rows:

chromiumperffyi=> select * from (select revision, count(*) as c from changes group by revision) as c order by c.c desc limit 10;
                 revision                 | c
------------------------------------------+---
 0171b362d74772f1a57a91f9fd4761569e30fa91 | 1
 eea862d56d10b29a347d49bac268af0bf9f57d31 | 1
 e00674ac2c822e76c19f38bc6b7c79a0d8d980d2 | 1
 93c60cd3ef5908984e173112dc09345bb5714036 | 1
 e82f23b3adc13c37652719585b0ddf62a514e07f | 1
 138740d5a899e146fb2c3c7d5185f924ff3e8dba | 1
 63942671b93caa41959910a47bc834c5c68df15c | 1
 bc913a00924cc5e9072293b5d7c19cf1cc536b62 | 1
 2fe4e328552fe9e4f38b66be416c6be5d113113b | 1
 deb84849f66aa9db12a4c4bd8eb0bb3e18fd441d | 1
(10 rows)

chromiumperffyi=> select count(*) from changes where revision = '' or revision is null;
 count
-------
     0
(1 row)

Comment 2 by no...@chromium.org, Aug 3 2016

Owner: no...@chromium.org
Status: Started (was: Untriaged)
the trigger is that chromium/src commit rate surpassed current change horizon configuration value (3000).  We had 3967 commits since https://crrev.com/14a3cda4337f498d01ccbf897884f343ee099805

I will double the change horizon and re-populate changes table
Project Member

Comment 3 by bugdroid1@chromium.org, Aug 3 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/78b745cdd539c16257999ed7a08b70b5dc91a5a4

commit 78b745cdd539c16257999ed7a08b70b5dc91a5a4
Author: nodir <nodir@chromium.org>
Date: Wed Aug 03 16:41:28 2016

chromium.perf.fyi: double changeHorizon

changehorizon is max number of changes that we store in the buildbot's
"changes" table. This number is not enough for chromium/src commit rate
as of today, at least for chromium.perf.fyi. Space is cheap.

R=vadimsh@chromium.org
BUG= 634014 

Review-Url: https://codereview.chromium.org/2208433004

[modify] https://crrev.com/78b745cdd539c16257999ed7a08b70b5dc91a5a4/masters/master.chromium.perf.fyi/master.cfg

Project Member

Comment 4 by bugdroid1@chromium.org, Aug 3 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/e663e5bd4130f861a67d149f562260d014428db8

commit e663e5bd4130f861a67d149f562260d014428db8
Author: nodir <nodir@google.com>
Date: Wed Aug 03 16:43:57 2016

Comment 5 by no...@chromium.org, Aug 3 2016

i have to manually restart master (without master manager) because I need to stop it, then modify .gitiles_poller_cursor, then start it

Comment 6 by no...@chromium.org, Aug 3 2016

Status: Fixed (was: Started)
https://build.chromium.org/p/chromium.perf.fyi/console works

Comment 7 by no...@chromium.org, Aug 3 2016

Status: Started (was: Fixed)
hm, no it does not

Comment 8 by no...@chromium.org, Aug 3 2016

it works now, but I will keep this bug open JIC
the commits are being populated right now

chromiumperffyi=> select count(*) from changes ;
 count
-------
  3787
(1 row)

chromiumperffyi=> select count(*) from changes ;
 count
-------
  3920
(1 row)

Comment 9 by no...@chromium.org, Aug 3 2016

now I see duplicate revisions in the changes table. Looks like population did not work well.
I've deleted rows from tables scheduler_changes, change_properties, sourcestamp_changes and changes and gitiles poller refilling the tables back right now starting from ae59b40c3055d9f5ef53fc0237bd236e4761833e which is ~5000 commits back
Status: Fixed (was: Started)
number of rows in the changes table reached 5058, oldest being f67cb8db57391f982744179442638bfbd5595320 which was on July 08

Sign in to add a comment