chromium.webkit and tryserver.blink Milo data is stale. |
||||
Issue descriptionFrom: https://bugs.chromium.org/p/chromium/issues/detail?id=699553 It's obscure, but it looks like the Milo Pub/Sub push failed to initialize when the master was restarted. I don't see any error messages other than: 2017-03-07 18:38:42-0800 [-] Unhandled error in Deferred: 2017-03-07 18:38:42-0800 [-] Unhandled Error Traceback (most recent call last): Failure: exceptions.AssertionError: However, the last push happened before the waterfall was restarted. After the restart, no "Last send session" messages after the restart, but lots of other Pub/Sub events (build registration, etc.) I'm not sure *what* went wrong. I'm going to schedule a restart and see if it successfully starts up this time. +hinoka@ for any ideas.
,
Mar 8 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/6b951904c02baf5c9ffedd75ca45d0e6bdff2a55 commit 6b951904c02baf5c9ffedd75ca45d0e6bdff2a55 Author: Dan Jacques <dnj@google.com> Date: Wed Mar 08 18:48:13 2017
,
Mar 8 2017
That looks very similar to crbug.com/699366
,
Mar 8 2017
699366
,
Mar 8 2017
https://bugs.chromium.org/p/chromium/issues/detail?id=699366 3rd try in getting it linkified...
,
Mar 8 2017
Actually this may be an unrelated issue. tryserver.blink is also exhibiting this behavior. The commonality is that both masters have been restarted after this landed: Make the buildbot master report metrics for all ThreadPools https://chromium-review.googlesource.com/c/451158/ Which looks like it touches something in the critical path for the pubsub sender. Just speculation at this point
,
Mar 8 2017
,
Mar 8 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/f9802c46927d35c001feecf031193b3bff1c22a0 commit f9802c46927d35c001feecf031193b3bff1c22a0 Author: Ryan Tseng <hinoka@google.com> Date: Wed Mar 08 21:04:23 2017
,
Mar 9 2017
https://chromium-review.googlesource.com/c/451158/ was reverted shortly after landing, so probably not related.
,
Mar 9 2017
The restart seems to have fixed the stale data here. The leading theory is that it was related to the CL in #9, but the stack trace doesn't give us much to work with :(
,
Mar 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/7776995235449cac2c6350867560a1c0a232a689 commit 7776995235449cac2c6350867560a1c0a232a689 Author: Ryan Tseng <hinoka@google.com> Date: Fri Mar 10 01:06:45 2017 PubSub: Stop the buildbot master if we can't init a client As we move towards using the pubsub stream for all data, we need to make sure the startup sequence works, and doesn't just fail silently. This treats the failure to obtain a PubSub client as a critical failure in the master (if the client is configured to run in master_site_config). BUG= 699625 Change-Id: Iaf3e7c13d66ccad74c710fd605b40f9dcb89857f Reviewed-on: https://chromium-review.googlesource.com/451797 Commit-Queue: Ryan Tseng <hinoka@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/7776995235449cac2c6350867560a1c0a232a689/scripts/master/pubsub_json_status_push.py
,
Mar 10 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/a67a90726af8f27b09f9c031276e8d100fb0732e commit a67a90726af8f27b09f9c031276e8d100fb0732e Author: Ryan Tseng <hinoka@google.com> Date: Fri Mar 10 01:10:10 2017
,
Mar 10 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/a67a90726af8f27b09f9c031276e8d100fb0732e commit a67a90726af8f27b09f9c031276e8d100fb0732e Author: Ryan Tseng <hinoka@google.com> Date: Fri Mar 10 01:10:10 2017
,
Mar 10 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/a67a90726af8f27b09f9c031276e8d100fb0732e commit a67a90726af8f27b09f9c031276e8d100fb0732e Author: Ryan Tseng <hinoka@google.com> Date: Fri Mar 10 01:10:10 2017
,
Apr 14 2017
Issue 699553 has been merged into this issue. |
||||
►
Sign in to add a comment |
||||
Comment 1 by bugdroid1@chromium.org
, Mar 8 2017