buildbot pubsub drops some pending builds |
||||||
Issue description
This happens sometimes
$ cit log cat master_twistd_log master.tryserver.chromium.linux | grep 5971259 NOTE: All times are in local system time (-7 hour(s)).
2017-02-22 12:40:01PM: [-] Processing new build request: {'bsid': 5972882, 'buildername': u'linux_layout_tests_slimming_paint_v2', 'brid': 5971259}
2017-02-22 12:40:01PM: [-] PubSub: Successfully recorded build request 5971259/linux_layout_tests_slimming_paint_v2
2017-02-22 12:40:01PM: [-] PubSub: Status update (requestSubmitted): linux_layout_tests_slimming_paint_v2/5971259
2017-02-22 12:40:01PM: [-] added buildset 5972882 to database (build requests: {u'linux_layout_tests_slimming_paint_v2': 5971259})
2017-02-22 12:40:48PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:41:23PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:41:57PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:42:32PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:43:06PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:43:40PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:44:40PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:45:14PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:45:50PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:46:26PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:47:00PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:47:35PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:48:09PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:48:44PM: [-] PubSub: WARNING - no build request found for 5971259
2017-02-22 12:49:19PM: [-] PubSub: WARNING - no build request found for 5971259
...etc...
As a result the pubsub sender just pretends that build request doesn't exist, even tho it does exist. Funnily enough, when that build request does gets scheduled, the callback get called successfully, so it's not lost in the ether.
I suspect theres a race conditions somewhere or other issue that causes this build request to somehow fall off the radar.
,
Mar 20 2017
,
Mar 20 2017
,
May 4 2017
,
May 4 2017
Issue 718529 has been merged into this issue.
,
May 4 2017
Looks like it's really dropping a bunch.
,
May 4 2017
,
May 5 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/9259eac8194314afa0f7d4b455824cb991413d2f commit 9259eac8194314afa0f7d4b455824cb991413d2f Author: Ryan Tseng <hinoka@google.com> Date: Fri May 05 17:44:11 2017
,
May 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/d262a9497b79aa5352ea85c68f4b132872127e37 commit d262a9497b79aa5352ea85c68f4b132872127e37 Author: Ryan Tseng <hinoka@google.com> Date: Fri May 05 20:46:31 2017 Buildbot pubsub: Remove LRU for buildsets We actually already store the buildsets in the _build_requests dict (Because the native buildrequest object caches the buildset, and we persist buildrequest objects). The LRU actually doesn't work and cause buildrequests to get dropped. BUG= 695155 Change-Id: I7f0866749a5fcb7a6e90d8f03137e26eecf56c29 Reviewed-on: https://chromium-review.googlesource.com/498067 Reviewed-by: Daniel Jacques <dnj@chromium.org> Commit-Queue: Ryan Tseng <hinoka@chromium.org> [modify] https://crrev.com/d262a9497b79aa5352ea85c68f4b132872127e37/scripts/master/pubsub_json_status_push.py
,
May 5 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/e7752d1aed357fcff97f62e58e3237cfdd65e639 commit e7752d1aed357fcff97f62e58e3237cfdd65e639 Author: Ryan Tseng <hinoka@google.com> Date: Fri May 05 20:57:38 2017
,
May 5 2017
Restarted chromium.chromiumos, I think this is fixed (the error doesn't come up in the log anymore). The other masters need to be restarted still to pick this up, but they generally get restarted organically and this should get picked up eventually (particularly if there's going to be a network reset this weekend...). |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by a...@chromium.org
, Feb 27 2017