Compare https://build.chromium.org/p/chromium.perf/builders/Win%20x64%20Builder and https://luci-milo.appspot.com/buildbot/chromium.perf/Win%20x64%20Builder/ Anything obvious causing the currently building builds to not show up?
Let's not implicitly assign ownership. I also note that even among complete builds it's lagging. Latest complete build is #26577, latest in Milo is #26586. The Pub/Sub subscription does not have a backlog. Searching through logs: https://pantheon.corp.google.com/logs/viewer?project=luci-milo&key1=default&minLogLevel=0&expandAll=false&resource=appengine.googleapis.com&logName=projects%2Fluci-milo%2Flogs%2Fappengine.googleapis.com%252Frequest_log&advancedFilter=metadata.serviceName%3D%22appengine.googleapis.com%22%0Alog%3D%22appengine.googleapis.com%2Frequest_log%22%0AprotoPayload.resource%3D%22%2Fpubsub%2Fbuildbot%22%0A%22Checking%20for%20build%20chromium.perf%2FWin%20x64%20Builder%22&lastVisibleTimestampNanos=1476150933727961000 The last message pushed from the master is at 6:55 PM (it is 7:03). My guess: the Twisted process is at 100% CPU and is overloaded, and is therefore not pushing out updates. Possible solution that come to mind: 1) Rewrite Pub/Sub push to work around this sort of problem. This might not work b/c BuildBot is the core problem here. 2) Remove BuildBot logging from chromium.perf in favor of LogDog. Master load will probably drop to 1% :)
CPU for this master is definitely capped out (149% on "top").
Also no exceptions in the master's log.
I'll take this, I found a few bugs in the pipeline.
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/fb0560def63fcd666d8ef9ece802d88622d200fc commit fb0560def63fcd666d8ef9ece802d88622d200fc Author: hinoka <hinoka@google.com> Date: Tue Oct 11 21:43:12 2016 pubsub: pending fix 1. Rename misspelled "pendingBuildStatues" to "pendingBuildStates" 2. Yieldind a deferred list returns a list of tupes with success, we only want to return the data. BUG= 654623 Review-Url: https://codereview.chromium.org/2406133002 [modify] https://crrev.com/fb0560def63fcd666d8ef9ece802d88622d200fc/scripts/master/pubsub_json_status_push.py
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/ab30d82eb01708c79fec5db8833523af53d8169e commit ab30d82eb01708c79fec5db8833523af53d8169e Author: hinoka <hinoka@google.com> Date: Wed Oct 12 02:18:48 2016 Milo: Display pending builds This fixes some bugs (naming mismatches) with the receiving end of pubsub, and also adds pending build data into a builder view. The rename was done here: https://codereview.chromium.org/2406133002/ BUG= 654623 Review-Url: https://codereview.chromium.org/2409953002 [modify] https://crrev.com/ab30d82eb01708c79fec5db8833523af53d8169e/milo/appengine/buildbot/builder.go [modify] https://crrev.com/ab30d82eb01708c79fec5db8833523af53d8169e/milo/appengine/buildbot/structs.go
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager.git/+/f5224160c37436254be092b669d93f766eb8129c commit f5224160c37436254be092b669d93f766eb8129c Author: hinoka <hinoka@google.com> Date: Wed Oct 12 05:39:07 2016
Reusing this bug for a similar issue...
Pending builds show up (woooo) Only for recently restarted masters though https://luci-milo.appspot.com/buildbot/chromium.fyi/CrWinClang%28dbg%29/
Ping! Any progress on current builds? Pending builds now show up, but not current builds.
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/a94e9e33c00d7ecc2acd4a8489732f026c768a35 commit a94e9e33c00d7ecc2acd4a8489732f026c768a35 Author: hinoka <hinoka@google.com> Date: Wed Oct 26 21:39:47 2016 Milo: Clean up builds that are no longer current. Sometimes when masters restart, they don't flush builds, so that current builds continue to linger in Milo despite being expired (and finished). This uses the signal that if a build exists in Milo but isn't in the master JSON, then it should get marked as expired. BUG= 654623 Review-Url: https://codereview.chromium.org/2406163003 [modify] https://crrev.com/a94e9e33c00d7ecc2acd4a8489732f026c768a35/milo/appengine/buildbot/pubsub.go [modify] https://crrev.com/a94e9e33c00d7ecc2acd4a8489732f026c768a35/milo/appengine/buildbot/pubsub_test.go
Almost, these are the CLs left: Bug fix with expiring builds: https://codereview.chromium.org/2446273005/ Add ?limit=... https://codereview.chromium.org/2328733002/ Actually add it to the view: https://codereview.chromium.org/2321243002
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/b0a9fe11208861902c7767fbb3eb7adc98bff727 commit b0a9fe11208861902c7767fbb3eb7adc98bff727 Author: hinoka <hinoka@google.com> Date: Thu Oct 27 23:38:16 2016 Milo: Pubsub pending build fixes. This fixes the large pending build (and master json size) issue by stripping out all files, comments, and property information from pending builds (they aren't used or displayed anywhere anyways.) This shrunk chromium.fyi's compressed json size from 2MB down to 60KB This also cleans up some debug statements. BUG= 654623 Review-Url: https://codereview.chromium.org/2450343003 [modify] https://crrev.com/b0a9fe11208861902c7767fbb3eb7adc98bff727/milo/appengine/buildbot/pubsub.go
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/d7f554b78efddab387c85fd80e89410011a35683 commit d7f554b78efddab387c85fd80e89410011a35683 Author: hinoka <hinoka@google.com> Date: Fri Oct 28 20:04:49 2016 Milo: Add ?limit= support to builders So you can see more than 25 builds BUG= 654623 Review-Url: https://codereview.chromium.org/2328733002 [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/api/proto/pb.discovery.go [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/buildbot/builder.go [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/buildbot/html.go [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/buildbucket/html.go [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/frontend/expectations/buildbot-buildbot.TestableBuilder-Basic_Test_no_builds.html [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/frontend/expectations/buildbot-buildbot.TestableBuilder-Basic_Test_with_builds.html [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/frontend/templates/buildbot/pages/builder.html [modify] https://crrev.com/d7f554b78efddab387c85fd80e89410011a35683/milo/appengine/settings/settings.go
It does now: https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/
Comment 1 by d...@chromium.org
, Oct 11 2016Owner: ----