We need to monitor: 1) DeferredResource's thread pool - queue lengths, etc. 2) DeferredResource's RPCs - success/failure, latency
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/a2b51ea70a3c6f13a9b8169f30a24017d93c8bec commit a2b51ea70a3c6f13a9b8169f30a24017d93c8bec Author: David Sansome <dsansome@chromium.org> Date: Wed Mar 08 02:28:40 2017 Make the buildbot master report metrics for all ThreadPools. We need to roll infra_libs in master.DEPS before submitting this. BUG= 698992 Change-Id: Ie858e3f0e3cefffe5c34c6038c8ceb627e88a1e4 Reviewed-on: https://chromium-review.googlesource.com/451158 Commit-Queue: Dave Sansome <dsansome@chromium.org> Reviewed-by: Phil Wright <philwright@chromium.org> Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/a2b51ea70a3c6f13a9b8169f30a24017d93c8bec/scripts/master/deferred_resource.py [modify] https://crrev.com/a2b51ea70a3c6f13a9b8169f30a24017d93c8bec/scripts/master/monitoring_status_receiver.py [modify] https://crrev.com/a2b51ea70a3c6f13a9b8169f30a24017d93c8bec/scripts/master/unittests/deferred_resource_test.py [modify] https://crrev.com/a2b51ea70a3c6f13a9b8169f30a24017d93c8bec/third_party/twisted_10_2/twisted/python/threadpool.py [modify] https://crrev.com/a2b51ea70a3c6f13a9b8169f30a24017d93c8bec/DEPS
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/internal.DEPS/+/3969219956a2048223b35b425b0bf2935237740a commit 3969219956a2048223b35b425b0bf2935237740a Author: David Sansome <dsansome@chromium.org> Date: Wed Mar 08 02:36:43 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/189134428d5e8a831c066b25b74079fd68e67e2f commit 189134428d5e8a831c066b25b74079fd68e67e2f Author: Dave Sansome <dsansome@chromium.org> Date: Wed Mar 08 03:08:00 2017 Revert "Make the buildbot master report metrics for all ThreadPools." This reverts commit a2b51ea70a3c6f13a9b8169f30a24017d93c8bec. Reason for revert: infra_libs isn't on slaves Original change's description: > Make the buildbot master report metrics for all ThreadPools. > > We need to roll infra_libs in master.DEPS before submitting this. > > BUG= 698992 > > Change-Id: Ie858e3f0e3cefffe5c34c6038c8ceb627e88a1e4 > Reviewed-on: https://chromium-review.googlesource.com/451158 > Commit-Queue: Dave Sansome <dsansome@chromium.org> > Reviewed-by: Phil Wright <philwright@chromium.org> > Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org> > Reviewed-by: Nodir Turakulov <nodir@chromium.org> > TBR=nodir@chromium.org,sergeyberezin@chromium.org,dsansome@chromium.org,philwright@chromium.org,chromium-reviews@chromium.org NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 698992 Change-Id: Ib22846b42a0a77c343bd31404593ba02cafd9c6b Reviewed-on: https://chromium-review.googlesource.com/451063 Reviewed-by: Dave Sansome <dsansome@chromium.org> Commit-Queue: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/189134428d5e8a831c066b25b74079fd68e67e2f/scripts/master/deferred_resource.py [modify] https://crrev.com/189134428d5e8a831c066b25b74079fd68e67e2f/scripts/master/monitoring_status_receiver.py [modify] https://crrev.com/189134428d5e8a831c066b25b74079fd68e67e2f/scripts/master/unittests/deferred_resource_test.py [modify] https://crrev.com/189134428d5e8a831c066b25b74079fd68e67e2f/third_party/twisted_10_2/twisted/python/threadpool.py [modify] https://crrev.com/189134428d5e8a831c066b25b74079fd68e67e2f/DEPS
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/7cf1933d551f00fa0feb297e5094a0aa3d6b0308 commit 7cf1933d551f00fa0feb297e5094a0aa3d6b0308 Author: David Sansome <dsansome@chromium.org> Date: Tue Mar 28 08:06:51 2017 Use dependencies from third_party in run_slave.py. In particular don't load any packages from dist-packages. This breaks importing google.protobuf, when google is a namespace package and google.protobuf is present both in third_party and as a system package. Also add infra_libs which will be required to re-land https://chromium-review.googlesource.com/c/451158/ BUG= 698992 Change-Id: Ib2e89924d26fcf4aee74edd51fdb2882e6b686f8 Reviewed-on: https://chromium-review.googlesource.com/457377 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org> Commit-Queue: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/7cf1933d551f00fa0feb297e5094a0aa3d6b0308/slave/run_slave.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/5ce52388f2415c1e1524555e2d884654cdd587c0 commit 5ce52388f2415c1e1524555e2d884654cdd587c0 Author: Dave Sansome <dsansome@chromium.org> Date: Tue Mar 28 08:38:49 2017 Revert "Use dependencies from third_party in run_slave.py." This reverts commit 7cf1933d551f00fa0feb297e5094a0aa3d6b0308. Reason for revert: Breaks Windows Original change's description: > Use dependencies from third_party in run_slave.py. > > In particular don't load any packages from dist-packages. This breaks > importing google.protobuf, when google is a namespace package and > google.protobuf is present both in third_party and as a system package. > > Also add infra_libs which will be required to re-land > https://chromium-review.googlesource.com/c/451158/ > > BUG= 698992 > > Change-Id: Ib2e89924d26fcf4aee74edd51fdb2882e6b686f8 > Reviewed-on: https://chromium-review.googlesource.com/457377 > Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> > Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org> > Commit-Queue: Dave Sansome <dsansome@chromium.org> > TBR=vadimsh@chromium.org,dsansome@chromium.org,tansell@chromium.org,sergeyberezin@chromium.org,chromium-reviews@chromium.org NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 698992 Change-Id: Ia8d383d0956e4cf158c3086fe082ef13125f78b8 Reviewed-on: https://chromium-review.googlesource.com/461599 Reviewed-by: Dave Sansome <dsansome@chromium.org> Commit-Queue: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/5ce52388f2415c1e1524555e2d884654cdd587c0/slave/run_slave.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/0183d60e6f3a5bb5a9c731ce4b0efd1884845ffd commit 0183d60e6f3a5bb5a9c731ce4b0efd1884845ffd Author: David Sansome <dsansome@chromium.org> Date: Fri Mar 31 02:26:23 2017 Use dependencies from third_party in run_slave.py. In particular don't load any packages from dist-packages. This breaks importing google.protobuf, when google is a namespace package and google.protobuf is present both in third_party and as a system package. Keep the whole original sys.path on Windows because pywin32 is in there which is needed by twisted. Also add infra_libs which will be required to re-land https://chromium-review.googlesource.com/c/451158/ BUG= 698992 Change-Id: Ie1681a855bad05e44b8c04e6b14a8dc8e7102367 Reviewed-on: https://chromium-review.googlesource.com/461626 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/0183d60e6f3a5bb5a9c731ce4b0efd1884845ffd/slave/run_slave.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/f623ec6927abc1d11ff877ac13d09958342a315f commit f623ec6927abc1d11ff877ac13d09958342a315f Author: David Sansome <dsansome@chromium.org> Date: Tue Apr 11 00:25:26 2017 Make the buildbot master report metrics for all ThreadPools. This is a re-land of https://chromium-review.googlesource.com/451158. It should work this time because https://chromium-review.googlesource.com/461626 added infra_libs to the slave's pythonpath. Bug: 698992 Change-Id: I81bae0cb43346c3550a1c8337ac9fd628eb4f687 Reviewed-on: https://chromium-review.googlesource.com/470946 Commit-Queue: Dave Sansome <dsansome@chromium.org> Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> [modify] https://crrev.com/f623ec6927abc1d11ff877ac13d09958342a315f/scripts/master/deferred_resource.py [modify] https://crrev.com/f623ec6927abc1d11ff877ac13d09958342a315f/scripts/master/monitoring_status_receiver.py [modify] https://crrev.com/f623ec6927abc1d11ff877ac13d09958342a315f/scripts/master/unittests/deferred_resource_test.py [modify] https://crrev.com/f623ec6927abc1d11ff877ac13d09958342a315f/third_party/twisted_10_2/twisted/python/threadpool.py [modify] https://crrev.com/f623ec6927abc1d11ff877ac13d09958342a315f/DEPS
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/6c28a4c5eaa9a38f9a7a32a2af0c4435286aea7c commit 6c28a4c5eaa9a38f9a7a32a2af0c4435286aea7c Author: David Sansome <dsansome@chromium.org> Date: Tue Apr 11 06:13:38 2017 Oops, set http_client_name on a few more things that use DeferredResource. Bug: 698992 TBR=chrishall Change-Id: I093e917d61bff16824ed665d4de041f240bc31f2 Reviewed-on: https://chromium-review.googlesource.com/474724 Reviewed-by: Dave Sansome <dsansome@chromium.org> Commit-Queue: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/6c28a4c5eaa9a38f9a7a32a2af0c4435286aea7c/scripts/master/pubsub_json_status_push.py [modify] https://crrev.com/6c28a4c5eaa9a38f9a7a32a2af0c4435286aea7c/scripts/master/cbe_json_status_push.py
Comment 1 by dsansome@chromium.org
, Mar 7 2017