New issue
Advanced search Search tips

Issue 748707 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Get rid of GAE RPCs from gae_ts_mon upon instance initialization

Project Member Reported by mar...@chromium.org, Jul 25 2017

Issue description

/restricted/ereporter2/request/59763c0300ff0913a6d06a7b170001737e636173742d69736f6c61746500016261636b656e643a323831392d61663733353437000100
on the internal isolate server

Traceback (most recent call last):
  File "appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "main_backend.py", line 38, in <module>
    app = create_application()
  File "main_backend.py", line 34, in create_application
    gae_ts_mon.initialize(backend, is_enabled_fn=is_enabled_callback)
  File "gae_ts_mon/config.py", line 201, in initialize
    monitors.AppengineCredentials()))
  File "gae_ts_mon/common/monitors.py", line 136, in __init__
    credentials = credential_factory.create(self._SCOPES)
  File "gae_ts_mon/common/monitors.py", line 91, in create
    http = self.base.create([self.IAM_SCOPE]).authorize(http)
  File "gae_ts_mon/common/monitors.py", line 63, in create
    app_identity.get_service_account_name())
  File "appengine/api/app_identity/app_identity.py", line 408, in get_service_account_name
    rpc.wait()
  File "appengine/api/apiproxy_stub_map.py", line 556, in wait
    assert self.__rpc.state == apiproxy_rpc.RPC.FINISHING, repr(self.state)
DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded.


Ironically the RPC is for a logging.info() call so it is easy to remove.
https://github.com/luci/luci-py/blob/e59e10036a6e05e668ef0dcd621f3bd9833eb585/client/third_party/infra_libs/ts_mon/common/monitors.py#L62

Original source:
https://chromium.googlesource.com/infra/infra/+/master/packages/infra_libs/infra_libs/ts_mon/common/monitors.py

 
Project Member

Comment 1 by bugdroid1@chromium.org, Jul 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2d9fed25d5a4c420df48491696fa180db217d8b8

commit 2d9fed25d5a4c420df48491696fa180db217d8b8
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Jul 25 22:45:54 2017

ts_mon: Remove unnecessary RPC call on instance startup

Doing AppEngine RPC as part of instance startup is known to increase the odds of
failure during instance initialization.

R=vadimsh@chromium.org
BUG= 748707 

Change-Id: I6e2cc896cb1a5783227a01c41a03273c1c359f6f
Reviewed-on: https://chromium-review.googlesource.com/585673
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/2d9fed25d5a4c420df48491696fa180db217d8b8/packages/infra_libs/infra_libs/ts_mon/common/monitors.py

Project Member

Comment 2 by bugdroid1@chromium.org, Jul 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/3737c64693c62ba160b0ab17af54f74b8dc65a0e

commit 3737c64693c62ba160b0ab17af54f74b8dc65a0e
Author: maruel <maruel@chromium.org>
Date: Wed Jul 26 21:56:29 2017

Update infra_libs to 1.1.15 / 0b44aba87c1c6538439df6d24a409870810747ab

Hopefully (?) this should help reduce the amount of failures during instance
initialization.

R=vadimsh@chromium.org
BUG= 748707 

Review-Url: https://codereview.chromium.org/2991803002

[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/appengine/swarming/server/bot_archive.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/appengine/third_party/gae_event_mon/README.swarming
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/appengine/third_party/gae_ts_mon/README.md
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/appengine/third_party/gae_ts_mon/README.swarming
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/appengine/third_party/gae_ts_mon/__init__.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/appengine/third_party/gae_ts_mon/config.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/README.swarming
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/event_mon/config.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/event_mon/protos/chrome_infra_log_pb2.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/event_mon/protos/goma_stats_pb2.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/event_mon/router.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/httplib2_utils.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/__init__.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/distribution.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/errors.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/helpers.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/http_metrics.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/interface.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/metric_store.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/metrics.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/monitors.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/common/targets.py
[modify] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/config.py
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/README.md
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/acquisition_network_device.proto
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/acquisition_network_device_pb2.py
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/acquisition_task.proto
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/acquisition_task_pb2.py
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/any.proto
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/any_pb2.py
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/REAME.md
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/__init__.py
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/acquisition_network_device.proto
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/acquisition_network_device_pb2.py
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/acquisition_task.proto
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/acquisition_task_pb2.py
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/metrics.proto
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/current/metrics_pb2.py
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/metrics.proto
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/metrics_pb2.py
[delete] https://crrev.com/37d8dd14e7186c39c1ba22c5cd7d7ac66c0b04a2/client/third_party/infra_libs/ts_mon/protos/new/__init__.py
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/timestamp.proto
[rename] https://crrev.com/3737c64693c62ba160b0ab17af54f74b8dc65a0e/client/third_party/infra_libs/ts_mon/protos/timestamp_pb2.py

Owner: mar...@chromium.org
Status: Fixed (was: Available)
This particular issue is fixed, will file another bug for follow ups but they are hard to get a good stack trace. :/

Sign in to add a comment