New issue
Advanced search Search tips

Issue 660974 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: ----



Sign in to add a comment

/_ah/push-handlers/milo causing OOM

Project Member Reported by seanmccullough@google.com, Oct 31 2016

Issue description


The pubsub listener seems to be chewing through a lot of RAM since it was enabled on staging:
https://screenshot.googleplex.com/KaeXwvNVBnD

The pubsub push handler is failing repeatedly with error messages like this:

"Exceeded soft private memory limit of 128 MB with 179 MB after servicing 1 requests total"

Eating 179MB of ram from one request sounds suspicious. A slow memory leak would probably take more than one request to cause OOM.

Need to fix this before even thinking about turning it on for prod.

If the messages actually do take up this much memory we should think about moving the pubsub handler to a separate module from the rest of the app so we can size its instances appropriately.


 
Labels: -Pri-2 Pri-1
I deleted the push subscription on the milo side. Errors should probably stop soon, but retries from the pubsub system might keep going for a few days.
Status: Fixed (was: Available)

Sign in to add a comment