Looking into this. Initial investigation:

* Taking the JSON and POST-ing it in a unit test works correctly, so it's not the JSON.
* We have a lot of requests to the dashboard running out of memory. Those tasks don't print the chartjson. maybe it is too big.
* Trying a new version that prints a condensed version of chartjson.

Hitting this issue on other bots, too:

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Linux%20Perf%20%283%29

Changing name to reflect reality. 

The dashboard doesn't get to the point where it prints out the condensed chartjson before OOM-ing. Rolling back to a version from before the changes I made yesterday, since this started yesterday morning.

Dave is working on seeing if he can repro the errors using the JSON from the buildbots.

It seems to be a memory leak. We've rolled back to a version of dashboard from August 30 and it works correctly, will try to narrow down the problematic CL tomorrow.

Working theory:

clean-aiolos-dccd is at instance class F4_1G; the soft memory limit is 1024 MB.

I imply that the app.yaml specified this instance class at some point in the past, and app engine memorized it even after the instance class was deleted from the app.yaml file. So it's been using F4_1G for a long time.

Uploading the bisect module on sept 14, which did specify an instance class of B1, caused app engine to update the default module's instance class to the default value, which is F1 (128 MB limit)

Issue 649446 has been merged into this issue.

Ping

Ping. Did this get fixed?

Has this been fixed? The bots are green.

Sorry about this! It was fixed.