The issue is similar to crbug.com/765380. 
CC: wonderfly@ (the owner of the bug)

I don't have access to cros-autotest-bot project to check quota or cleanup.

Independent of this, this is concerning if we keep leaking resources and hit quota. 

@wonderfly Have we thought of increasing quota until we come up with better solution?

Just checked the builder status seems to be green for now.

https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=lakitu-gpu-paladin&buildBranch=master

> this is concerning if we keep leaking resources and hit quota. 

Most of the leaks were due to the builder machines getting aborted so there isn't much we can do - from the infrastructure layer. Hopefully when we move away from the waterfalls things will get better - there will be fewer aborts.

In the meantime, I have cleaned up stale resources and made sure all resource usages were way below quota.

As per our discussion, wonderfly can you please setup a notification to cloud-image@ or even send pager to primary/secondary oncall (not sure if that is doable), when we are near quota limits. 

Eventually we need to add piece of code while draining waterfall to clean up cros-autotest-bot.

@wonderfly   can  you try enabling "Stackdriver error reporting" notifications in pantheon for the project.
https://pantheon.corp.google.com/user-preferences/communication?_ga=2.158971770.-1537615205.1531806244

I think once notifications are enalbed, it would make sense to allow lakitu-developers to access the project.

Stackdriver error reporting was already enabled on the project, but unfortunately it's not what we need here. As per its description it sends a notification when "A logging configuration error has occured in the project", and what we want is a notification when a resource quota is near to be hit. I looked at other notification types and none of them does this. I'll explore the direction of a cron job. Shouldn't be too hard. In the mean time, I've given lakitu-dev all editor's access to the project.