Don't assign such fine-grained OOM scores |
||||
Issue descriptionThis is a bit of a fork from bug #728865 . It's also related to b/62353695 (Reduce ARC container RAM usage when not running an app). -- The idea here is this: 1. OOM scores _do not_ need to be strictly ordered. They kernel can pick between two processes with identical OOM scores and it will try to pick the process with the biggest "bang for the buck" (AKA the one it can free the most memory from). 2. The kernel OOM scores do not need to match the order that userspace would kill things if it got the chance. Said another way: the goals of the OOM killer and the goals of the userspace code that responds to "memory pressure" are different. The userspace code should have a goal of getting rid of the "least useful" thing. Ideally it should be able to get rid things fast enough. ...but if it doesn't, the kernel OOM killer kicks in as an "oh crap" killer. The kernel's OOM killer's goal is to be fast and efficient. Said more colorfully: if you're in a hot air balloon and you're sinking, you start rummaging through your supplies and find all the useless stuff and toss it overboard. ...but if you're about to crash into a tree and need to go upward fast then you don't care if something is useful. You pick the first heavy thing you can find and it goes overboard. If we have more "identical" OOM scores the kernel kill can help with picking the heavy things. --- So with all that, I'd advocate that we perhaps "undo" some of the fine grained OOM scores and go back to maybe 2-4 different "levels". Maybe: 1. Background tabs and background android stuff: kill these first 2. Foreground stuff: kill these next 3. The main chrome UI process: kill this next, which will presumably cause chrome to restart but that's better than a full CPU reboot. -- I'm not 100% sure where the "GPU process" would go, but I'd sorta think it shouldn't be OOM killed. If we OOM kill that then it will just respawn. That might/could help us if there was a GPU process leak, but if we're at low memory for some other reason then we'll just keep killing and respawning the GPU process. Probably looking for GPU process leaks belongs somewhere in userspace. ...so probably the GPU process should just be marked as OOM score -1. -- The above are just ideas. If I'm misunderstanding things, please correct me.
,
Jun 8 2017
> When tabs are discarded or OOM-killed, there is a concrete risk that we throw > away state that is useful to the user. This will remain true for a while > at best. Yup. I'm not saying it's good to throw the ice chest overboard (it's got all our food and drinks), but it's better than crashing into the tree. ...but I'm saying that we should have thought of that before the tree was in front of us. :-P --- > We try hard to guess which tabs are the least useful. Once we have made that > guess, it seems logical that we would like the OOM killer to try to follow > the same choice. I'd be OK with some amount of tendency towards killing less useful stuff, but there needs to be a _much_ bigger emphasis on how much memory is used than there is today. A) I guess one option is to try to make userspace account for "amount of memory" a little bit more, but as per above, I think "amount of memory" isn't as useful of a metric when we're not desperate. AKA as long as we're not acting out of desperation then I'm OK ignoring "memory used" and relying solely on "usefulness". B) Another option is to have userspace keep assigning an OOM score but compress the OOM scores a bit. AKA instead of assigning scores between 300 and 600, assign scores between 300 and 310, or something like that (someone would need to do the math to see exactly what the range should be). This provides information to the kernel about usefulness but doesn't fully override it's "heavy hammer" approach. --- > Maybe you're saying that if we need 100MB and the LRU tab is only > 50MB and the next one is 200MB we might as well OOM-kill the latter first, > instead of both. That may be true, but then the tab discarder should > follow the same logic as well. It's more like if we need 100 MB and we pick a background android process worth 5 MB then that wasn't too useful. If freeing 5 MB would have been enough then we (presumably) wouldn't be in the "oh crap" OOM kill situation--userspace would have taken care of things. The fact that we're in OOM kill means our normal mitigations failed. We avoided the 5 MB issue with the hack that's in bug #728865 , but that gets us into the next situation. Now imagine we've got an background Android process that's 500 MB. We'd rather kill that. AKA: memory used is the important thing for the OOM killer. --- Basically I'm arguing that we shouldn't focus on trying to pick the exactly right thing in OOM kill. We should be focusing on picking the best "big thing". Then we should keep focusing on avoiding the OOM killer wherever possible.
,
Nov 16 2017
We have work in-progress to do a better job of proactively discarding tabs, and hopefully we can find resources to apply that similarly to ARC processes. We're always going to have a risk of hitting kernel OOMs, though, so setting OOM score adjustments makes sense, but not with the fine-grained scores we're setting right now. It should be straightforward to group processes into a small set of OOM-score-adj bands, so that the kernel can pick the biggest process from the "least needed" band if/when we do hit OOM. If we use coarse-grained bands, based on each process' state (i.e. focused, active, important) then we also don't need to calculate an ordering - we can set the individual process' OOM-score-adj directly as soon as its state changes, moving it between the different buckets, for example.
,
Nov 21 2017
I agree with the basic logic that on emergency situation, we should allow a little out-of-order OOM killing so the system has a better chance to get back to normal. I'll check kernel OOM formula for setting proper values for each bucket. Since the original problem seems to be caused by a bunch of "focused tabs" with OOM score 300. Moving them back to "background tabs buckets" as soon as possible would be a vital step.
,
Mar 2 2018
,
Aug 3
This bug has an owner, thus, it's been triaged. Changing status to "assigned". |
||||
►
Sign in to add a comment |
||||
Comment 1 by semenzato@chromium.org
, Jun 8 2017