I think the right solution is to revert "upstart - disable OOM killer for all jobs" https://crrev.com/c/5701 .
The default oom_score_adj shall be 0 instead of -1000. Renderer's oom_score_adj is set to 300 and will be killed earlier. If oom-killer is invoked and all processes' oom_score_adj are -1000, kernel panic is triggered.
Any suggestion?

We should try to adjust chrome oom default score to zero first as system services invoked by upstart should be protected w/ -1000 to make the system stable?

In the current Chrome OS, not only system services, every process sets oom_score_adj to -1000 by default. Not only chrome could have memory leak or abnormal memory usage issue, e.g. the recent arc_camera3_service memory leak issue [1]. Any process with oom_score_adj=-1000 and has memory leak could potentially exhaust the system memory and trigger kernel panic.

The upstream upstart set default oom_score_adj to 0 [2]. And as the original bug [3] suggest, the main purpose of "upstart - disable OOM killer for all jobs" [4] is to investigate OS memory compression. I think setting default oom_score_adj to -1000 is for experiment, only a stable system, only a small set of processes should set oom_score_adj to -1000.

I propose setting default oom_score_adj to 0 and some important services set oom_score_adj to -1000 by themselves.

E.g. in gLinux, most system services set oom_score_adj to 0.
Only 4 processes set oom_score_adj to -1000: auditd, sshd, dmeventd, systemd-udevd.
These 4 processes write -1000 to /proc/self/oom_score_adj [5].

[1]:https://listnr.corp.google.com/report/85419030956
[2]:https://bazaar.launchpad.net/~upstart-devel/upstart/trunk/view/head:/init/job_class.h#L116
[3]: https://crbug.com/199548 
[4]:https://crrev.com/c/5701
[5]:https://cs.corp.google.com/piper///depot/google3/third_party/systemd/src/udev/udevd.c?rcl=46931103&l=1171

The process to change the oom_score_adj:
1. send an RFC email to collect important system services that should set negtive oom_score_adj
2. Changing the oom_score_adj of these system services
3. Set the default oom_score_adj to 0 by reverting https://crrev.com/c/5701

A script to list oom_score_adj of all processes:
https://user.git.corp.google.com/vovoy/utils/+/master/dut_utils/oom_score.py

I analyzed 10000 kernel panic reports caused by "Out of memory and no killable processes". The attachment list the process consumes most memory and is not killable when kernel panic is triggered.

Deleted: largest_process.txt 983 KB

largest_process.txt 983 KB View Download

The crash reports is available on https://crash.corp.google.com . E.g. Report ID 7f0687f8e4a5f36e is available on https://crash.corp.google.com/browse?q=ReportID%3D%277f0687f8e4a5f36e%27

I don't understand, it says that taskscheduler sometimes consumes 34gb ?
or chrome at 17gb?

Updates the crash reports summary.
memory_hogs.txt: list the process consumes most memory.
memory_hogs_details.txt: List a line of summary for each crash reports.

Deleted: memory_hogs.txt 1.6 KB

memory_hogs.txt 1.6 KB View Download

Deleted: memory_hogs_details.txt 980 KB

memory_hogs_details.txt 980 KB View Download

There is bug on the parsing script. It's fixed.

Also note that the program size is ram + swap used, so the program size can by greater than total ram.

e.g.:
Report ID       , anon(KB), swapfree(KB), total(GB),     version,      board,    largest_proc, size(KB)
d5f356f589bfa245,  6857388,            0,         8,  10323.67.0,        eve,             vlc, 18023808

vlc used 6.8 GB ram and 11 GB swap.

there's a lot of dev-mode related processes in there (vlc, kodi, mono, matlab, games, etc...).  can you rerun your summary but filter out devices w/dev-mode enabled ?

however, related to that, it doesn't seem like we run crosh with oom adjusted, so i suspect everything people run from there are getting -1000.  we should fix that ... probably want to do it in the process_proxy code ?

Here is an example crash report:
https://crash.corp.google.com/browse?q=ReportID%3D%2769ef425c282bf51b%27

I could not tell from the crash report if it's in dev mode, but the majority of the process list shall be from dev mode.

IMO, if we can fix the panic issue in dev mode without affecting the normal mode user, it's worth doing.

I am writing a doc to explain my plan to adjust oom_score_adj.
https://docs.google.com/document/d/1NIul6tcKDfiC5J37q8_7hw1MrTz6mCqdvlUOzkwjKuc/edit

CrOS's crash uploader will set image_type=dev when in dev mode, but i don't know how crash/ exposes that

thanks for the link to the doc.

The dev mode can be tell from Product data -> boot_mode = dev.

e.g.
https://crash.corp.google.com/browse?q=ReportID%3D%27d5f356f589bfa245%27#2

I will modify the script to analyze crash reports w/o dev mode.

The result without dev mode.

count of reports that the largest process used > 500 MB: 5555
count of distinguishable largest processes: 19

Largest process name, count in crash reports, dev mode excluded:

chrome:
chrome          5289
TaskSchedulerSi   49 chrome thread, showing child thread because main thread (chrome) doesn't exist.
renderer_crash_    1 chrome thread
TaskSchedulerFo    1 chrome thread

upstart services:
shill             77
udevd              4
bluetoothd         3
permission_brok    2
dbus-daemon        2
cras               1
update_engine      1

crosh:
memtester         23 from crosh command memory_test

android:
com.rovio.tnt      1 angry birds evolution, android app with wrong oom_score_adj
Chrome_ProcessL    1 should be a thread of an android app, android app with wrong oom_score_adj
.katana:browser    1 part of facebook android app, android app with wrong oom_score_adj

other:
x.client:glview   88 unknown
futility           9 firmware utility
quipper            1 part of debugd
gs                 1 related to gstoraster and cupsd

Deleted: memory_hogs_details_nodev.txt 1.0 MB

memory_hogs_details_nodev.txt 1.0 MB View Download

Chrome on Chrome OS is killable with fixes in  https://crbug.com/850457