I have been trying to find solutions for a Rocky 9 workstation that becomes completely unresponsive once all of its RAM and swap are full. I remember that on CentOS 7 the kernel would kill processes earlier and I’ve found this long explanation of the swap situation on newer kernels. This lead me to systemd-oomd.
I’ve managed to install it, add the required psi=1 parameter to the kernel’s commandline and it’s working and killing processes when the system is out of RAM and swap. So far so good.
The only problem with this is that on a graphical desktop, the user’s slice is getting killed instead of the single GUI app that is using too much memory. I’m trying to find out how to make oomd aware of single applications but unfortunately there is very little information to be found about systemd-oomd. I know that it relies on cgroups, but my GUI application is started inside a desktop session.
Is there a way to make processes start in their own cgroups below the general /user.slice/user-?.slice/session-?.scope cgroup that all programs that are launched within a desktop session are ending up in? Would that even help to make systemd-oomd not kill the whole X session but instead kill the offending app that’s using too much RAM?