I have migrated an AWS EC2 host from Amazon Linux 2 to Rocky Linux 9 and applied CIS Benchmark hardening (SOX requirement) but am seeing drastic performance differences after the upgrade related to auditd and it’s plugins. Most noticeable is that the audisp-af_unix plugin starts consuming large amounts of memory over time, similar to a memory leak and it got to a point where the auditd service had to be restarted to prevent an OOM.
These are the installed packages
$ rpm -qa | grep audi
audit-libs-3.1.5-4.el9.x86_64
python3-audit-3.1.5-4.el9.x86_64
audit-3.1.5-4.el9.x86_64
rpm-plugin-audit-4.16.1.3-37.el9.x86_64
audispd-plugins-3.1.5-4.el9.x86_64
and the kernel running on the instance
$ uname -r
5.14.0-570.26.1.el9_6.x86_64
And the configuration for audisp-af_unix plugin
$ sudo cat /etc/audit/plugins.d/af_unix.conf
# This file controls the configuration of the
# af_unix socket plugin. It simply takes events
# and writes them to a unix domain socket. This
# plugin can take 2 arguments, the path for the
# socket and the socket permissions in octal.
active = yes
direction = out
path = /sbin/audisp-af_unix
type = always
args = 0600 /var/run/audispd_events
format = string
And the audit.rules post hardening (note these are the same rules for the most part and used in the same order as on Amazon Linux 2 - which is based off Centos 7 - and it ran on it without any performance impact, but it was auditd 2.x, Rocky has 3.1.5)
## This file is automatically generated from /etc/audit/rules.d
-D
-b 8192
-f 1
-w /etc/selinux -p wa -k MAC-policy
-w /usr/share/selinux -p wa -k MAC-policy
-a always,exit -F arch=b64 -S execve -F key=execve
-w /bin -p w
-w /boot -p w
-w /etc -p w
-w /sbin -p w
-w /usr/bin -p w
-w /usr/local/bin -p w
-w /usr/local/sbin -p w
-w /usr/sbin -p w
-w /usr/share/keyrings -p w
-w /var/spool/cron -p w
-a always,exit -F arch=b64 -S creat,open,openat,truncate,ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b32 -S creat,open,openat,truncate,ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b64 -S creat,open,openat,truncate,ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b32 -S creat,open,openat,truncate,ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b64 -S chmod,fchmod,fchmodat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S chown,fchown,lchown,fchownat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S setxattr,lsetxattr,fsetxattr,removexattr,lremovexattr,fremovexattr -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S chmod,fchmod,fchmodat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S chown,fchown,lchown,fchownat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S setxattr,lsetxattr,fsetxattr,removexattr,lremovexattr,fremovexattr -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S adjtimex,settimeofday -k time-change
-a always,exit -F arch=b32 -S adjtimex,settimeofday -k time-change
-a always,exit -F arch=b32 -S clock_settime -F a0=0x0 -k time-change
-a always,exit -F arch=b64 -S clock_settime -F a0=0x0 -k time-change
-w /etc/localtime -p wa -k time-change
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat -F auid>=1000 -F auid!=unset -k delete
-a always,exit -F arch=b32 -S unlink,unlinkat,rename,renameat -F auid>=1000 -F auid!=unset -k delete
-a always,exit -F arch=b64 -S mount -F auid>=1000 -F auid!=unset -k mounts
-a always,exit -F arch=b32 -S mount -F auid>=1000 -F auid!=unset -k mounts
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock -p wa -k logins
-w /usr/sbin/insmod -p x -k modules
-w /usr/sbin/rmmod -p x -k modules
-w /usr/sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules
-a always,exit -F arch=b64 -S sethostname,setdomainname -k system-locale
-a always,exit -F arch=b32 -S sethostname,setdomainname -k system-locale
-w /etc/issue -p wa -k system-locale
-w /etc/issue.net -p wa -k system-locale
-w /etc/hosts -p wa -k system-locale
-w /etc/hostname -p wa -k system-locale
-w /etc/sysconfig/network -p wa -k system-locale
-w /etc/sysconfig/network-scripts -p wa -k system-locale
-w /etc/NetworkManager -p wa -k system-locale
-w /var/run/utmp -p wa -k session
-w /var/log/wtmp -p wa -k session
-w /var/log/btmp -p wa -k session
-w /etc/sudoers -p wa -k scope
-w /etc/sudoers.d -p wa -k scope
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
-w /etc/nsswitch.conf -p wa -k identity
-w /etc/pam.conf -p wa -k identity
-w /etc/pam.d -p wa -k identity
-a always,exit -F arch=b64 -C euid!=uid -F auid!=unset -S execve -k user_emulation
-a always,exit -F arch=b32 -C euid!=uid -F auid!=unset -S execve -k user_emulation
-w /var/log/sudo.log -p wa -k sudo_log_file
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/pkexec -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/write -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/locate -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/screen -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/lib/polkit-1/polkit-agent-helper-1 -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/libexec/openssh/ssh-keysign -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/libexec/utempter/utempter -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/grub2-set-bootflag -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/mount.nfs -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/pam_timestamp_check -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/unix_chkpwd -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/chcon -F perm=x -F auid>=1000 -F auid!=unset -k perm_chng
-a always,exit -F path=/usr/bin/setfacl -F perm=x -F auid>=1000 -F auid!=unset -k perm_chng
-a always,exit -F path=/usr/bin/chacl -F perm=x -F auid>=1000 -F auid!=unset -k perm_chng
-a always,exit -F path=/usr/sbin/usermod -F perm=x -F auid>=1000 -F auid!=unset -k usermod
-a always,exit -F path=/usr/bin/kmod -F perm=x -F auid>=1000 -F auid!=unset -k kernel_modules
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module,create_module,query_module -F auid>=1000 -F auid!=unset -k kernel_modules
-e 2
Upon auditd startup audisp-af_unix plugin immediately goes into queue full.
Aug 08 15:32:50 host audisp-af_unix[2390538]: Bad or not enough arguments, using defaults
Aug 08 15:32:50 host audisp-af_unix[2390538]: audisp-af_unix plugin is listening for events
Aug 08 15:32:50 host augenrules[2390541]: /sbin/augenrules: Audit system is in immutable mode - exiting with no changes
Aug 08 15:32:50 host systemd[1]: Started Security Auditing Service.
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: auditd queue full reporting limit reached - ending dropped event notifications
And that is also seen here
$ sudo auditctl --signal cont && sudo cat /var/run/auditd.state
audit version = 3.1.5
current time = 08/08/25 15:33:25
process priority = -4
writing to logs = yes
current log size = 7836 KB
max log size = 8192 KB
logs detected last rotate/shift = 0
space left on partition = yes
Logging partition free space 81895 MB
space_left setting 10000 MB
admin_space_left setting 1000 MB
logging suspended = no
file system space action performed = no
admin space action performed = no
disk error detected = no
Number of active plugins = 2
current plugin queue depth = 4000
max plugin queue depth used = 4000
plugin queue size = 4000
plugin queue overflow detected = yes
plugin queueing suspended = no
listening for network connections = no
I’ve tried a lot of combinations, so far with no luck so was curious if anyone has ideas around this.
SELinux is set to Permissive mode. The host is doing a lot of disk I/O on a regular basis, so it’s busy generally when used, but not a lot of CPU and not a lot of network connections.