Rocky 9 audisp-af_unix queue full

Rocky Linux Help & Support
1

I have migrated an AWS EC2 host from Amazon Linux 2 to Rocky Linux 9 and applied CIS Benchmark hardening (SOX requirement) but am seeing drastic performance differences after the upgrade related to auditd and it’s plugins. Most noticeable is that the audisp-af_unix plugin starts consuming large amounts of memory over time, similar to a memory leak and it got to a point where the auditd service had to be restarted to prevent an OOM.

These are the installed packages

$ rpm -qa | grep audi
audit-libs-3.1.5-4.el9.x86_64
python3-audit-3.1.5-4.el9.x86_64
audit-3.1.5-4.el9.x86_64
rpm-plugin-audit-4.16.1.3-37.el9.x86_64
audispd-plugins-3.1.5-4.el9.x86_64

and the kernel running on the instance

$ uname -r
5.14.0-570.26.1.el9_6.x86_64

And the configuration for audisp-af_unix plugin

$ sudo cat /etc/audit/plugins.d/af_unix.conf
# This file controls the configuration of the
# af_unix socket plugin. It simply takes events
# and writes them to a unix domain socket. This
# plugin can take 2 arguments, the path for the
# socket and the socket permissions in octal.

active = yes
direction = out
path = /sbin/audisp-af_unix
type = always
args = 0600 /var/run/audispd_events
format = string

And the audit.rules post hardening (note these are the same rules for the most part and used in the same order as on Amazon Linux 2 - which is based off Centos 7 - and it ran on it without any performance impact, but it was auditd 2.x, Rocky has 3.1.5)

## This file is automatically generated from /etc/audit/rules.d
-D
-b 8192
-f 1
-w /etc/selinux -p wa -k MAC-policy
-w /usr/share/selinux -p wa -k MAC-policy
-a always,exit -F arch=b64 -S execve -F key=execve
-w /bin -p w
-w /boot -p w
-w /etc -p w
-w /sbin -p w
-w /usr/bin -p w
-w /usr/local/bin -p w
-w /usr/local/sbin -p w
-w /usr/sbin -p w
-w /usr/share/keyrings -p w
-w /var/spool/cron -p w
-a always,exit -F arch=b64 -S creat,open,openat,truncate,ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b32 -S creat,open,openat,truncate,ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b64 -S creat,open,openat,truncate,ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b32 -S creat,open,openat,truncate,ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=unset -k access
-a always,exit -F arch=b64 -S chmod,fchmod,fchmodat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S chown,fchown,lchown,fchownat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S setxattr,lsetxattr,fsetxattr,removexattr,lremovexattr,fremovexattr -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S chmod,fchmod,fchmodat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S chown,fchown,lchown,fchownat -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b32 -S setxattr,lsetxattr,fsetxattr,removexattr,lremovexattr,fremovexattr -F auid>=1000 -F auid!=unset -k perm_mod
-a always,exit -F arch=b64 -S adjtimex,settimeofday -k time-change
-a always,exit -F arch=b32 -S adjtimex,settimeofday -k time-change
-a always,exit -F arch=b32 -S clock_settime -F a0=0x0 -k time-change
-a always,exit -F arch=b64 -S clock_settime -F a0=0x0 -k time-change
-w /etc/localtime -p wa -k time-change
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat -F auid>=1000 -F auid!=unset -k delete
-a always,exit -F arch=b32 -S unlink,unlinkat,rename,renameat -F auid>=1000 -F auid!=unset -k delete
-a always,exit -F arch=b64 -S mount -F auid>=1000 -F auid!=unset -k mounts
-a always,exit -F arch=b32 -S mount -F auid>=1000 -F auid!=unset -k mounts
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock -p wa -k logins
-w /usr/sbin/insmod -p x -k modules
-w /usr/sbin/rmmod -p x -k modules
-w /usr/sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules
-a always,exit -F arch=b64 -S sethostname,setdomainname -k system-locale
-a always,exit -F arch=b32 -S sethostname,setdomainname -k system-locale
-w /etc/issue -p wa -k system-locale
-w /etc/issue.net -p wa -k system-locale
-w /etc/hosts -p wa -k system-locale
-w /etc/hostname -p wa -k system-locale
-w /etc/sysconfig/network -p wa -k system-locale
-w /etc/sysconfig/network-scripts -p wa -k system-locale
-w /etc/NetworkManager -p wa -k system-locale
-w /var/run/utmp -p wa -k session
-w /var/log/wtmp -p wa -k session
-w /var/log/btmp -p wa -k session
-w /etc/sudoers -p wa -k scope
-w /etc/sudoers.d -p wa -k scope
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
-w /etc/nsswitch.conf -p wa -k identity
-w /etc/pam.conf -p wa -k identity
-w /etc/pam.d -p wa -k identity
-a always,exit -F arch=b64 -C euid!=uid -F auid!=unset -S execve -k user_emulation
-a always,exit -F arch=b32 -C euid!=uid -F auid!=unset -S execve -k user_emulation
-w /var/log/sudo.log -p wa -k sudo_log_file
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/pkexec -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/write -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/locate -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/screen -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/lib/polkit-1/polkit-agent-helper-1 -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/libexec/openssh/ssh-keysign -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/libexec/utempter/utempter -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/grub2-set-bootflag -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/mount.nfs -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/pam_timestamp_check -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/sbin/unix_chkpwd -F perm=x -F auid>=1000 -F auid!=unset -k privileged
-a always,exit -F path=/usr/bin/chcon -F perm=x -F auid>=1000 -F auid!=unset -k perm_chng
-a always,exit -F path=/usr/bin/setfacl -F perm=x -F auid>=1000 -F auid!=unset -k perm_chng
-a always,exit -F path=/usr/bin/chacl -F perm=x -F auid>=1000 -F auid!=unset -k perm_chng
-a always,exit -F path=/usr/sbin/usermod -F perm=x -F auid>=1000 -F auid!=unset -k usermod
-a always,exit -F path=/usr/bin/kmod -F perm=x -F auid>=1000 -F auid!=unset -k kernel_modules
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module,create_module,query_module -F auid>=1000 -F auid!=unset -k kernel_modules
-e 2

Upon auditd startup audisp-af_unix plugin immediately goes into queue full.

Aug 08 15:32:50 host audisp-af_unix[2390538]: Bad or not enough arguments, using defaults
Aug 08 15:32:50 host audisp-af_unix[2390538]: audisp-af_unix plugin is listening for events
Aug 08 15:32:50 host augenrules[2390541]: /sbin/augenrules: Audit system is in immutable mode - exiting with no changes
Aug 08 15:32:50 host systemd[1]: Started Security Auditing Service.
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: queue to plugins is full - dropping event
Aug 08 15:32:53 host auditd[2390535]: auditd queue full reporting limit reached - ending dropped event notifications

And that is also seen here

$ sudo auditctl --signal cont && sudo cat /var/run/auditd.state
audit version = 3.1.5
current time = 08/08/25 15:33:25
process priority = -4
writing to logs = yes
current log size = 7836 KB
max log size = 8192 KB
logs detected last rotate/shift = 0
space left on partition = yes
Logging partition free space 81895 MB
space_left setting 10000 MB
admin_space_left setting 1000 MB
logging suspended = no
file system space action performed = no
admin space action performed = no
disk error detected = no
Number of active plugins = 2
current plugin queue depth = 4000
max plugin queue depth used = 4000
plugin queue size = 4000
plugin queue overflow detected = yes
plugin queueing suspended = no
listening for network connections = no

I’ve tried a lot of combinations, so far with no luck so was curious if anyone has ideas around this.

SELinux is set to Permissive mode. The host is doing a lot of disk I/O on a regular basis, so it’s busy generally when used, but not a lot of CPU and not a lot of network connections.

2

Firstly, I don’t think ‘audispd-plugins’ is installed by default, so are you saying you had to install it to compy with some security edict? If so, what exact function does it serve? Does it relay the events to somwhere important?

I’ve seen a lot of auditing overload in the last few years, where thousands of events are being generated that no one ever looks at. Check how many events per minute you are actually generating. In my case, production applications are not using the same volume as auditd (avoid clash of disk i/o). Group the events by type, you might be generating thousands of repeat events.

1 Like
3

You’re correct, it’s not installed by default, as part of the migration to Rocky all the Ansible automation used for AL2 was refactored for Rocky to keep same functionality, where there were differences it was adapted. In Rocky auditd version 3.x has plugins so the path is different and was updated to reflect that.

The audisp-af_unix plugin was configured on AL2 for integration with Rapid7 and the same was done for Rocky.

I am curious if Rocky Linux has auditd v4.x available in any of its repos?

4

From a quick check I’d say auditd v4.x is not available in the official Rocky 9.x repos, but even if it was, how would that solve anything?

5

I saw the Rapid7 docs mention issues with auditd version 3.1.1 SIEM (InsightIDR) - auditd Compatibility Mode for Linux Assets | Rapid7 Agent Documentation but it’s not clear if all 3.x versions are impacted or just that one, as we are on 3.1.5, and was considering if there’s audit 4.x available for Rocky 9, as what I’ve found so far is that’s only available on Rocky 10.

6

There isn’t. It would require Red Hat to integrate auditd 4.x with RHEL9 and since Rocky is 1:1 with Red Hat, then if they don’t have it, we don’t have it. The link you posted says to upgrade insight agent to 4.x, not auditd however. Therefore you need to download newer insight agent from Rapid7. If you are not using Rapid7 insight agent, then the link posted is irrelevant.

Chances of someone packing it separately are extremely low, considering how auditd is deeply rooted in the system. So if you need auditd 4.x, you’ll need to use EL10.

You said you migrated from Amazon Linux 2, how much cpu/ram did you allocate to the machine running Rocky? Min specs require at least 2GB ram. If you have less than this, then that would suggest the source of the problem - in which case, increase the specs of the VM to have at least 2GB ram.

7

Yeah, my question about audit v4.x was more due to the absence of more information from Rapid7 about whether only 3.1.1 was impacted or 3.1.1 and higher, thinking if audit v4 exists I could try to use that. We do have a newer version of the Insight Agent than 4.0.8 so that’s not a concern, their documentation isn’t super clear IMO.

I also was thinking it might not be doable to backport audit 4 onto RHEL 9 and derivatives, but it was worth double checking as well.

Hosts are not small, range is from 32 cores 250GB RAM to 96 cores 750GB of RAM.

I think the following may be the center point of the issues, trying to see if that theory holds water.

The af_unix plugin for audispd must be available and not used by other clients. Because af_unix can only take a single client, the plugin must be available solely for Compatibility Mode use.

To me this means there’s an order of operations, the audit config has to be put in place, set to compatibility mode, then auditd needs to be restarted to pick up the changes, this includes the audispd-af_unix plugin, and only after this is up and running can the Rapid7 agent be re/started to read from the plugin socket. My thought is that this order of operations isn’t happening explicitly, and the audispd-af_unix plugin is a kind of producer consumer queue where it starts producing but the agent isn’t consuming, leading to a build up of events in memory.

I’ll test this theory by configuring one host and restarting auditd and then Rapid7 agent and checking if the audispd-af_unix plugin’s RSS usage continues to grow over time or not.

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.