Sshd user enumeration vulnerability caused by specific PAM return value

[glapnte]

Hello everyone! Ever since we started using PAM for 2fa, we noticed that sshd’s behavior was not the same for existing and non existing users, we didn’t have the time to troubleshoot so they put an intern on the case (me). When trying to login via a password we have (existing user):

ssh user@host
(user@host) Password:
(user@host) Password:
(user@host) Password:
user@host: Permission denied (publickey,keyboard-interactive,hostbased).

Now for non existing users we have:

ssh user@host
(user@host) Password: 
Connection closed by XXX.XXX.XXX.XX port 22

After a lot of debugging, we found out that a specific RHEL patch (preserve-pam-errors.patch) causes the issue when a PAM module returns the value PAM_USER_UNKNOWN (affects RHEL’s versions of openssh 8.0p1 and up) . It’s seems like RHEL is aware of this since it’s getting patched in centos-stream sshd doesn't propose to enter password again when a non-existing user is specified (!87) · Merge requests · Red Hat / centos-stream / rpms / openssh · GitLab

Here are some steps to reproduce the issue on Rocky8/9

1. Setup the sshd_config

Make sure you are using keyboard-interective:pam for password authentification, here are our auth methods for example (password +2fa):

AuthenticationMethods hostbased publickey,keyboard-interactive:pam keyboard-interactive:pam,keyboard-interactive:pam
PasswordAuthentication no
ChallengeResponseAuthentication yes
UsePAM yes

2. Put a PAM module that returns PAM_USER_UNKNOWN when a user is invalid, you could use pam_debug but you can achieve the same by putting pam_access.so as a required module in /etc/pam.d/password-auth.

password-auth stack (make sure your are using it in /etc/pam.d/sshd):

auth        required      pam_env.so
auth        required    pam_access.so
auth        sufficient    pam_unix.so try_first_pass nullok 
auth        sufficient    pam_sss.so use_first_pass
auth        required      pam_deny.so

3. systemctl restart sshd
4. try to do a password login with a valid and invalid user.

A quick fix would be to do something like this:

auth        required      pam_env.so
auth        [success=ok new_authtok_reqd=ok ignore=ignore user_unknown=ignore default=bad]     pam_access.so
auth        sufficient    pam_unix.so try_first_pass nullok 
auth        sufficient    pam_sss.so use_first_pass
auth        required      pam_deny.so

My main question is how would you handle this ? Should an issue be opened with Rocky/RHEL ?

Thanks.

[nazunalika]

Hello, welcome to the forums!

This seems like an interesting case, so I had to take some time to look into this. This would be an issue to report to Red Hat at their issues tracker. However…

@glapnte

It’s seems like RHEL is aware of this since it’s getting patched in centos-stream sshd doesn’t propose to enter password again when a non-existing user is specified (!87) · Merge requests · Red Hat / centos-stream / rpms / openssh · GitLab

While I do see that this was fixed in CentOS Stream 10 (in August), I don’t see any commits of it being patched in Stream 9.

My thought is that if they haven’t released a patch for 8 or 9 for this, they may not see it as a big enough of an issue or even a big enough vulnerability. Reason I say this is because of custom pam configurations that a user may do, which are much different from standard pam configurations that RHEL provides by default and via authselect, can cause (or I guess in this case expose) behavior like this.

Using a standard pam configuration provided by authselect, I don’t get the behavior you’re describing, whether it’s a local or remote (FreeIPA in this case) user.

[nazu@router ~]$ grep -E '^auth|^password' /etc/pam.d/password-auth
auth        required                                     pam_env.so
auth        required                                     pam_faildelay.so delay=2000000
auth        required                                     pam_faillock.so preauth silent
auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
auth        [default=1 ignore=ignore success=ok]         pam_localuser.so
auth        sufficient                                   pam_unix.so
auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
auth        sufficient                                   pam_sss.so forward_pass
auth        required                                     pam_faillock.so authfail
auth        required                                     pam_deny.so
account     required                                     pam_faillock.so
account     required                                     pam_unix.so
account     sufficient                                   pam_localuser.so
account     sufficient                                   pam_usertype.so issystem
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required                                     pam_permit.so
password    requisite                                    pam_pwquality.so local_users_only
password    sufficient                                   pam_unix.so sha512 shadow  use_authtok
password    [success=1 default=ignore]                   pam_localuser.so
password    sufficient                                   pam_sss.so use_authtok
password    required                                     pam_deny.so

## User does NOT exist here
[nazu@router ~]$ ssh asdf@localhost
(asdf@localhost) Password:
(asdf@localhost) Password:
(asdf@localhost) Password:
asdf@localhost's password:
Permission denied, please try again.
asdf@localhost's password:
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535

## User exists here, this user is local.
[nazu@router ~]$ ssh rpmbuild@localhost
(rpmbuild@localhost) Password:
(rpmbuild@localhost) Password:
(rpmbuild@localhost) Password:
rpmbuild@localhost's password:
Permission denied, please try again.
rpmbuild@localhost's password:
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535

## User exists, this user is remote.
[nazu@router ~]$ ssh test_ipa_user@localhost
(test_ipa_user@localhost) Password:
(test_ipa_user@localhost) Password:
(test_ipa_user@localhost) Password:
test_ipa_user@localhost's password:
Permission denied, please try again.
test_ipa_user@localhost's password:
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535

It seems that under specific custom configuration conditions, it exposes the problem you’re describing. But the standard configurations provided do not. This is probably why it hasn’t been patched in 8 or 9 yet, but this is just a guess.

My opinion is that this is something to bring up to Red Hat. If you happen to have RHEL systems in your environment and the company you work for has a support contract with Red Hat, I would open a support case.