You can do SSH keys with Yubikey, I’ve set that up before now. And from the administration side, always need to remember removing SSH keys when the person has left or whatever and get them to return the Yubikey as well.
Really? A user may share their key but will not share their password, write it down etc. Good luck with that. Hope they do not respond to various phishing attacks by entering their passwords? Here is an example of my password as required (not actual) e57-lE=D-w36_Rd Care to tell me the likely hood of finding that written down. Keys are far better then passwords. Management of keys can present a problem but so can passwords. If it is that much of a problem, then other source of keys like Yubi might be a good option. Education is critical in a network. Effectiveness is based on a lot of things including users, program, training etc. Any mishandling can cause a breach be it a password or key. I might get your password, password repository, your CA signing key, authentication key etc.
Anything can be breached. That is the nature of connected networks. There are many vulnerabilities in all networks, including yours that are known to adversaries but not the good guys. Who is exploiting them and why they are is part of the equation. How is persistence being achieved? How is it being exploited? What is being exfiltrated and how? That is where using tools like IDS/IPS to help understand what is going on is important. The OP was having 3000 hits a day. That is rough but very LOW. I would be much more concerned if I was getting 3000 hits a day on port xxxx where my ssh connections were. Then I KNOW it is targeted and react accordingly. And yes I have been the guy monitoring the hits on our network. It is the nature of the business.
If you read my earlier post you would also see MFA stated as a further hardening and also a best security practice, along with keys - if you really want even Yubi keys. Huge problems exist with MFA at the user level. Several recent attacks succeeded when the attackers contacted the user directly and got them to divulge their TOPT. How is that user education program going for you?
I never said a user might not share their password. They do, and unbelievably will even tell us they have FFS. The issue is that SSH key authentication is, in practice, WORSE than a standard password for ordinary users in terms of security.
SSH key authentication suffers from several problems. The first and biggest issue is that you have absolutely no control over whether the key is protected by a passphrase, or any way to find out. This means that if the key is ever compromised, then your security is for jack, and you and even the user to whom the key belongs will have absolutely no idea if that is the case.
The second issue related to the first is that you have to get the key from machine to machine, and now you have to use a USB drive or lots of other stupid methods, which wildly increase the chances of compromise.
At the end of the day, an SSH key is nothing more than a long random password. We had to cut off SSH key access to our HPC system because it was being actively used to compromise systems, as users were not securing them with a password, which means we are outsourcing our security.
Back in 2020, I ended up getting up at ~02:00 because I couldn’t sleep, and I cut off SSH key-based authentication to our system in the middle of a sector-wide ongoing attack. When I reviewed the logs later that day, I realised that had I waited until the morning, we would have been compromised. SSH keys for ordinary users are for people who don’t understand the risks. Put it this way a hacker in China cannot see a password on a post it node under my keyboard. They can, however, steal an unprotected SSH key and use it from China.
By the way, 3,000 hits per day on your SSH ports are inconsequential. That is just what you get for being connected to the internet. We see many times that number. Having fail2ban installed is, however, critical in managing the onslaught. I did try moving SSH to a random port a few years back on a test server. It was found within a day and was receiving thousands of probes from multiple IP addresses across multiple address ranges. The evidence is therefore that moving it buys you nothing other than inconvenience.
Note my day job is running an HPC system at a university. We have 100+ users on the system via SSH on a typical day. They are scientists, not system administrators, and they are all over the world. We are currently blocking a number of countries, most notably Russia, after persistent and targeted attacks in early 2022. Played a game of whack-a-mole for a while and then gave up and blocked all Russian IP address ranges, and the attack stopped.
Crowdsec is a good alternative to fail2ban, just thought I’d put that out there.
I don’t think that SSH public key authentication is good for security, either. The way it works, you are encouraged to create a single keypair for all purposes. It is possible to create multiple keypairs with different passphrases for different purposes but using multiple keys is inconvenient and I think few people do this in practice. It means that capturing one passphrase and one private key is enough to gain access to all systems where the person has access. It’s even worse when someone creates a key without a passphrase, for example to automate some tasks, and later reuses that key for everything else.
It is strange that every single best practice recommendation mentions using Public key authentication and to disable root as the primary tools for security. Almost all recommend Port change also for reasons described above. Other practices are often more specific to configuration. These practices were developed with input from agencies like the NSA to best secure networks, you know, the guys who love weakness in other networks. These keys are immune to brute force attacks. Passwords are not. It is mentioned above that keys are just long passwords. They are NOT. They are basically PKI encryption. Think end of universe for breaking them with current technology. Passwords are OFTEN reused without control across multiple networks. They can be cracked, they can be intercepted, they can be given away, etc. It is very common to hack an account based on a intrusion from a different network. Keys are immune to this. You only share your public key. Your private key is yours. You do not care about other networks giving up your password information. The issues the two of you have mentioned with keys is the same for passwords as they are very commonly re-used and as a result can be compromised through other means. Networks I have been on often have very complex requirements to try to prevent brute force but this will push someone to reuse. One I worked on, had a requirement for 2 of each character type (upper, lower, numerical, special), no more than 3 of any repeating types, minimum of 15 characters. They are almost always written down. MFA attached to a key will very much remove the issues with passphrase. Passwords do not have any type of passphrase, ever anyway. I can even watch you to learn your password, and have. I use at least 2 physical key devices for work, multiple keys everywhere else. Management is the, pun intended, key to this. It is very possible and even simple to remove the use of authorized_keys from the user. There are other management processes to control users outside of education. There is absolutely nothing that can be done with a key that cannot be done with a password. There are many benefits to keys that do not exist with passwords. This is why passwords are going to be a thing of the past.
A question for those thinking passwords are better, have you set PubkeyAuthentication no? if not, you already allow them, which is better. AuthenticationMethods publickey and PasswordAuthentication no are happily set to this my networks.
I’ve been personally interested in security, hacking, cracking etc. for over 20 years and I have limited trust in established security recommendations. In the end you are responsible for the security of your systems, not the people who write these recommendations. I strongly disagree with everyone who presents SSH public key authentication as a universal solution for improving SSH security. In some cases it can improve security, in some cases it can worsen security. Using a single keypair for all account@host combinations where you have access is surely a bad idea, probably worse than using password authentication, unless you really use a single (even if strong) password everywhere. If you use at least 2 different passwords when connecting with password authentication, and you want to migrate to public key authentication without worsening security, you need to create at least 2 different keypairs protected by different passphrases. SSH makes it a pain to use multiple keypairs, so most people use just one. Often it’s not even protected with a passphrase, so any malware can just steal it without the need to capture the passphrase.
No, I have never set “PubkeyAuthentication no”, but there are no other people accessing my servers, so there is no need for me to disable public key authentication. In some cases I have to use this form of authentication for automated tasks and for git over SSH (especially with submodules). @jabuzzard has already said that he has disabled public key authentication and it makes sense on servers accessed by many people.
Being 30+ years in with 7 as ISSM all on DoD networks…
Wait, never mind, I see.
That’s fine, I have no interest in continuing this conversation with you either.
Just to recap this entire issue - @hartings machine had unauthorized ssh access, which is believed to have been from a non-privileged user. But then the root password got changed.
If root wasn’t able to ssh in directly as per the previous discussion, there must have been some kind of security vulnerability that wasn’t patched in time? Possibly PHP? That’s the crux of my question - how did the unauthorized access then get into root?
This shouldn’t be surprising at all given the large number of local privilege escalation vulnerabilities discovered in the Linux kernel during the past 4 months, starting from Copy Fail and followed by many others.
Got it - basically it’s imperative to patch/reboot to new kernel as soon as one is released.
It is far broader then just an LPE vulnerability on a kernel. This is where one needs to look at a defense in depth approach and better yet zero trust philosophy with it.
First question is how did the attack occur and did it really happen. This was the point of the OP’s first post.
Is it possible the password was accidentally changed - I have seen all added admin accounts wiped off of an AD system. We determined that an error in configuring keycloak allowed the admin to delete the accounts when they thought they were just un-configuring keycloak. Did someone think they were changing a user password when they changed the root password? Root password change typo? Was it changed via automation… Etc.
Assuming the change was malicious, how did it happen.
Are my external facing devices up to date. Routers/Firewalls/F5 etc. There have been a number of significant vulnerabilities detected in this type of equipment recently.
What ingresses exist in the network? HTTP, HTTPS, SMTP, VPN, SSH, RDP, VNC, DNS etc. Any opening in the firewall in an ingress regardless of local firewall settings. Nginx has recently had significant vulnerabilities reported. SSH has recently had a significant one, etc. Are the entry points minimized to ensure traceability? Is the software being used up to date and up to the task?
What is my password policy. Is it complex enough to make guessing difficult? Does it change often enough? Am I monitoring for brute force attacks against them. The OP saw 3000 hits to a non standard ssh port per day. Is this an brute force attack? How is my training to reduce phishing attacks and social engineering? In a significant number of attacks, the adversary was given the password by various means including password reset. Once they have the password, lateral movement and escalation of priv may be trivial. The Help Desk is often the area with the most turn-around of employees, the most jr employees and yet is one of the most critical in preventing attacks!
Once established on my local system with a non-priv user, what can they do, even not exploiting an LPE. Do they have a path to use sudo to switch root? If the local user has full sudo rights and the password has been compromised, sudo to root is trivial. Is NOPASSWD used in sudo? Can another application be exploited, Firefox, docker, etc?
There have been 13 kernel updates in July/August with many LPE’s. Assuming (zero trust not withstanding), the users are exploiting LPE attacks, they must be on the system to be successful to begin with. What allowed them in? If they are in, your data IS being compromised regardless of root. Have they established persistence? What external to the system logging do I have as local logs are suspect?
Patching at the 3 day level is not easily sustainable in most environments, especially in 24/7/365 operations. Keeping an adversary off of the network is critical. Once in, making their lateral movement difficult is imperative and detection essential. LPEs are a big risk, no doubt, but once in, you already to not control your network!