Crawler tsunami on my web server

Rocky Linux Help & Support
1

Hi,

I’m hosting a couple dozen websites on a server running Rocky Linux 8. For the last two weeks or so, my monitoring server reports unusual system load and activity on this server. A peek in /var/log/httpd confirms that there is suddenly a tsunami of crawler activity. My guess would be AI-related stuff, but the thing is, I’m constantly at 99% CPU activity with php-fpm processes galore.

I experimented a bit, and here’s a solution based on Fail2ban that seems to work (sort of). First, I create the /etc/fail2ban/filter.d/apache-http-request.conf file like this:

[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*"

Then I put the following stanza in my /etc/fail2ban/jail.d/custom.conf file:

[apache-http-request]
enabled  = true
port     = http,https
filter   = apache-http-request
logpath  = /var/log/httpd/*access_log
maxretry = 10
findtime = 60
bantime  = 24h

So far I’ve blocked about 200 crawler addresses. Now I’m facing two problems.

  1. What about legitimate traffic? Can a human user navigate the blog without being kicked out after clicking a few pages? (Maybe you can give it a try?)

  2. I don’t want to kick out search engine crawlers, because I want sites like my blog to stay on the radar.

Any suggestions ?

2

I clicked around a bit on the site and didn’t get blocked, so that seems fine.

Some larger projects including GNOME have started using GitHub - TecharoHQ/anubis: Weighs the soul of incoming HTTP requests using proof-of-work to stop AI crawlers to protect their services from AI crawlers. Though you might need to set up your reverse proxy to skip it for the crawlers you want, as I don’t believe it distinguishes search engine crawlers from AI crawlers yet.

3

Hi, it seems like you are looking for a fully automated solution, but may I suggest creating some shell scripts you place in /etc/cron.hourly which email you hourly summaries of what your server is seeing, for example extract and sort the top 20 IP addresses (or alternatively summarizing by the first three octets, think “snowshoe”), based on the number of accesses so far during the current date, or based on the total bytes transferred so far during the current date, you can then use firewalld to block the associated CIDR address block if the activity is malicious or excessive. After blocking a few dozen CIDR address blocks, you just might find a lot less load on your server. And if you need help with creating those scripts, you can feed examples of the log lines into Microsoft Copilot, and ask it to create sample scripts for your, which you can then tweak.

4

I fiddled around with Fail2ban and found a solution that works perfectly and keeps the most obnoxious crawlers out, without banning search engine crawlers though.