403 Forbidden when downloading RPMs with YUM

Rocky Linux Tech Help
1

On RockyLinux 9.4 I am trying to update my system with Yum and I get these 403 Forbidden errors below.
I understand I am blocked ?
I just ordered this server and I worked 3 hours trying to debug things in YUM, packages, FTP, etc. This is extremly frustrating.
Someone thought it is a good idea to block IPs in such critical RPM mirror servers ?
And why aren’t there any alternatives configured ?
My IP is 188.245.110.173

systemd-252-32.el9_4.6.x86_64.rpm                                                                                                       259 kB/s | 4.0 MB     00:15
[MIRROR] cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm: Status code: 403 for https://rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.4/AppStream/x86_64/os/Packages/c/cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm (IP: 34.49.197.38)
[MIRROR] cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm: Status code: 403 for https://rocky-linux-asia-south1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.4/AppStream/x86_64/os/Packages/c/cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm (IP: 34.49.113.43)
(27/30): cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm                                                                                                 70 kB/s | 1.1 MB     00:15
[MIRROR] python-unversioned-command-3.9.18-3.el9_4.3.noarch.rpm: Status code: 403 for https://rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.4/AppStream/x86_64/os/Packages/p/python-unversioned-command-3.9.18-3.el9_4.3.noarch.rpm (IP: 34.49.197.38)
[MIRROR] python-unversioned-command-3.9.18-3.el9_4.3.noarch.rpm: Status code: 403 for https://rocky-linux-asia-south1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.4/AppStream/x86_64/os/Packages/p/python-unversioned-command-3.9.18-3.el9_4.3.noarch.rpm (IP: 34.49.113.43)
2

I’ve passed this post over to the folks who maintain the CIQ mirrors for Rocky Linux.

I cannot replicate this behavior myself, so, it could be that you are hitting some sort of WAF on their side.

Are you updating a single system, or multiple systems all sharing the same IP? That could explain the intermittent behavior.

As a workaround, you may edit the files in for your repositories in /etc/yum.repos.d/ to change from mirrorlist to baseurl and point to any mirror that you find works, including the default option: dl.rockylinux.org, which is backed by our CDN (Fastly)–and should not block requests.

1 Like
3

In addition to what @neil wrote you can also do:

dnf clean all
dnf update

as this can clear the dnf cache and get you connected to a different mirror without having o specify an exact mirror. It could be hit-and-miss and you may get connected to the same mirror, or may not. I have had similar issues, when mirrors were out-of-date, and using this process helped get around it.

1 Like
4

I’ve heard back from the CIQ team and have a few questions. They looked at the logs for the load balancers across the deployment for the past 60 days and have no requests from the IP you provided.

Can you please confirm that this is the correct IP?

Alternatively, is it possible that there is something in between your server and the internet which is intercepting these requests blocking some of the requests leading to the behavior you are seeing?

I have just tried to trigger a blockage from these mirrors on my machine and have been unable to, which points towards it being possibly a WAF or NGFW blocking the traffic to a particular domain.

–Neil

5

Another thought: is it possible you have dualstack egress? They do see some blocks occurring for some IPv6 addresses, but the behavior for those blockages is inconsistent with normal usage.

Could you confirm your IPv6 address(es), if applicable?

6

It is a virtual server from Hetzner, I checked now and when setting it up to only have IPv4, and the status in their html interface says “unasigned” for ipv6. I think any ipv6 should be disabled.

“dualstack egress”, I am not familiar with this, is it something like “ipv6 over ipv4” ? If is a software installed, I don’t have that :slight_smile:

I did a traceroute, would this help ?

traceroute rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud
traceroute to rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud (34.49.197.38), 30 hops max, 60 byte packets
 1  _gateway (172.31.1.1)  5.682 ms  5.645 ms  5.615 ms
 2  11513.your-cloud.host (195.201.64.79)  0.677 ms  0.618 ms  0.591 ms
 3  * * *
 4  spine8.cloud1.nbg1.hetzner.com (85.10.247.149)  1.836 ms spine7.cloud1.nbg1.hetzner.com (85.10.247.145)  1.911 ms spine8.cloud1.nbg1.hetzner.com (85.10.247.149)  2.320 ms
 5  * * *
 6  core12.nbg1.hetzner.com (213.239.239.141)  1.523 ms core11.nbg1.hetzner.com (213.239.203.101)  1.115 ms core12.nbg1.hetzner.com (213.239.239.141)  1.088 ms
 7  core1.fra.hetzner.com (213.239.245.250)  4.318 ms core0.fra.hetzner.com (213.239.252.21)  3.489 ms core1.fra.hetzner.com (213.239.245.254)  3.386 ms
 8  142.250.160.234 (142.250.160.234)  3.842 ms * *
 9  209.85.244.249 (209.85.244.249)  4.758 ms 142.251.65.131 (142.251.65.131)  4.196 ms 209.85.142.69 (209.85.142.69)  3.766 ms
10  142.250.210.209 (142.250.210.209)  3.720 ms 142.250.236.31 (142.250.236.31)  3.926 ms 142.250.214.195 (142.250.214.195)  3.771 ms
11  38.197.49.34.bc.googleusercontent.com (34.49.197.38)  3.674 ms  3.761 ms  3.768 ms

If I run yum update now, it says Nothing to do., maybe it took the updates from somewhere else, but still if I try that problematic location by wget, I always get the 403 forbidden:

[root@sitespeed]# wget https://rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud/pub/rocky/9.4/AppStream/x86_64/os/Packages/c/cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm
--2024-08-19 19:28:00--  https://rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud/pub/rocky/9.4/AppStream/x86_64/os/Packages/c/cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm
Resolving rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud (rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud)... 34.49.197.38, 2600:1901:0:26a7::
Connecting to rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud (rocky-linux-asia-east1.production.gcp.mirrors.ctrliq.cloud)|34.49.197.38|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-08-19 19:28:01 ERROR 403: Forbidden.

Same wget command works from other RockyLinux 6.4 severs at Hetzner.

I didn’t try the dnf clean all; dnf update; yet, because I am thinking debugging this over wget might simplify things, thou I seen some WAF that return 403 based on user agent too. Still, that url over wget works from other servers.

My IP still 188.245.110.173, I do not see any IPV6 related config.
I could mess up with mirrors or proxies to get around this, but maybe is more productive to find the cause of this at the source server / waf of these mirrors ?

7

I tried pinging 34.49.197.38, it replies in 3-4ms, which is strangely fast. Shows in the above traceroute too.

Also tried adding random string after .rpm, like .rpm?2, attempting to avoid any cache, same result.

Same problematic url changed to dl.rockylinux.org works OK:

wget https://dl.rockylinux.org/pub/rocky/9.4/AppStream/x86_64/os/Packages/c/cloud-init-23.4-7.el9_4.5.0.1.noarch.rpm
8

I made a request with an unique string adding ?issue-15519 after rpm url. Maybe someone responsible can look for that issue-15519 in the requests, in case for some reason my IP shows different.

9

I use Hetzner, they don’t block any outgoing traffic so the problem won’t be there. It could well be problems with the particular mirror you were using. The main one that you chose dl.rockylinux.org is OK to use of course, although sometimes it might actually be better to browse the Rocky Linux mirror list and choose a mirror that is closest to you when not using the automatic mirrorselect method.

10

I tried same wget from two hetzner servers, they both resolve that url to same ip 34.49.197.38, on one server it works, on one it returns 403 Forbidden, I am sure I am blocked at that end, but I thought it is usefull to fix these at the mirror’s end to avoid future problems.

I commented out the mirrors in yum repos and only left the main url to download from dl.rockylinux.org it worked OK.

Previously when it reached a 403 forbidden error, it was not clear from yum output if it continued and downloaded from another location, I found from the logs that it did. So I guess this is fixed.

11

Yeah using mirrorselect if it cannot connect to a mirror for some reason, it will usually download from another one. If you are now using baseurl, you are now only downloading from dl.rockylinux.org so means if you cannot connect to this for some reason, then it won’t try another mirror.

1 Like