Hi all,
I have installed Rocky Linux 9 as a VM on KVM with a web proxy inbetween.
Firefox, wget and curl use the proxy successfully for http and https - no authentication required nor configured.
I have configured dnf in /etc/dnf/dnf.conf to use the same proxy, also without authentication.
Now dnf runs into a timeout questioning mirrors.rockylinux.org - and curl as well as wget also.
Any other https-destination works well with curl and wget, but rockylinux.org does not (it is no dns resolution problem) every request runs into a timeout.
Any hint or help is greatly appreciated.
Dirk
I am sadly prevented from posting curl -v examples (positive and negative) here by a rule saying I must not post more than 2 llinks, so I have to explain the details.
In a positive case, curl connects to the destination and downloads the requested file.
In case of dl.rockylinux.org e.g. curl connects to the nginx there, but the requested file is never delivered. The communication stops at
< Connection: keep-alive
and hangs.
Dirk
Does dl.rockylinux.org try to deliver the requested file(s) on a separate communication channel?
If you put it in a code block, it doesn’t get picked up as links. (Code, logs, commands, terminal output, etc, should go in there anyway. (readability))
https://www.example.com/
https://www.example.com/
Can’t help you with the proxy issue, but I hope you get your problem solved. 
This is a positive example:
[hostmeister@mgmt01-pvirtclus01-01 ~]$ curl -v https://www.geniodata.de
* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1,172.16.99.30'
* Uses proxy env variable https_proxy == 'http://172.16.99.30:3128/'
* Trying 172.16.99.30:3128...
* Connected to 172.16.99.30 (172.16.99.30) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.geniodata.de:443
> CONNECT www.geniodata.de:443 HTTP/1.1
> Host: www.geniodata.de:443
> User-Agent: curl/7.76.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
- snip -
* subject: CN=*.geniodata.de
* start date: Mar 1 00:00:00 2023 GMT
* expire date: Mar 31 23:59:59 2024 GMT
* subjectAltName: host "www.geniodata.de" matched cert's "*.geniodata.de"
* issuer: C=DE; ST=Baden-W�rttemberg; L=Durmersheim; O=EUNETIC GmbH; CN=EuropeanSSL Server CA 2
* SSL certificate verify ok.
- snip -
> GET / HTTP/1.1
> Host: www.geniodata.de
> User-Agent: curl/7.76.1
> Accept: */*
>
- snip -
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 13 Apr 2023 07:40:31 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< Last-Modified: Mon, 01 Feb 2021 15:54:58 GMT
< ETag: "7072-5ba4860393ec4"
< Accept-Ranges: bytes
< Content-Length: 28786
< Content-Type: text/html
<
- snip -
<!DOCTYPE html>
<!-- saved from url=(0021)https://geniodata.de/ -->
<html lang="de-DE" class="js"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
And this is the failing rockylinux.org connection:
[root@mgmt01-pvirtclus01-01 ~]# curl -v http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/repodata/repomd.xml
* Uses proxy env variable no_proxy == '172.16.99.30'
* Uses proxy env variable http_proxy == 'http://172.16.99.30:3128'
* Trying 172.16.99.30:3128...
* Connected to 172.16.99.30 (172.16.99.30) port 3128 (#0)
> GET http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/repodata/repomd.xml HTTP/1.1
> Host: dl.rockylinux.org
> User-Agent: curl/7.76.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 4166
< Content-Type: text/xml
< Server: nginx
< Last-Modified: Wed, 12 Apr 2023 02:20:22 GMT
< ETag: "64361566-1046"
< Accept-Ranges: bytes
< Date: Thu, 13 Apr 2023 07:48:17 GMT
< X-Served-By: cache-chi-kigq8000065-CHI, cache-hhn-etou8220067-HHN
< X-Cache: MISS, MISS
< X-Cache-Hits: 0, 0
< X-Timer: S1681372097.843817,VS0,VE490
< X-Cache: MISS from opnsense-pvirtclus01-01.chtzn.sccon.de
< X-Cache-Lookup: MISS from opnsense-pvirtclus01-01.chtzn.sccon.de:3128
< Via: 1.1 varnish, 1.1 varnish, 1.1 opnsense-pvirtclus01-01.chtzn.sccon.de (squid/5.8)
< Connection: keep-alive
<
That is where it hangs and never finishes.
Are you able to connect successfully from the proxy itself to dl.rockylinux.org?
No. The proxy is internal behind NAT - it cannot reach dl.rockylinux.org also, but everything else.
But - a connect happens successfully (see above), the files are just not delivered.
I’ve just installed Squid on a separate machine, and from my Rocky 9 it’s working fine, and can connect to the dl.rockylinux.org URL’s that you’ve been trying, eg:
curl -v http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/repodata/repomd.xml
It would help to know how your proxy is configured, with what, and what exactly is the configuration. In my example, a basic install of Squid and it’s default config with localnet enabled works perfectly fine. I cannot replicate this issue. Either that, or your IP is perhaps on some kind of restricted list.
Would be good to see if you can connect directly to that URL when bypassing the proxy. If so, then that would confirm that your proxy config is the problem. If it’s still unable to, then that could confirm your IP address being the problem.
Since all other websites I tried can be accessed via the proxy, I doubt it is the proxy config (it is squid on opnsense in a pretty basic configuration).
My best guess is that - since the opnsense firewall is a vm on an internal bridge using NAT to get outside - the repo servers of rockylinux.org try to open a backchannel for delivering files. That of course fails because of NAT.
I do not see any other explanation for the phenomenon that the http(s) connect to rockylinux is successfull but then file deliverage runs into a timeout.
Mine is behind a nat and works fine. But normal debugging and diagnosis should involve testing to see if it works with and without the proxy. It all depends if you are willing or not to do it, or to share the config of the proxy. Otherwise it’s impossible to help you.
That said you should be using the mirrorlist options in the repo files anyway, but that also works for me behind a proxy. I even downloaded and installed packages right now as well via the proxy. If it was as you said being behind a nat, then it would fail for me to.
If it’s not squid, then opnsense could be the problem. But since I can use Rocky mirrors and urls via a proxy, it’s not a problem on the Rocky side.
1 Like
Thanks a lot, that excludes at least one possible cause.
I can download from rockylinux fine from the server the opnsense-vm is running on - so generally it works.
And, by the way, the mirrorlist option in the repo file leads to the same timeout - I just posted on of the many tests I made.
I will post the squid.conf later
I have a KVM server, that I ran opnsense on once as a VM, and it was double-natted. Being that the LAN and WAN interfaces were internal IP ranges, and behind another router/firewall. On the LAN side of the opnsense I had a Rocky installation and the WAN side was connected to a switch which was connected to the firewall/router. This also worked for me for internet access, although there wasn’t any proxy. I don’t have the VM to be able to install squid and try it. Either way, even opnsense should work also.
Even in that configuration my Rocky VM would download via the opnsense and via the router/firewall without issue, no problems connecting to anything to install/update packages. It looked more or less like this:
Rocky VM → LAN opnsense → WAN opnsense → Firewall/Router → Internet
SO IP wise:
Rocky: 192.168.10.10
LAN opnsense: 192.168.10.1
WAN opnsense: 192.168.2.10
LAN firewall/router: 192.168.2.1
192.168.10.10 → 192.168.10.1 → 192.168.2.10 → 192.168.2.1 → Internet
This is why I believe this is a local config issue be it squid or opnsense, since otherwise it should just work.
This is the opnsense’s squid.conf content:
tls_outgoing_options options=NO_TLSv1 cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
acl bump_step1 at_step SslBump1
acl bump_step2 at_step SslBump2
acl bump_step3 at_step SslBump3
acl bump_nobumpsites ssl::server_name "/usr/local/etc/squid/nobumpsites.acl"
ssl_bump peek bump_step1 all
ssl_bump splice all
ssl_bump peek bump_step2 all
ssl_bump splice bump_step3 all
ssl_bump bump
ssl_bump peek bump_step1 all
ssl_bump peek bump_step2 bump_nobumpsites
ssl_bump splice bump_step3 bump_nobumpsites
ssl_bump stare bump_step2
ssl_bump bump bump_step3
sslproxy_cert_error deny all
acl ftp proto FTP
http_access allow ftp
acl PURGE method PURGE
http_access allow localhost PURGE
http_access deny PURGE
acl CONNECT method CONNECT
icap_enable on
adaptation_send_client_ip on
adaptation_send_client_ip off
adaptation_send_username on
adaptation_send_username off
icap_client_username_encode on
icap_client_username_encode off
icap_preview_enable on
icap_preview_enable off
icap_enable off
include /usr/local/etc/squid/pre-auth/*.conf
include /usr/local/etc/squid/post-auth/*.conf
cache deny all
cache_mem 0
cache_replacement_policy heap LFUDA
coredump_dir /var/squid/cache
refresh_pattern pkg\.tar\.zst$ 0 20% 4320 refresh-ims
refresh_pattern d?rpm$ 0 20% 4320 refresh-ims
refresh_pattern deb$ 0 20% 4320 refresh-ims
refresh_pattern udeb$ 0 20% 4320 refresh-ims
refresh_pattern Packages\.bz2$ 0 20% 4320 refresh-ims
refresh_pattern Sources\.bz2$ 0 20% 4320 refresh-ims
refresh_pattern Release\.gpg$ 0 20% 4320 refresh-ims
refresh_pattern Release$ 0 20% 4320 refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 129600 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 129600 reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 129600 reload-into-ims
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
dns_v4_first on
pinger_enable off
access_log none
cache_store_log none
cache_store_log stdio:/var/log/squid/store.log
via off
httpd_suppress_version_string on
delay_pools 1
delay_class 1 3
delay_access 1 allow all
delay_pools 1
delay_class 1 1
delay_access 1 allow all
logfile_rotate 0
error_directory /usr/local/etc/squid/errors/local
there is a lot dynamic logic also which I omitted - it would have made everything unreadable.
Just to round that up: I did not find a solution. I just assume this is somehow related to the dnf client having to pass an mtu restricted vlan (that is the only explanation I find).
My workaround is directly using mirrors - all mirrors I tried directly work fine for me, just rockylinux.org does not, so I do not see much benefit in further sorting this out.
Thanks for all people trying to help.
1 Like