Dl.rockylinux.org HTTP/2 stream errors, HTTP 504 errors

It’s currently very difficult to run basic dnf commands due to dl.rockylinux.org being unstable. At the moment it seems to affect Rocky 10 only.

11:17:05  ==> googlecompute.sc-rocky-baseimage: [MIRROR] python3-libs-3.12.11-3.el10.x86_64.rpm: Curl error (92): Stream error in the HTTP/2 framing layer for https://dl.rockylinux.org/pub/rocky/10/BaseOS/x86_64/os/Packages/p/python3-libs-3.12.11-3.el10.x86_64.rpm [HTTP/2 stream 251 was not closed cleanly: PROTOCOL_ERROR (err 1)]

11:17:05  ==> googlecompute.sc-rocky-baseimage: [FAILED] python3-libs-3.12.11-3.el10.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
11:15:50  ==> tencentcloud-cvm.sc-rocky-baseimage: [MIRROR] perl-overload-1.37-512.2.el10_0.noarch.rpm: Status code: 504 for https://dl.rockylinux.org/pub/rocky/10/AppStream/x86_64/os/Packages/p/perl-overload-1.37-512.2.el10_0.noarch.rpm (IP: 146.75.122.132)

11:15:51  ==> tencentcloud-cvm.sc-rocky-baseimage: [MIRROR] perl-subs-1.04-512.2.el10_0.noarch.rpm: Status code: 504 for https://dl.rockylinux.org/pub/rocky/10/AppStream/x86_64/os/Packages/p/perl-subs-1.04-512.2.el10_0.noarch.rpm (IP: 146.75.122.132)

11:15:51  ==> tencentcloud-cvm.sc-rocky-baseimage: [FAILED] perl-subs-1.04-512.2.el10_0.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

Is there a proper channel where repo issues can be reported? We’ve already resorted to disabling mirrorlist for all repositories because of regularly recurring issues where mirrors aren’t properly in sync, leading to package installation failures unless you use –skip-broken or –nobest and other hacks.

10.1 was just released. The dl.rockylinux.org mirror (project managed) is backed by the fastly CDN. On my side, I’m not receiving 504’s or issues updating or installing systems from our mirror, nor from any other connected mirrors my systems happen to pick up.

In fact, I tried to download one of the packages you listed and it pulls fine from a normal wget.

[label@sani tmp]$ wget https://dl.rockylinux.org/pub/rocky/10/AppStream/x86_64/os/Packages/p/perl-subs-1.04-512.2.el10_0.noarch.rpm
Saving 'perl-subs-1.04-512.2.el10_0.noarch.rpm'
HTTP response 200  [https://dl.rockylinux.org/pub/rocky/10/AppStream/x86_64/os/Packages/p/perl-subs-1.04-512.2.el10_0.noarch.rpm]
perl-subs-1.04-512.2 100% [=============================================================================================>]   11.44K    --.-KB/s
                          [Files: 1  Bytes: 11.44K [18.98KB/s] Redirects: 0  Todo: 0  Errors: 0                          ]

What would be helpful is to know where you’re located. There is no guarantee that the fastly CDN has updated its endpoints right away globally. However, we have many mirrors that have synced up.

This is the current BaseOS 10.1 propagation.

This is current AppStream 10.1 propagation.

Here is one of my 10.0 systems updating.

The repo list.

The dnf update prompt.

I’m not receiving errors.

We do have mirrors who consistently fall out of sync. However, we have given mirrors more than a week to sync 10.1 content, allowing us to push final updates upon release (see the propagation chart to see this calculated as mirrors are verified).

This is unnecessary and you shouldn’t have to do this.

My suggestion is to wait or pick a mirror that is geographically close to you that may be in sync. Note that as we manage dl.rockylinux.org, it is always up to date and it is contingent on the fastly CDN POP’s to update their cache.

1 Like

We build images in an automated fashion across multiple cloud providers and regions, so it’s hard for us to hardcode a particular set of mirrors. Lately the issue occurs both from us-east-1 on Google Cloud Platform and from eu-frankfurt on Tencent Cloud. I also successfully manage to download the files in question using the URLs mentioned in the error message, so it’s hard to reproduce for sure.

Since you mentioned Fastly, I have also seen issues with downloading DPDK from the official source:

09:12:29  
==&gt; udp-router-builder.amazon-ebs.udp-router-builder: + wget --timeout=30 --no-verbose --tries=3 <a href="https://fast.dpdk.org/rel/dpdk-25.07.tar.xz">https://fast.dpdk.org/rel/dpdk-25.07.tar.xz</a> -O dpdk-25.07.tar.xz

09:12:29  
==&gt; udp-router-builder.amazon-ebs.udp-router-builder: Unable to establish SSL connection.

The common denominator is Fastly (assuming that’s where `fast` in `fast.dpdk.org` comes from).

Not really sure what else to try, I guess I can re-enable mirrors for our Rocky 10 builds and hope they remain more in sync than Rocky 9 mirrors have traditionally been.

Any other suggestions?

Tried re-enabling use of mirrors and immediately got reminded why we disabled them in the first place (they’re slow):

12:44:58  ==> tencentcloud-cvm.sc-rocky-baseimage: (19/176): hdparm-9.65-6.el10.x86_64.rpm          13 kB/s |  96 kB     00:07

12:45:12  ==> tencentcloud-cvm.sc-rocky-baseimage: (20/176): libpcap-1.10.4-7.el10.x86_64.rpm       12 kB/s | 174 kB     00:13

12:45:16  ==> tencentcloud-cvm.sc-rocky-baseimage: (21/176): libpciaccess-0.16-16.el10.x86_64.rpm  6.9 kB/s |  27 kB     00:03

12:45:29  ==> tencentcloud-cvm.sc-rocky-baseimage: (22/176): libibverbs-57.0-2.el10.x86_64.rpm      13 kB/s | 450 kB     00:33

12:45:35  ==> tencentcloud-cvm.sc-rocky-baseimage: (23/176): libpq-16.8-2.el10_0.x86_64.rpm         14 kB/s | 255 kB     00:18

12:45:36  ==> tencentcloud-cvm.sc-rocky-baseimage: (24/176): mailcap-2.1.54-8.el10.noarch.rpm       26 kB/s |  34 kB     00:01

12:45:45  ==> tencentcloud-cvm.sc-rocky-baseimage: (25/176): lsof-4.98.0-7.el10.x86_64.rpm          14 kB/s | 227 kB     00:15

12:46:07  ==> tencentcloud-cvm.sc-rocky-baseimage: (26/176): net-tools-2.0-0.73.20160912git.el10.x  14 kB/s | 304 kB     00:21

12:46:14  ==> tencentcloud-cvm.sc-rocky-baseimage: (27/176): ocl-icd-2.3.2-8.el10.x86_64.rpm       9.4 kB/s |  66 kB     00:06

12:46:17  ==> tencentcloud-cvm.sc-rocky-baseimage: (28/176): protobuf-c-1.5.0-6.el10.x86_64.rpm     11 kB/s |  32 kB     00:02

12:46:19  ==> tencentcloud-cvm.sc-rocky-baseimage: (29/176): nano-8.1-3.el10.x86_64.rpm             17 kB/s | 735 kB     00:42

I guess we can disable for just one cloud but that feels hacky too.

Re-enabled the `fastestmirror` plugin, for some reason, on an instance running in `eu-frankfurt` it decided to pick a random mirror in Uzbekistan.

12:54:37  ==> tencentcloud-cvm.sc-rocky-baseimage: [MIRROR] net-tools-2.0-0.73.20160912git.el10.x86_64.rpm: Status code: 404 for https://mirror.dc.uz/rocky/10.1/BaseOS/x86_64/os/Packages/n/net-tools-2.0-0.73.20160912git.el10.x86_64.rpm (IP: 91.212.89.107)

I very much doubt the pipes between Uzbekistan and the main data center hub in Europe is better/faster than literally all other mirrors in the whole world, including closer once somewhere in Europe.

More errors:

12:58:02  ==> tencentcloud-cvm.sc-rocky-baseimage: [MIRROR] kernel-core-6.12.0-124.8.1.el10_1.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for https://mirror.dc.uz/rocky/10.1/BaseOS/x86_64/os/Packages/k/kernel-core-6.12.0-124.8.1.el10_1.x86_64.rpm [OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0]
12:57:44  ==> tencentcloud-cvm.sc-rocky-baseimage: [MIRROR] nspr-4.36.0-4.el10_0.x86_64.rpm: Status code: 404 for https://mirror.dc.uz/rocky/10.1/AppStream/x86_64/os/Packages/n/nspr-4.36.0-4.el10_0.x86_64.rpm (IP: 91.212.89.107)

On Google this time:

13:58:46  ==> sc-rocky-bastion.googlecompute.ruby3: [MIRROR] perl-Pod-Html-1.25-481.1.el9_6.noarch.rpm: Status code: 504 for https://dl.rockylinux.org/pub/rocky/9/AppStream/x86_64/os/Packages/p/perl-Pod-Html-1.25-481.1.el9_6.noarch.rpm (IP: 199.232.194.132)
13:58:46  ==> sc-rocky-bastion.googlecompute.ruby3: [FAILED] perl-Pod-Html-1.25-481.1.el9_6.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

Meanwhile on another cloud in another part of the world:
13:59:04 ==> sc-rocky-bastion.tencentcloud-cvm.ruby2: [MIRROR] gcc-toolset-14-gcc-14.2.1-9.el9_6.x86_64.rpm: Curl error (92): Stream error in the HTTP/2 framing layer for ``https://dl.rockylinux.org/pub/rocky/9/AppStream/x86_64/os/Packages/g/gcc-toolset-14-gcc-14.2.1-9.el9_6.x86_64.rpm`` [HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)]
13:59:04 ==> sc-rocky-bastion.tencentcloud-cvm.ruby2: [FAILED] gcc-toolset-14-gcc-14.2.1-9.el9_6.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

we’ve also encountered similar issues on Rocky 9 from a datacenter in Israel

dnf.librepo.log

2025-12-02T02:13:47+0000 INFO Downloading: https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=extras-9
2025-12-02T02:13:47+0000 INFO Downloading: https://dl.rockylinux.org/pub/rocky/9.7/extras/x86_64/os/repodata/repomd.xml
2025-12-02T02:13:49+0000 INFO Error during transfer: Status code: 504 for https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/g/grub2-tools-extra-2.06-114.el9_7.0.1.x86_64.rpm (IP: 199.232.194.132)
2025-12-02T02:13:49+0000 INFO Downloading: https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/g/grub2-tools-extra-2.06-114.el9_7.0.1.x86_64.rpm
2025-12-02T02:13:49+0000 INFO Error during transfer: Status code: 504 for https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/k/kernel-core-5.14.0-611.9.1.el9_7.x86_64.rpm (IP: 199.232.194.132)
2025-12-02T02:13:49+0000 INFO Downloading: https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/k/kernel-core-5.14.0-611.9.1.el9_7.x86_64.rpm
2025-12-02T02:13:54+0000 INFO Error during transfer: Curl error (18): Transferred a partial file for https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/k/kernel-modules-5.14.0-611.9.1.el9_7.x86_64.rpm [transfer closed with 38288629 bytes remaining to read]
2025-12-02T02:13:54+0000 INFO Downloading: https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/k/kernel-modules-5.14.0-611.9.1.el9_7.x86_64.rpm
2025-12-02T02:13:59+0000 INFO Error during transfer: Status code: 504 for https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/Packages/g/grub2-tools-extra-2.06-114.el9_7.0.1.x86_64.rpm (IP: 199.232.194.132)
2025-12-02T02:13:59+0000 INFO Error while downloading: Cannot download Packages/g/grub2-tools-extra-2.06-114.el9_7.0.1.x86_64.rpm: All mirrors were tried

dnf.log

2025-12-02T02:13:59+0000 DDEBUG Plugins were unloaded.
2025-12-02T02:13:59+0000 SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/cli/cli.py", line 229, in do_transaction
    self.download_packages(install_pkgs, self.output.progress, total_cb)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 1311, in download_packages
    self._download_remote_payloads(payloads, drpm, progress, callback_total)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 1240, in _download_remote_payloads
    raise dnf.exceptions.DownloadError(errors._irrecoverable())
dnf.exceptions.DownloadError: grub2-tools-extra-1:2.06-114.el9_7.0.1.x86_64: Cannot download, all mirrors were already tried without success

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 176, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.9/site-packages/dnf/cli/cli.py", line 236, in do_transaction
    raise dnf.exceptions.Error(errstr)
dnf.exceptions.Error: Error downloading packages:
  grub2-tools-extra-1:2.06-114.el9_7.0.1.x86_64: Cannot download, all mirrors were already tried without success
2025-12-02T02:13:59+0000 CRITICAL Error: Error downloading packages:
  grub2-tools-extra-1:2.06-114.el9_7.0.1.x86_64: Cannot download, all mirrors were already tried without success

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.