I’m thinking this sync issue may still be present.
In my homelab, on my Rocky9 machines, I have a nightly cron job that mails me when patches are available. In theory, for kernel updates, I should be getting consistent results across all the machines. But only a subset are reporting. And on those when I come to an upgrade I either get 404 errors or (after a dnf clean all) the repo server doesn’t think the .22 kernel is available.
Of my 6 machines, 1 is fully patched, 2 have the .22 headers but not any of the other files (so there’s a mismatch between headers and kernel), and the remaining 3 don’t have any .22 files and only have .21.
I know I could remove the mirrorlist and just use baseurl, but I figure that if I’m seeing this issue across multiple installs then others might be as well.
@sweh Thanks for bringing this up. I observed the same issue. I manage 3 rocky9 machines. When I’ve noticed out-of-sync differences from dnf among the 3 in the past, I’ve generally been able to fix it by running a ‘dnf clean all’ or waiting a day before doing an update.
But there appears to be something seriously broken with mirror synchronization right now.
One machine saw the .22 kernel on June 19. I waited, wanting to keep all 3 on the same kernel, but the next day, it must have chosen a different mirror, because it was no longer available. The next several days were bizarre: now-you-see-me-now-you-don’t on all three machines, and never consistently the same across all three.
On Monday the 23rd I got irritated enough to browse through the mirror database to find one that was geographically nearby and had the .22 kernel (including the kernel-devel package that I needed from the appstream repo). It took me several rounds of trial and error and reading the dnf manpage to figure out how to temporarily use an alternate mirror without changing any of the existing config files.
I’m offering it here in case it helps others, or if experts want to critique it.
I hope it is obvious that the above URLs are fake – you’ll have to find mirror URLs that work for you. FWIW I’m located in the US/Eastern time zone, so there are lots of mirrors for me to choose from that are geographically nearby.
This should be fixed now. The problem was about 50% of the mirrors weren’t synced for some reason, but this was resolved yesterday by the infra team and the mirrors now seem to have synced.
There might still be some that haven’t caught up yet, but generally it should now be OK.
It’s Thursday evening 06/26 and I’m hitting this on all my new builds today, it is not fixed.
[MIRROR] kernel-modules-core-5.14.0-570.22.1.el9_6.x86_64.rpm: Curl error (18): Transferred a partial file for http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/k/kernel-modules-core-5.14.0-570.22.1.el9_6.x86_64.rpm [transfer closed with 30977213 bytes remaining to read]
[MIRROR] findutils-4.8.0-7.el9.x86_64.rpm: Status code: 504 for http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/f/findutils-4.8.0-7.el9.x86_64.rpm (IP: 199.232.198.132)
[MIRROR] libnfnetlink-1.0.1-23.el9_5.x86_64.rpm: Status code: 504 for http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/l/libnfnetlink-1.0.1-23.el9_5.x86_64.rpm (IP: 199.232.198.132)
[MIRROR] kernel-modules-core-5.14.0-570.22.1.el9_6.x86_64.rpm: Curl error (18): Transferred a partial file for http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/k/kernel-modules-core-5.14.0-570.22.1.el9_6.x86_64.rpm [transfer closed with 22588605 bytes remaining to read]
[FAILED] kernel-modules-core-5.14.0-570.22.1.el9_6.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
Error: Error downloading packages:
kernel-modules-core-5.14.0-570.22.1.el9_6.x86_64: Cannot download, all mirrors were already tried without success
Your errors are unrelated to the mirrors syncing. You are connecting to our mirror (dl.rockylinux.org) and you’re receiving 504’s or partial downloads. However, that is not the behavior I am seeing in my own systems.
No, but it may have been a temporary issue this evening, though we had the same issue throughout the last week off and on. I did notice this evening that mirrors.rockylinux.org was throwing a 504, after I had given up and went looking to create a mirror of our own. I noticed that came online an hour or so later and updates were pulled down.
Why wouldn’t it try any other mirror besides dl.rockylinux.org? These are stock boxes. I assumed from the “No more mirrors to try - All mirrors were already tried without success” error that it had tried other mirrors, but is that not the case?
On another machine that I tried to patch at the same time I got lots of 404 errors, before it successfully downloaded 23 rpms. Just trying to get systemd-udev-252-51.el9_6.1.x86_64.rpm hit 13 404s
eg
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/rocky-linux/9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 162.255.226.75)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-west1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 34.49.195.49)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for http://ash.mirrors.clouvider.net/rocky/9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 45.250.25.130)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://pit.mirror.jurassicinnovations.com/rocky/9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 193.30.34.70)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-west4.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 35.186.229.59)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for http://nyc.mirrors.clouvider.net/rocky/9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 94.154.159.241)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-south1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 34.36.6.112)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-east4.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 34.49.194.122)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://mirror.web-ster.com/rocky/9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 65.182.224.39)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-east5.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 34.144.209.88)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-central1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 34.49.215.167)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for http://mirror.cs.vt.edu/pub/rocky/9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 128.173.237.17)
[MIRROR] systemd-udev-252-51.el9_6.1.x86_64.rpm: Status code: 404 for https://rocky-linux-us-west2.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/BaseOS/x86_64/os/Packages/s/systemd-udev-252-51.el9_6.1.x86_64.rpm (IP: 34.36.50.162)
(3/23): systemd-udev-252-51.el9_6.1.x86_64.rpm 881 kB/s | 1.9 MB 00:02
