Mirror repo consistency checks implemented?

I migrated some systems to R8 and R9 and did some manual integrity checks. One of such checks is a dnf distrosync replayed hours and days later. What I noticed was that some “mirrors” do provide obsoleted rpms as latest one and forcing this way a downgrade. This has security implications. Its obvious that a mirror network never is completely in sync but I would at least expect a time window that defines a mirror in sync. Like, packages released >2 weeks ago should be on all mirrors. Currently I’m getting downgrades for tzdata and libsolv (versions from Nov. and Oct.), but the latest are from Dec 05 and 14. Also, applications like firefox and thunderbird were on such transactions. So, albeit the correct procedure should be a dnf update, I wonder if the mirrors a checked for consistency? For instance, keeping some valid/cuurent compose/repomd checksums, and using it against a mirror? The mirrormanager would then allow/lists only mirrors that are up to date …

In normal circumstances, you shouldn’t be running a dnf distrosync. I also certainly hope that you “migrated” and didn’t upgrade the systems at all as that is not supported.

Mirrors are absolutely checked for consistency and pulled out if their repodata does not match our master mirror. We have mirror manager configured to crawl our mirrors every 8 hours at a minimum. The repomd.xml is the piece that is checked and validated.

If there are mirrors that you believe are out of date and are not properly being pulled out, you may want to provide the list here or to our infrastructure mattermost channel, so we can verify that is the case.

1 Like

This raises some question marks. I migrated the systems in the scope of the major version (8->8, and 9->9). I assume that you mean with upgrade c7 → r8 or cs8 → r9. Normally clear but sometimes its good to verify the semantics :slight_smile:

That sounds reasonable. I will observe this … and report it.

Are you sure the consistency checks are really carried out?
Over here in Austria I have problems updating our Rocky Linux 8.7 systems. The mirror in the current mirror list is rockylinux.anexia.at (for country=AT => Austria).
This mirror seems stale, the last sync seems to have been on 28-Nov-2022.
dnf updates fail, because we are using kmod-wireguard from elrepo, which is in sync with current 8.7.
If I change the country to “DE” in the /etc/yum.repos.d/Rocky-*.repo everything works out fine.

I checked the mirror you’re referring to, and I see that it is up to date. I can tell this by the latest kernel that’s available here and a recently pushed package called dpdk that’s here. They have el8_7 in their names and have a january date.

Thanx for your reply!
I contacted the mirror team of Anexia directly yesterday, and they did a manual sync about 15:30 GMT.
They mentioned there were some sync problems due to newly introduced rate limiting.
Problem is fixed now.

Out of curiosity - where are this checks done?
In the mirror manager code? Just would like to take a glance at such method …

You can find the utilities here.

In particular, the scripts you’re looking for are the ‘mm2_crawler’ ones. Essentially, we crawl 1/4 of the mirrors every 2 hours. The crawls run for ~2 hours (on average). So in a day, a mirror should expect to be crawled three (ish) times. It’s not a huge deal if a single crawl fails, or is missed even (e.g. due to resource contention a job may be evicted).

Hello,

http://mirror.admax.se/rocky/9.1/AppStream/x86_64/os/Packages/k/
This mirror in Sweden is currently missing “kernel-headers-5.14.0-162.23.1”. We sync from mirrorlist = https://mirrors.rockylinux.org on systems based in Sweden and when they happen to pick this particular repository we are missing out on these updates. I take it this mirror should also be crawled regularly or taken out of the mirrorlist-pool?