Since a week most of our docker image builds are failing with a log similar to:
#25 10.64 Rocky Linux 8 - AppStream 1.9 MB/s | 20 MB 00:10
#25 10.64 Errors during downloading metadata for repository 'appstream':
#25 10.64 - Status code: 404 for https://rocky-linux-northamerica-northeast1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//8.10/AppStream/x86_64/os/repodata/fd61b328a631a56e0f7ee145dc4cf759b24b9a5ec6892c8c692d561f22117714-modules.yaml.xz (IP: 34.49.186.154)
#25 10.64 - Status code: 404 for http://distro.ibiblio.org/rocky/8.10/AppStream/x86_64/os/repodata/fd61b328a631a56e0f7ee145dc4cf759b24b9a5ec6892c8c692d561f22117714-modules.yaml.xz (IP: 152.19.134.43)
#25 10.64 - Status code: 404 for https://mirrors.iu13.net/rocky/8.10/AppStream/x86_64/os/repodata/fd61b328a631a56e0f7ee145dc4cf759b24b9a5ec6892c8c692d561f22117714-modules.yaml.xz (IP: 206.82.17.213)
...
#25 10.65 Error: Failed to download metadata for repo 'appstream': Yum repo downloading error: Downloading error(s): repodata/fd61b328a631a56e0f7ee145dc4cf759b24b9a5ec6892c8c692d561f22117714-modules.yaml.xz - Cannot download, all mirrors were already tried without success
I understand that everyone could just dnf clean all to try the luck again, but apart wasting mirrors bandwidth that’s not doable when automatically building docker images.
Yeah a lot of the problems are mirrors that we are not responsible for. And yes it’s annoying. If the people running them actually ran them properly and ensured that they were synced properly, then it would be fine.
Your best bet would be use the baseurl in all the /etc/yum.repos.d/*.repo files instead of mirror list. At least dl.rockylinux.org that the Rocky team is responsible for would work for you. Downside, if everyone did that it can overload the Rocky infrastructure.
baseurl list
List of URLs for the repository. Defaults to [].
URLs are tried in the listed order (equivalent to yum’s “failovermethod=priority” behaviour).
Therefore, picking some URLs from the list returned by mirrorlist server – the seemingly most reliable mirrors – and adding dl.rockylinux.org as last (failsafe) on the list ought to be a bit better than everybody beelining for the dl first.
I see… Those should be the mirrors that are near the github actions servers in Europe.
Anyways, the issue here is that those mirrors are not in sync, even if it may be more servers affected in other regions. In the meantime, your workaround seems like a good idea, but the solution in my opinion is to either sync those mirrors, or remove them from the mirror list.
Here is the mirror list that is being returned right now (I’ve tested that a few times over today/yesterday and the list is stable, only a few mirrors change between requests)
# repo = rocky-AppStream-9.6 arch = x86_64 country = ES country = NL country = FR country = RS country = DE country = RU country = UA country = HU country = IT country = SE country = FI country = NO country = MD country = RO country = MK country = PL country = GB country = GR country = BG country = DK country = CZ country = BE country = SK country = CH
http://mirror.uv.es/mirror/rocky-linux/9.6/AppStream/x86_64/os/
https://mirror.raiolanetworks.com/rocky-linux/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-southwest1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://rockylinux.hi.no/9.6/AppStream/x86_64/os/
http://mirror.etf.bg.ac.rs/rocky/9.6/AppStream/x86_64/os/
http://mirror.a1.mk/rocky/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west3.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
http://ftp.linux.cz/pub/linux/rocky/9.6/AppStream/x86_64/os/
http://ftp.agdsn.de/rocky-linux/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west9.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
http://mirror.bahnhof.net/pub/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.slu.cz/rocky/9.6/AppStream/x86_64/os/
https://mirror.yandex.ru/rockylinux/9.6/AppStream/x86_64/os/
https://mirrors.dotsrc.org/rockylinux/9.6/AppStream/x86_64/os/
http://ftp.fau.de/rockylinux/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-north1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://rocky-linux-europe-central2.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west2.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://mirror1.hs-esslingen.de/pub/Mirrors/rocky/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west6.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
http://mirror.netzwerge.de/rocky-linux/9.6/AppStream/x86_64/os/
http://mirror.team-host.ru/rocky/rocky-linux/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west1.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://rocky.task.gda.pl/9.6/AppStream/x86_64/os/
https://mirror.telepoint.bg/rocky-linux/9.6/AppStream/x86_64/os/
https://ftp.energotel.sk/pub/linux/rocky/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west8.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://ftp.lysator.liu.se/pub/rocky/9.6/AppStream/x86_64/os/
http://mirror.in2p3.fr/pub/linux/rocky/9.6/AppStream/x86_64/os/
http://mirror.accum.se/mirror/rockylinux.org/9.6/AppStream/x86_64/os/
http://rockylinux.mirror.intermax.nl/9.6/AppStream/x86_64/os/
https://mirror.init7.net/rockylinux/9.6/AppStream/x86_64/os/
http://rockylinux.volia.net/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.karneval.cz/pub/linux/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.netsite.dk/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.plusline.net/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.nl.altushost.com/rocky/9.6/AppStream/x86_64/os/
https://mirrors.nxthost.com/rocky/9.6/AppStream/x86_64/os/
https://mirrors.neterra.net/rocky/9.6/AppStream/x86_64/os/
https://rockylinux.mirrorservice.org/pub/rocky/9.6/AppStream/x86_64/os/
https://distrohub.kyiv.ua/rockylinux/9.6/AppStream/x86_64/os/
https://mirrors.melbourne.co.uk/rocky/9.6/AppStream/x86_64/os/
https://rockylinux.mirrors.ovh.net/9.6/AppStream/x86_64/os/
http://mir01.syntis.net/rockylinux/9.6/AppStream/x86_64/os/
https://rocky.mirror.ukraine.com.ua/9.6/AppStream/x86_64/os/
https://rocky-linux-europe-west4.production.gcp.mirrors.ctrliq.cloud/pub/rocky//9.6/AppStream/x86_64/os/
https://mirror.puzzle.ch/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.nsc.liu.se/rockylinux/9.6/AppStream/x86_64/os/
http://mirrors.hostiserver.com/rocky/9.6/AppStream/x86_64/os/
https://ftp.sh.cvut.cz/rocky/9.6/AppStream/x86_64/os/
https://mirror.prolocation.net/rocky/9.6/AppStream/x86_64/os/
https://ftp.halifax.rwth-aachen.de/rockylinux/9.6/AppStream/x86_64/os/
http://rockylinux.netforce.hosting/rocky/9.6/AppStream/x86_64/os/
http://rocky.reloumirrors.net/9.6/AppStream/x86_64/os/
http://fosszone.csd.auth.gr/rockylinux/9.6/AppStream/x86_64/os/
https://mirror.eu.ossplanet.net/rockylinux/9.6/AppStream/x86_64/os/
http://mirror.hosthink.net/rocky/9.6/AppStream/x86_64/os/
http://quantum-mirror.hu/mirrors/pub/rocky/9.6/AppStream/x86_64/os/
http://ftp.cc.uoc.gr/mirrors/linux/rockylinux/9.6/AppStream/x86_64/os/
http://mirrors.pidginhost.com/rocky-linux/9.6/AppStream/x86_64/os/
As a possible solution (that needs to be implemented on Rocky side), why not before returning the mirror list to a client, verify that those mirrors are in sync?
This can be done by comparing the repomd.xml hash with the base repo one. (Or check the mirrors every x minutes instead of on every request, so that it is unlikely that one of them loses sync between refreshes).
As you said before, there is already some logic in place to return a specific list based on some criteria, so hopefully this feature is easy to implement.
Except that’s not enough. We’ve seen mirrors that have synced those, but not synced the remaining files. Which then means you can’t download the rpm anyway. The mirrors should also run with a particular parameter as well --delay-updates which some aren’t and also causes problems.
I’ve raised all this internally yesterday because it is annoying that the people operating the mirrors aren’t doing their job properly. We’ve also found a lot more mirrors out-of-date than the ones listed above. We are working on it, but the solution isn’t going to be found immediately.