We’ve been upgrading our cluster at Montana State University, about 110 nodes total. I’ve been experiencing a weird slowdown when I run dnf makecache. Each metadata file downloads about 90-99% at a few MB/s, then the last bit slows down until the over all download speed is 130K/s. One node is now taking about 5 minutes to get the metadata. I’ve tried setting fastestmirror=True and while it searches for the tastest one it doesn’t improve the speed much. It still gets 90-99% then drops to low KB/s again. I’m getting this on multiple networks within MSU.
Last summer I was able to use PDSH to update 25 at a time and never experienced this. Any thoughts? I’m stumped.
A few questions - Are you hitting different mirrors each time? Are you experiencing this behavior from public mirrors or dl.rockylinux.org or both? If you curl the repodata directly from an affected mirror, do you experience the same issue?
Hi, thanks for the response. Looks like we’re hitting the mirrors list. The base URL for dl.rockylinux.org is commented out. I haven’t tried curl’ing the repodata direct.
Hi, I’m trying to use dnf -d 7 makecache but it doesn’t report which mirror is actually being selected by the mirror list. I uncommented the baseurl=http://dl.rockylinux.org/$contentdir/$releasever/AppStream/$basearch/os/ and commented out the mirror list and I’m seeing the same behavior. It downloaded 99% at a total of 4.1MB/s and then slowed down for the last 1% to the tune of 124K/s by the time it finished.
I turned off all AV scanning and here’s some output that shows what’s going on.
[root@tempest-login dnf-x54g559-dz760hk9]# dnf -d 9 makecache
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.7.0
Command: dnf -d 9 makecache
Installroot: /
Releasever: 8
cachedir: /var/cache/dnf
Base command: makecache
Extra commands: [‘-d’, ‘9’, ‘makecache’]
Making cache files for all metadata files.
appstream: has expired and will be refreshed.
baseos: has expired and will be refreshed.
extras: has expired and will be refreshed.
powertools: has expired and will be refreshed.
cuda-rhel8-x86_64: has expired and will be refreshed.
epel: has expired and will be refreshed.
influxdb: has expired and will be refreshed.
packages-microsoft-com-prod: has expired and will be refreshed.
repo: downloading from remote: appstream
error: Curl error (28): Timeout was reached for http://dl.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/repodata/97dc40b53f287a000a7fc9a1f699529d2ef219d5aca5af3580730edba7a72b8a-updateinfo.xml.gz [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] (http://dl.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/repodata/97dc40b53f287a000a7fc9a1f699529d2ef219d5aca5af3580730edba7a72b8a-updateinfo.xml.gz).
^CRocky Linux 8 - AppStream 99% [===============================================================-] 19 MB/s | 9.6 MB 00:00 ETRocky Linux 8 - AppStream 245 kB/s | 9.6 MB 00:40
Errors during downloading metadata for repository ‘appstream’: