Partial vault mirror

I have an upcoming project that will require version 8.8 on multiple servers. I currently operate my own up-to-date mirror but once 8.9 came out the 8.8 files were gone, as expected, and I did not have the foresight to make a copy.
I would like to get a copy of 8.8 downloaded locally on my mirror, granted in a separate directory, but have been unable to do so. I first tried rsync before finding out that the vault at dl.rockylinux.org does not support it. So I moved on to wget and have been stymied by a 301 redirect that causes the directories, such as AppStream, to be written as files. It also causes the entire of the vault directory structure to be created even when I tell wget not to do so. Here are the commands I’ve been trying.

wget -m -np -nH --exclude-directories=aarch64,live,Live,ResilientStorage,isos,NFV,nfv,RT,images,rockyrpi,virt,source,8.3,8.4,8.4-RC1,8.5,8.6,8.7,9.0,9.0-ISO,9.1,9.3 https://dl.rockylinux.org/vault/rocky/8.8
wget -m -np -nH --exclude-directories=aarch64,live,Live,ResilientStorage,isos,NFV,nfv,RT,images,rockyrpi,virt,source,8.3,8.4,8.4-RC1,8.5,8.6,8.7,9.0,9.0-ISO,9.1,9.3 https://dl.rockylinux.org/vault/rocky/

Note that I am attempting to keep the amount of data copied to a minimum hence all of the excluded directories.
Interestingly enough attempting to grab a single file, such as the CHECKSUM within isos, will download just fine with no redirect though trying a subdirectory, such as isos, will fail in the same manner as before.
HTTrack is another software I tried but it seemed to do a similar thing by trying to download everything. I haven’t played with exclusions yet but the source URL was still https://dl.rockylinux.org/vault/rocky/8.8 so it shouldn’t have downloaded the 8.3, 8.7, or 9.0(for example) files yet it seemed to have started doing so after I canceled the download to find out what it was doing.

I know I can grab a DVD with the BaseOS but I do have a need for a good chunk of the software that is available and would rather not have to download it on each system from the main server. If anyone knows of a way to complete this please let me know. On a related note does anyone know of a way to auto copy the previous version before an rsync deletes it?

The rsync endpoint is not dl.rockylinux.org, it’s msync.rockylinux.org. This is much better than trying to use wget mirroring.

rsync -vrlptDSH \
  --exclude=*.~tmp~ \
  --exclude=... \            # specify an --exclude=... for each thing you want to exclude
  --delete-delay --delay-updates msync.rockylinux.org::rocky-vault/

Alternatively, if you just want 8.8.

rsync -vrlptDSH \
  --exclude=*.~tmp~ \
  --exclude=aarch64 \
  --exclude=live \
  --exclude=Live \
  --exclude=ResilientStorage \
  --exclude=isos \
  --exclude=NFV \
  --exclude=nfv \
  --exclude=RT \
  --exclude=images \
  --exclude=rockyrpi \
  --exclude=virt \
  --exclude=source \
  --delete-delay --delay-updates msync.rockylinux.org::rocky-vault/8.8
1 Like

Thanks for pointing out the correct rsync URL.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.