Partial vault mirror

Teams Infrastructure
1

I have an upcoming project that will require version 8.8 on multiple servers. I currently operate my own up-to-date mirror but once 8.9 came out the 8.8 files were gone, as expected, and I did not have the foresight to make a copy.
I would like to get a copy of 8.8 downloaded locally on my mirror, granted in a separate directory, but have been unable to do so. I first tried rsync before finding out that the vault at dl.rockylinux.org does not support it. So I moved on to wget and have been stymied by a 301 redirect that causes the directories, such as AppStream, to be written as files. It also causes the entire of the vault directory structure to be created even when I tell wget not to do so. Here are the commands I’ve been trying.

wget -m -np -nH --exclude-directories=aarch64,live,Live,ResilientStorage,isos,NFV,nfv,RT,images,rockyrpi,virt,source,8.3,8.4,8.4-RC1,8.5,8.6,8.7,9.0,9.0-ISO,9.1,9.3 https://dl.rockylinux.org/vault/rocky/8.8
wget -m -np -nH --exclude-directories=aarch64,live,Live,ResilientStorage,isos,NFV,nfv,RT,images,rockyrpi,virt,source,8.3,8.4,8.4-RC1,8.5,8.6,8.7,9.0,9.0-ISO,9.1,9.3 https://dl.rockylinux.org/vault/rocky/

Note that I am attempting to keep the amount of data copied to a minimum hence all of the excluded directories.
Interestingly enough attempting to grab a single file, such as the CHECKSUM within isos, will download just fine with no redirect though trying a subdirectory, such as isos, will fail in the same manner as before.
HTTrack is another software I tried but it seemed to do a similar thing by trying to download everything. I haven’t played with exclusions yet but the source URL was still https://dl.rockylinux.org/vault/rocky/8.8 so it shouldn’t have downloaded the 8.3, 8.7, or 9.0(for example) files yet it seemed to have started doing so after I canceled the download to find out what it was doing.

I know I can grab a DVD with the BaseOS but I do have a need for a good chunk of the software that is available and would rather not have to download it on each system from the main server. If anyone knows of a way to complete this please let me know. On a related note does anyone know of a way to auto copy the previous version before an rsync deletes it?

2

The rsync endpoint is not dl.rockylinux.org, it’s msync.rockylinux.org. This is much better than trying to use wget mirroring.

rsync -vrlptDSH \
  --exclude=*.~tmp~ \
  --exclude=... \            # specify an --exclude=... for each thing you want to exclude
  --delete-delay --delay-updates msync.rockylinux.org::rocky-vault/

Alternatively, if you just want 8.8.

rsync -vrlptDSH \
  --exclude=*.~tmp~ \
  --exclude=aarch64 \
  --exclude=live \
  --exclude=Live \
  --exclude=ResilientStorage \
  --exclude=isos \
  --exclude=NFV \
  --exclude=nfv \
  --exclude=RT \
  --exclude=images \
  --exclude=rockyrpi \
  --exclude=virt \
  --exclude=source \
  --delete-delay --delay-updates msync.rockylinux.org::rocky-vault/8.8
2 Likes
3

Thanks for pointing out the correct rsync URL.

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.