It’s written entirely in Go, supports different database backends, has REST-Style endpoints and easily serves 3.000+ requests per second (on my test box).
I hope this is a useful contribution to the project. PRs to improve it are welcome.
Reading through the GitHub page it sounds very promising and certainly more feature-rich then what I wrote. However, I am not sure if it is suitable for YUM/DNF repositories or if it seems to be primarily used for single stand-alone files.
The former is the component that gives users the ability to add mirrors to the public pool and also provides private mirror functionality as well. It tracks all mirrors and ensures content is in sync. This is written in python.
The latter is the front end component. It uses the data created by mirror manager to answer client requests for the best mirror. This is written in rust.
Hello, I’m from the OpenMandriva team. We are using Mirrorbits for managing our mirrors, as it’s easy to install as a docker stack. I admit I like the fact mirrorlist seems very light.
Do you still work on the project? Is there a documentation on how to set up the three database tables?
edit: what I like about mirrorbits is that you can have such outputs, but it needs a lot of memory to work. It doesn’t rely on repomd.xml but scans each and every file individually.
This allows you to add mirrors to the mirrors table and repos to the repos table.
The status table is populated automatically at run-time, but you can also access it via REST.
I don’t currently work on this project as it’s “feature-complete”.
If you want to use it in production please let me know so I can revisit it.
Thanks @smeier, I’m indeed very interested and it seems very complementary to mirrorbits.
As my knowledge of Go is quite limited yet, it will surely take me some time.
Have you ever considered producing a metalink v4 file instead of a mirrorlist file?
I hadn’t heard of metalink before and looking at my repo files, it seems to be used by EPEL only.
To generate such a file, the program would have to keep track of checksums and file sizes for repomd.xml, which adds complexity. Then it would have to generate XML, which is also more complex than just plain text.
But what gain is there? As far as I can see, dnf/yum will have more or less the same information to work with or maybe I am missing something? Looking at the Metalinker website, it doesn’t really seem to have caught on as a concept.
It is true that Rocky Linux does not use metalink by default in its repo files. Users can swap mirrorlist for metalink and they’ll immediately get that functionality if they wish to. Fedora Linux, EPEL, and CentOS Stream 9 use metalink with mirror manager (shared and provided by the fedora project). (OpenSUSE uses MirrorCache which also uses metalink.)
The point of metalink (in the case of mirror manager) is that it provides additional security checking against out of date mirrors in dnf. dnf will compare the repomd.xml checksums of the mirrors it contacts to the masters. The standard list doesn’t provide this capability. Overall, this helps ensure out of date mirrors (or rogue/malicious mirrors) are not used by dnf.
Mirror manager only tracks repomd.xml. I see this as easier than tracking every single file individually as this helps us with propagation checks, among other things.
When it comes to timestamp, my mirrorlist implementation actually considers this, when selecting mirrors from the database (ORDER BY status.timestamp DESC). That should hopefully eliminate abandoned/out-of-date mirrors.
Providing defense against rogue mirrors is certainly useful, but if that is metalink’s USP, I understand why it has not become the default.
