The RL9 mirror has nothing in “EFI/BOOT” directory. It should have files like “grubx64.efi”, “BOOTX64.EFI” and so on. Why is that? RL9 cannot be EFI netbooted?
Maybe you are looking at a mirror that hasn’t finished syncing yet:
https://download.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/EFI/BOOT/
has all the files mentioned.
1 Like
Sheesh, should’ve done that. Thanks.
Edit: looks like the repo has not been updated in weeks! Index of /Linux/rocky/9/BaseOS/x86_64/os/EFI/BOOT
It is missing most of the files and the last timestamp is July 5.
Also, for some reason the mirror list at Home - Mirror Manager loads reeaa-ll-l-y slow, takes minutes to load the page from just a blank page with a title to a list…
Yeah, with the RL9 release mirror sync is taking a while.
I do not understand. The release was over a week ago and still none of the mirrors I checked have these files?
I would have some feedback. This comes from a view point of user (since we will have RL9 machines) as well as a mirror maintainer (even though for some (bug??) reason our mirror keeps getting removed from the official mirror list).
First of all, the mirror list page is currently extremely slow to load.
It is also very rudimentary compared to most other distributions that show various levels of details about the mirrors (are they up to date and how much lagging behind at minimum).
Second, perhaps a tiered mirror system is needed. It seems also impossible to sync from any of the 3 master rsync mirrors - they are that slow - 1 of the IP addresses totally connection times out since forever and 2 of them time out transferring file list if timeout is set to less than 20-30 seconds (!).
Edit: compared to yesterday the situation seems better. Now it seems possible to get rsync past file list step and the mirror is updating from rsync001/2/3. Yesterday they always timed out at file list step.
It still keeps timing out later at some point transferring the actual files, but seems every time the rsync proceeds a little bit further.
Hello.
Mirror manager by design crawls the mirrors for each repository and architecture. If a repository for that mirror is out of date, it is removed from the pool for that specific repository+architecture combination. (We do have some mirrors that only sync x86 for example, so they’ll never show up for aarch64, ppc64le, or s390x on a mirrorlist request in dnf).
If a mirror is being removed, it is because updates were pushed on our tier 0 and mirror manager did a crawl and found the mirror’s repository+arch to be out of date at that moment in time.
It would be useful to see deeper statistics; it would require us to modify mirror manager to do so or a feature request upstream to Fedora though. I think the design philosophy of mirror manager is different from that of other distribution’s front end mirror systems, but I’m sure they would be open to adding in some deeper stats rather than just “by the numbers” (unless it’s already there and we’re unaware). The propagation page oddly is missing an index, but some stats can be found here. As for the mirror list page being slow to load, I have not seen that occur myself. I’ve browsed it several times over the past week and have not seen any issues in particular.
A tiered system is in the works with documentation and announcement about it on the way. Currently, our rsync pool systems only allow 30 connections at max instead of unlimited as we had traditionally. I think we were able to get away with having unlimited when we only had one major release and two architectures to deal with. Reducing the number of connections helped some mirrors get their syncs completed. With the addition of another major release plus two more architectures, keeping it unlimited was ultimately a detriment. Allowing 100+ connections at once and expecting rsync transfers and directory crawling to work efficiently wouldn’t be the right call if we continued down that path.
With our monitoring, we are not coming close to the maximum IOPS of the backend storage that holds the repositories, even when we were on unlimited. The systems themselves aren’t actually slow. I would say they’re way overtuned for what they do. With that being said, the directory crawling that rsync is doing has to go through over 160k+ files and 3.5k worth of directories. That’s a lot for rsync to go through and this is a case where ingesting an already created file list would be helpful (like what quick-fedora-mirror does). Another thing is that there could easily be a bandwidth limit being hit inbound with the number of connections coming in, on top of mirrors not necessarily close to us trying to sync. And this is why we are looking into setting up tiering to not only keep the syncing from us to tier 1’s efficient, but to allow tier 2’s to sync from a mirror that is geographically closer.
One major bottleneck was the fact that we weren’t able to hard link any of the packages. The example rsync script we provide uses the -H switch, which is only useful if we have hard linking on our end. And now that the hard linking was resolved over the last weekend, rsync transfers should be more efficient after initial crawl. Before our last update push, the number of connections to our rsync nodes had been cut from 30 to between 10 and 15. It is likely we’ll see this jump again.
As an aside, we are considering providing a quick-fedora-mirror type of alternative for mirrors to assist in the directory crawling. While it does the same initial function (such as checking the file list for updates), it also uses the file list it gets as its data to start transferring to avoid filesystem crawling altogether. This could potentially help us and the mirrors (regardless of tier) down the road. We are only at the investigation phase of this at the moment though. Granted, Fedora has 12+ million files, so mirrors having to crawl that is a massive no-go for them… and while I don’t foresee us ever reaching that number, I still see value in an already provided file list that is ingested by the mirror looking to sync the latest content.
2 Likes
Thank you for the reply.
Going back to the original issue. Indeed I can see the EFI boot files in Index of /pub/rocky/9/BaseOS/x86_64/os/EFI/BOOT/ but none of the other mirrors I have tried have more than 1 file and 1 directory in this location.
I even completed a full rsync from the master rsync servers to our mirror and it still did not show any of the expected files. I have left the mirroring to use the master rsync servers and latest sync (4 timer per day) was successful from rsync001 with results:
Number of files: 247,503 (reg: 243,946, dir: 3,545, link: 12)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 759,616,307,697 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 12,198,669
File list generation time: 215.579 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 31
Total bytes received: 12,198,680
sent 31 bytes received 12,198,680 bytes 51,799.20 bytes/sec
total size is 759,616,307,697 speedup is 62,270.21
And still there are no EFI/BOOT files!
Examples of mirrors missing EFI/BOOT files:
http://ftp.jaist.ac.jp/pub/Linux/rocky/9/BaseOS/x86_64/os/EFI/BOOT/
http://ftp.riken.jp/Linux/rocky/9/BaseOS/x86_64/os/EFI/BOOT/
http://ftp.iij.ad.jp/pub/linux/rocky/9/BaseOS/x86_64/os/EFI/BOOT/
I stand by my earlier comment that it is not normal that none of the mirrors I could find are fully synced after 2 weeks of release. It almost looks like there is a problem somewhere with rsync servers (NOT the http/https “download.rockylinux.org” where I was pointed earlier as showing that all is fine).
We used to mirror 4 times daily from one of the tier 1 rsync servers. I have now changed that to the master rsync servers.
We log the rsync results and there have not been errors in long time.
We are also official mirror for several other repositories and none of them keep removing us. Most of those other repositories do far, far more detailed monitoring and reporting and we never have issues with anything else than Rocky Linux. The monitoring shows everything is working fine.
You describe how the system is designed and works in an ideal world. But maybe there is an issue that makes mirror manager to make an invalid decision?
For example a network issue. Where do you monitor from? Only from a single location? Maybe it does not have the best network path to Asia?
For example, right now going to mirror manager “propagation” page at Propagation - Mirror Manager gives an error message:
So clearly the system is not working as it should be.
Also, the mirror server IPv6 address 2600:1f16:d4:7302:52a8:79f4:fe0f:12c1 has never since the beginning worked for us, the rsync connection just times out. This forces us to use IP addresses instead of hostnames, since eventually rsync would hit that IPv6 address when using hostname and keep timing out.
Right at this moment it is loading normally, but around the 2 days of my last posts it was unusably slow - I would reload the page and could check back only several minutes later to finally see the content loaded.
As to statistics, for example Arch Linux and CentOS have had nice mirror statistics. Just about all distributions have nice statistics for mirrors.
If the servers are not overutilized then it sounds like a network issue.
In my earlier posts above it can be seen that on some days they were unusably slow, rsync always timed out receiving file list even with a 30 second --timeout.
It is interesting that you say you needed to reduce the connection limit to allow mirrors to complete but in the next paragraph say that the hardware is “way overtuned” etc. That, too, points to a possible network problem. And it sounds like reducing the connection limit was only a workaround to get something working.
@neil would have to assist with this. I’m not entirely sure how the IPv6 parts are setup here.
Mirror manager looks at the repo data during a crawl and makes a decision. Its decisions are not arbitrary. And as far as I know, the mirror maintainer can login to mirror manager and request a crawl. @neil can likely detail this further.
In my opinion, it should have never been set to unlimited. Not even the Fedora Project uses unlimited on their tier 0 nodes, despite only allowing tier 1 mirrors to sync from them. I was not part of the setup of this part of the infrastructure and since we are still a project in its infancy, these are the growing pains we must face and deal with.
1 Like
Of course the decisions are not arbitrary, they follow a programmed logic.
What I am saying is that you do not seem to account for issues that happen in the real life, contrary to an imaginary perfect world.
It will be interesting to see what is the reply to my other post today about the original issue.
If mirror manager is supposed to remove out-of-date mirrors from the list then there seems to be some deep issues within the mirror system of Rocky Linux.
I should clarify that mirror manager looks for repository metadata at repodata/repomd.xml in each repository and the corresponding files there. Other data is not considered during a crawl. This means that if that repo metadata does not match our tier 0, then the mirror is removed until a successful crawl. Yes, this has its drawbacks and in a lot of cases can be a flawed approach. This is just the current design of mirror manager and how it determines propagation and what mirror is up to date with the master mirror. Apologies for any confusion on the meaning of “repo data” in the context of this thread.
This convinces me that there is something wrong at the mirror manager end (edit: or, of course, between the servers i.e. some network issue). I have personally witnessed several successful full rsyncs from the master mirror servers and our mirror has not been returned to the mirror list.
On the subject of mirror manager being slow - currently it loads in 9 seconds. Almost all of that time is spent downloading rocky.min.css, a 6 MB (!) large CSS from fastly, which is known to be not a good performer in this country. I do wonder why such a simple page would use 6 MB of CSS.
I’ve just logged in to mirror manager and loads fast for me (2 seconds). So the problem isn’t actually Mirror Manager at all, but rather it is location-specific. Maybe that is because Fastly doesn’t have enough POP’s in your area, but seems they will be expanding their network in that region.
Perhaps you should be complaining to Fastly about their lack of network resources in your region (since you mentioned they perform poorly), rather than blaming the Rocky Team or Mirror Manager for the issues you are experiencing.
Best to put things in a more constructive way without accusing or blaming people, to achieve said goals and resolve the issue.
Keep it as a suggestion, there is no need to de-list Japanese mirrors due to missing files.
OpenSUSE uses mirrorbrain/mirrorcache, it balances huge mirrors with terabytes over decades, along with small and incomplete ones. If something is missing (every day with Tumbleweed) or for security measures (core files), are downloaded from Germany or US.
1 Like
Quite a thread. Where was there any accusations or blaming?
Just now, FINALLY, the listed 3 Japanese and all other mirrors now have the “EFI/BOOT” files that was the OP of this thread.
What happened, was something finally fixed?
A mirror is removed for a specific arch+repo combination when the repodata does not match our master mirror, specifically repomd.xml and related files that sit in repodata that mirror manager’s crawler looks for. Files or directories such as images, ppc, .treeinfo, or other arch specific data that would be in BaseOS for example does not count toward propagation and is not checked.
As an example, we have a few mirrors who selectively choose what architectures they wish to sync from us. One of our engineers syncs only x86_64 and aarch64. I sync x86_64 and ppc64le for my private mirror. We have other mirrors who only want x86_64 and don’t sync source. They show up for the architectures and repositories they have available as that is what the crawler finds. They won’t show up for anything else they don’t serve. My mirror doesn’t show up in an aarch64 request for example.
% curl https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-9
# repo = rocky-BaseOS-9.0 arch = x86_64 country = US country = CA country = PR country = CR
http://intrepo.angelsofclockwork.net/os/r/9.0/BaseOS/x86_64/os/
% curl https://mirrors.rockylinux.org/mirrorlist?arch=ppc64le&repo=BaseOS-9
# repo = rocky-BaseOS-9.0 arch = ppc64le country = US country = CA country = PR country = CR
http://intrepo.angelsofclockwork.net/os/r/9.0/BaseOS/ppc64le/os/
% curl https://mirrors.rockylinux.org/mirrorlist?arch=aarch64&repo=BaseOS-9
# repo = rocky-BaseOS-9.0 arch = aarch64 country = US country = CA country = PR country = CR
https://dl.rockylinux.org/pub/rocky/9.0/BaseOS/aarch64/os/
http://mirror.atl.genesisadaptive.com/rocky/9.0/BaseOS/aarch64/os/
https://mirror2.sandyriver.net/pub/rocky/9.0/BaseOS/aarch64/os/
https://mirror.pit.teraswitch.com/rocky/9.0/BaseOS/aarch64/os/
As an aside, if a mirror doesn’t want to sync iso’s or images or they want to be specific on what architectures they carry, that’s 100% their call as a mirror operator. Mirrors will never be penalized or completely “de-listed” for not syncing everything from us. We don’t think it’s reasonable to do so.
I see that the “propagation” page in mirror manager has now been fixed also.
There still remains a typing error in it:
The same diagram is als available for the branches
Should be also.