Packages in dl repo changing size?

Infrastructure
1

I am noticing that packages served on the rocky9 download repo are changing over time, even though the filename is the same…

As an example, for package fonts-srpm-macros-2.0.5-7.el9.1.noarch.rpm the original version on the Rocky9 release ISO has the following:

Signature : RSA/SHA256, Mon 16 May 2022 08:08:00 AM EDT, Key ID 702d426d350d275d
Build Date : Mon 16 May 2022 08:07:43 AM EDT
Build Host : pb-9e1036bd-900d-4786-8a64-762dfaf3bec2-b-noarch

But the version (same filename) currently on the download repo has this:

rpm -qp --info http://dl.rockylinux.org/pub/rocky/9/AppStream/x86_64/os/Packages/f/fonts-srpm-macros-2.0.5-7.el9.1.noarch.rpm | grep ^Build

Build Date : Fri 13 May 2022 08:49:52 AM EDT
Build Host : pb-83a639b8-328f-4af5-ba0e-4767a3907396-b-noarch

i.e. the current repo copy is 3 days OLDER than the version that was on the ISO.

Further, the change on the repo has happened in the last day or so. Up until yesterday the repo version was identical to the ISO version. I have a copy of the RPM that I downloaded which has the 16 May build and other RPM details match that on the ISO.

So it looks like the package on the repo has reverted to an earlier 13 May build with same filename ?

As further example, not all the mirrors have synced from the ‘new’ older 13 May build yet:

rpm -qp --info http://rockylinux.ip-connect.info/9/AppStream/x86_64/os/Packages/f/fonts-srpm-macros-2.0.5-7.el9.1.noarch.rpm | grep ^Build

Build Date : Mon 16 May 2022 08:07:43 AM EDT
Build Host : pb-9e1036bd-900d-4786-8a64-762dfaf3bec2-b-noarch

So why are packages switching builds on the dl repo but with the same filenames and version ?

Note: This is not the first time I’ve noticed this, but it’s the first time I’ve bothered to look into why my locally cached copy doesn’t match the dl.rockylinux.org upstream.

2

Here are some observations:

  • Our ISO’s have had different iterations since July and we were facing some issues before, during, and after release with our package NEVRA’s. This would explain the discrepancy and it is likely something we missed.
  • The package did not change overnight. The last compose that showed the package changing was July 14th. After that, it has effectively been skipped each time.
  • Looking at two close mirrors to me, the date is showing as the 13th. So I cannot answer why the mirror you’ve picked is showing something different, unless they are not syncing properly.
# rpm -qpi https://rocky.deadbatteries.work/9/AppStream/x86_64/os/Packages/f/fonts-srpm-macros-2.0.5-7.el9.1.noarch.rpm | grep 'Build Date'
Build Date  : Fri 13 May 2022 12:49:52 PM UTC
# rpm -qpi https://mirror.fcix.net/rocky/9.0/AppStream/x86_64/os/Packages/f/fonts-srpm-macros-2.0.5-7.el9.1.noarch.rpm | grep 'Build Date'
Build Date  : Fri 13 May 2022 12:49:52 PM UTC

With all that being said, we’ll look into and try to resolve why package is effectively different between repo and ISO. It is likely an error within our new build system.

3

OK, thanks for investigating.

Just FYI, I checked the rocky.centos-repo.net mirror which is a “no-delete” mirror, and it too has the original package.

So depending which mirror you hit, you’ll either get the original 16 May built package, or the one built on 13 May that was published later.

This problem isn’t unique to that package, definitely something in the build system is causing packages to be ‘silently’ replaced on the repo after their initial releases. I’ll post back here next time I see one change…

1 Like