Downloading updates (reposync) bad repodata?

Lothar · August 31, 2021, 6:36pm

A slightly different question than the previous ‘updates/patches’ questions that have been posted…

I’m using ‘reposync’ to download the ‘os’ repository to a local ‘updates’ repo (for my local installations), and encountered a possible ‘poisoning’ of the remote (mirror(s) ) repodata and/or packages…

All “Appstream” RPM updates downloaded successfully, and most of the “BaseOS” packages, EXCEPT for the following:

fwupdate-11-3.el8.x86_64.rpm   FAILED
fwupdate-efi-11-3.el8.x86_64.r FAILED
fwupdate-libs-11-3.el8.x86_64. FAILED
grub2-common-2.02-99.el8.noarc FAILED
grub2-efi-aa64-modules-2.02-99 FAILED
grub2-efi-ia32-2.02-99.el8.x86 FAILED
grub2-efi-ia32-cdboot-2.02-99. FAILED
grub2-efi-ia32-modules-2.02-99 FAILED
grub2-efi-x64-2.02-99.el8.x86_ FAILED
grub2-efi-x64-cdboot-2.02-99.e FAILED
grub2-efi-x64-modules-2.02-99. FAILED
grub2-pc-2.02-99.el8.x86_64.rp FAILED
grub2-pc-modules-2.02-99.el8.n FAILED
grub2-tools-2.02-99.el8.x86_64 FAILED
grub2-tools-efi-2.02-99.el8.x8 FAILED
grub2-tools-extra-2.02-99.el8. FAILED
grub2-tools-minimal-2.02-99.el FAILED
1:grub2-efi-x64-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-x64-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-efi-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-efi-x64-cdboot-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-pc-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-cdboot-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
fwupdate-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-minimal-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-pc-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-aa64-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-common-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
fwupdate-efi-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-extra-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
fwupdate-libs-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.

I’ve check local file/directory permissions, and disk space (still gotz lotz), and then manually downloaded each of the ‘failed’ packages, and found the following…


-rw-r--r-- 1 root admin 32068 May 27 11:23 iso/fwupdate-11-3.el8.x86_64.rpm
-rw-r--r-- 1 root admin 32148 Jul 18 05:05 yum/fwupdate-11-3.el8.x86_64.rpm

-rw-r--r-- 1 root admin 64064 May 27 11:23 iso/fwupdate-efi-11-3.el8.x86_64.rpm
-rw-r--r-- 1 root admin 64128 Jul 18 05:05 yum/fwupdate-efi-11-3.el8.x86_64.rpm

-rw-r--r-- 1 root admin 36232 May 27 11:23 iso/fwupdate-libs-11-3.el8.x86_64.rpm
-rw-r--r-- 1 root admin 36300 Jul 18 05:05 yum/fwupdate-libs-11-3.el8.x86_64.rpm

-rw-r--r-- 1 root admin 910844 May 27 11:30 iso/grub2-common-2.02-99.el8.noarch.rpm
-rw-r--r-- 1 root admin 910916 Jul 18 05:39 yum/grub2-common-2.02-99.el8.noarch.rpm

-rw-r--r-- 1 root admin 1029280 May 27 11:29 iso/grub2-efi-aa64-modules-2.02-99.el8.noarch.rpm
-rw-r--r-- 1 root admin 1029360 Jul 18 05:38 yum/grub2-efi-aa64-modules-2.02-99.el8.noarch.rpm

-rw-r--r-- 1 root admin 461040 May 27 11:30 iso/grub2-efi-ia32-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 461116 Jul 18 05:39 yum/grub2-efi-ia32-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 1269048 May 27 11:30 iso/grub2-efi-ia32-cdboot-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 1269116 Jul 18 05:39 yum/grub2-efi-ia32-cdboot-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 1083748 May 27 11:30 iso/grub2-efi-ia32-modules-2.02-99.el8.noarch.rpm
-rw-r--r-- 1 root admin 1083896 Jul 18 05:39 yum/grub2-efi-ia32-modules-2.02-99.el8.noarch.rpm

-rw-r--r-- 1 root admin 485808 May 27 11:30 iso/grub2-efi-x64-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 485872 Jul 18 05:39 yum/grub2-efi-x64-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 1293464 May 27 11:30 iso/grub2-efi-x64-cdboot-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 1293532 Jul 18 05:39 yum/grub2-efi-x64-cdboot-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 1109336 May 27 11:30 iso/grub2-efi-x64-modules-2.02-99.el8.noarch.rpm
-rw-r--r-- 1 root admin 1109444 Jul 18 05:39 yum/grub2-efi-x64-modules-2.02-99.el8.noarch.rpm

-rw-r--r-- 1 root admin 41260 May 27 11:30 iso/grub2-pc-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 41328 Jul 18 05:39 yum/grub2-pc-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 934864 May 27 11:30 iso/grub2-pc-modules-2.02-99.el8.noarch.rpm
-rw-r--r-- 1 root admin 934940 Jul 18 05:39 yum/grub2-pc-modules-2.02-99.el8.noarch.rpm

-rw-r--r-- 1 root admin 2069244 May 27 11:30 iso/grub2-tools-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 2069252 Jul 18 05:39 yum/grub2-tools-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 484380 May 27 11:30 iso/grub2-tools-efi-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 484432 Jul 18 05:39 yum/grub2-tools-efi-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 1121164 May 27 11:30 iso/grub2-tools-extra-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 1121216 Jul 18 05:39 yum/grub2-tools-extra-2.02-99.el8.x86_64.rpm

-rw-r--r-- 1 root admin 212952 May 27 11:30 iso/grub2-tools-minimal-2.02-99.el8.x86_64.rpm
-rw-r--r-- 1 root admin 213032 Jul 18 05:39 yum/grub2-tools-minimal-2.02-99.el8.x86_64.rpm

Checking the RPM ‘metadata’, the only difference is the ‘build info’ and file modification dates (seen above)…

> diff /tmp/iso.rpm /tmp/yum.rpm
10c10
< Signature   : RSA/SHA256, Thu 27 May 2021 11:30:11 AM MDT, Key ID 15af5dac6d745a60
---
> Signature   : RSA/SHA256, Sun 18 Jul 2021 05:39:09 AM MDT, Key ID 15af5dac6d745a60
12c12
< Build Date  : Thu 27 May 2021 11:21:59 AM MDT
---
> Build Date  : Sun 18 Jul 2021 05:29:49 AM MDT

The rest of the ‘metadata’ is identical, which makes me suspect that ‘modified’ packages were inserted into the remote (I check several of the ‘mirror’ sites) Packages subdirectory either without updating their version/release info, and/or not updating the repodata…

> rpm -q -i -p iso/grub2-efi-ia32-2.02-99.el8.x86_64.rpm | tee /tmp/iso.rpm
warning: iso/grub2-efi-ia32-2.02-99.el8.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 6d745a60: NOKEY
Name        : grub2-efi-ia32
Epoch       : 1
Version     : 2.02
Release     : 99.el8
Architecture: x86_64
Install Date: (not installed)
Group       : System Environment/Base
Size        : 1474064
License     : GPLv3+
Signature   : RSA/SHA256, Thu 27 May 2021 11:30:11 AM MDT, Key ID 15af5dac6d745a60
Source RPM  : grub2-2.02-99.el8.src.rpm
Build Date  : Thu 27 May 2021 11:21:59 AM MDT
Build Host  : ord1-prod-x86build002.svc.aws.rockylinux.org
Relocations : (not relocatable)
Packager    : infrastructure@rockylinux.org
Vendor      : Rocky
URL         : http://www.gnu.org/software/grub/
Summary     : GRUB for EFI systems.
Description :

The GRand Unified Bootloader (GRUB) is a highly configurable and
customizable bootloader with modular architecture.  It supports a rich
variety of kernel formats, file systems, computer architectures and
hardware devices.

This subpackage provides support for efi-ia32 systems.

> rpm -q -i -p yum/grub2-efi-ia32-2.02-99.el8.x86_64.rpm | tee /tmp/yum.rpm
warning: yum/grub2-efi-ia32-2.02-99.el8.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 6d745a60: NOKEY
Name        : grub2-efi-ia32
Epoch       : 1
Version     : 2.02
Release     : 99.el8
Architecture: x86_64
Install Date: (not installed)
Group       : System Environment/Base
Size        : 1474064
License     : GPLv3+
Signature   : RSA/SHA256, Sun 18 Jul 2021 05:39:09 AM MDT, Key ID 15af5dac6d745a60
Source RPM  : grub2-2.02-99.el8.src.rpm
Build Date  : Sun 18 Jul 2021 05:29:49 AM MDT
Build Host  : ord1-prod-x86build002.svc.aws.rockylinux.org
Relocations : (not relocatable)
Packager    : infrastructure@rockylinux.org
Vendor      : Rocky
URL         : http://www.gnu.org/software/grub/
Summary     : GRUB for EFI systems.
Description :

The GRand Unified Bootloader (GRUB) is a highly configurable and
customizable bootloader with modular architecture.  It supports a rich
variety of kernel formats, file systems, computer architectures and
hardware devices.

This subpackage provides support for efi-ia32 systems.

Lothar · September 3, 2021, 6:27pm

The issue still exists (for ‘me’)…

fwupdate-11-3.el8.x86_64.rpm   FAILED
fwupdate-efi-11-3.el8.x86_64.r FAILED
fwupdate-libs-11-3.el8.x86_64. FAILED
grub2-common-2.02-99.el8.noarc FAILED
grub2-efi-aa64-modules-2.02-99 FAILED
grub2-efi-ia32-2.02-99.el8.x86 FAILED
grub2-efi-ia32-cdboot-2.02-99. FAILED
grub2-efi-ia32-modules-2.02-99 FAILED
grub2-efi-x64-2.02-99.el8.x86_ FAILED
grub2-efi-x64-cdboot-2.02-99.e FAILED
grub2-efi-x64-modules-2.02-99. FAILED
grub2-pc-2.02-99.el8.x86_64.rp FAILED
grub2-pc-modules-2.02-99.el8.n FAILED
grub2-tools-2.02-99.el8.x86_64 FAILED
grub2-tools-efi-2.02-99.el8.x8 FAILED
grub2-tools-extra-2.02-99.el8. FAILED
grub2-tools-minimal-2.02-99.el FAILED
1:grub2-efi-x64-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-x64-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-efi-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-efi-x64-cdboot-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-pc-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-cdboot-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
fwupdate-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-minimal-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-pc-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-aa64-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-common-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
fwupdate-efi-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-extra-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
fwupdate-libs-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.

Fri Sep  3 12:24:55 MDT 2021

Has anyone else encountered ‘this’ issue ?

Is there anybody there ? Doesn’t anybody care ? Doesn’t anybody see what I see ?
(sung to the tune from “1776”)

stack · September 9, 2021, 12:27am

Greetings,
There were a few packages that we needed to reset early on. Since there were no changes the build number didn’t reset. Sorry if it caused any concern.

If you want a complete mirror, I’d suggest finding one of the official rsync mirrors in your country and just doing a once-a-day sync from one of the second tier mirrors:
https://mirrors.rockylinux.org/mirrormanager/mirrors/Rocky/8.4

Hope that helps.
~Stack~

neil · September 9, 2021, 12:46am

To add on to this: if the trouble you were having is with these reset packages (the grub ones), then please also rest assured this will not happen again. It was a one-time mistake that was not caught before the release went GA. We’ve added checks in to ensure it does not occur again.

Lothar · September 9, 2021, 6:12pm

Oh queso… Your replies prodded me to do some more testing, and experimentation, resulting in the following…

Using 2 of the ‘official’ mirrors, referenced on the Rocky website, the original issue still exists…


##baseurl=https://repos.eggycrew.com/rockylinux/8.4/BaseOS/x86_64/os/
##baseurl=https://repo.uccs.edu/rocky-linux/8.4/BaseOS/x86_64/os/

fwupdate-efi-11-3.el8.x86_64.r FAILED
fwupdate-11-3.el8.x86_64.rpm   FAILED
fwupdate-libs-11-3.el8.x86_64. FAILED
grub2-common-2.02-99.el8.noarc FAILED
grub2-efi-aa64-modules-2.02-99 FAILED
grub2-efi-ia32-2.02-99.el8.x86 FAILED
grub2-efi-ia32-cdboot-2.02-99. FAILED
grub2-efi-ia32-modules-2.02-99 FAILED
grub2-efi-x64-2.02-99.el8.x86_ FAILED
grub2-efi-x64-cdboot-2.02-99.e FAILED
grub2-efi-x64-modules-2.02-99. FAILED
grub2-pc-2.02-99.el8.x86_64.rp FAILED
grub2-pc-modules-2.02-99.el8.n FAILED
grub2-tools-2.02-99.el8.x86_64 FAILED
grub2-tools-efi-2.02-99.el8.x8 FAILED
grub2-tools-extra-2.02-99.el8. FAILED
grub2-tools-minimal-2.02-99.el FAILED
1:grub2-efi-x64-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-x64-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-efi-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-efi-x64-cdboot-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-pc-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-cdboot-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
fwupdate-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-minimal-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-pc-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-aa64-modules-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
1:grub2-efi-ia32-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-common-2.02-99.el8.noarch: [Errno 256] No more mirrors to try.
fwupdate-efi-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-extra-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.
fwupdate-libs-11-3.el8.x86_64: [Errno 256] No more mirrors to try.
1:grub2-tools-2.02-99.el8.x86_64: [Errno 256] No more mirrors to try.

Thu Sep  9 11:47:51 MDT 2021

BUT, by modifying my config file for the reposync (for Rocky 8.4) to use the ‘base’ repo location, the reposync process completed successfully… (wee WEEEE !)

AND, included 19 additional updated packages that weren’t referenced by the ‘mirror’ sites… (ie. bpftool, kernel*, perf)…

##https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/

(1/36): fwupdate-11-3.el8.x86_64.rpm                       |  31 kB   00:00
(2/36): fwupdate-efi-11-3.el8.x86_64.rpm                   |  63 kB   00:00
(3/36): fwupdate-libs-11-3.el8.x86_64.rpm                  |  35 kB   00:00
(4/36): grub2-common-2.02-99.el8.noarch.rpm                | 890 kB   00:00
(5/36): bpftool-4.18.0-305.17.1.el8_4.x86_64.rpm           | 6.6 MB   00:01
(6/36): grub2-efi-ia32-2.02-99.el8.x86_64.rpm              | 450 kB   00:00
(7/36): grub2-efi-aa64-modules-2.02-99.el8.noarch.rpm      | 1.0 MB   00:00
(8/36): grub2-efi-ia32-cdboot-2.02-99.el8.x86_64.rpm       | 1.2 MB   00:00
(9/36): grub2-efi-ia32-modules-2.02-99.el8.noarch.rpm      | 1.0 MB   00:00
(10/36): grub2-efi-x64-2.02-99.el8.x86_64.rpm              | 474 kB   00:00
(11/36): grub2-efi-x64-cdboot-2.02-99.el8.x86_64.rpm       | 1.2 MB   00:00
(12/36): grub2-pc-2.02-99.el8.x86_64.rpm                   |  40 kB   00:00
(13/36): grub2-efi-x64-modules-2.02-99.el8.noarch.rpm      | 1.1 MB   00:00
(14/36): grub2-pc-modules-2.02-99.el8.noarch.rpm           | 913 kB   00:00
(15/36): grub2-tools-efi-2.02-99.el8.x86_64.rpm            | 473 kB   00:00
(16/36): grub2-tools-2.02-99.el8.x86_64.rpm                | 2.0 MB   00:00
(17/36): grub2-tools-minimal-2.02-99.el8.x86_64.rpm        | 208 kB   00:00
(18/36): grub2-tools-extra-2.02-99.el8.x86_64.rpm          | 1.1 MB   00:00
(19/36): kernel-abi-stablelists-4.18.0-305.17.1.el8_4.noar | 5.9 MB   00:00
(20/36): kernel-4.18.0-305.17.1.el8_4.x86_64.rpm           | 5.9 MB   00:01
(21/36): kernel-cross-headers-4.18.0-305.17.1.el8_4.x86_64 |  11 MB   00:02
(22/36): kernel-debug-4.18.0-305.17.1.el8_4.x86_64.rpm     | 5.9 MB   00:02
(23/36): kernel-core-4.18.0-305.17.1.el8_4.x86_64.rpm      |  36 MB   00:13
(24/36): kernel-debug-devel-4.18.0-305.17.1.el8_4.x86_64.r |  19 MB   00:06
(25/36): kernel-debug-core-4.18.0-305.17.1.el8_4.x86_64.rp |  63 MB   00:23
(26/36): kernel-debug-modules-extra-4.18.0-305.17.1.el8_4. | 7.2 MB   00:02
(27/36): kernel-debug-modules-4.18.0-305.17.1.el8_4.x86_64 |  54 MB   00:18
(28/36): kernel-devel-4.18.0-305.17.1.el8_4.x86_64.rpm     |  18 MB   00:07
(29/36): kernel-headers-4.18.0-305.17.1.el8_4.x86_64.rpm   | 7.1 MB   00:02
(30/36): kernel-doc-4.18.0-305.17.1.el8_4.noarch.rpm       |  23 MB   00:09
(31/36): kernel-modules-extra-4.18.0-305.17.1.el8_4.x86_64 | 6.6 MB   00:02
(32/36): kernel-modules-4.18.0-305.17.1.el8_4.x86_64.rpm   |  28 MB   00:10
(33/36): kernel-tools-4.18.0-305.17.1.el8_4.x86_64.rpm     | 6.1 MB   00:02
(34/36): kernel-tools-libs-4.18.0-305.17.1.el8_4.x86_64.rp | 5.9 MB   00:02
(35/36): perf-4.18.0-305.17.1.el8_4.x86_64.rpm             | 8.1 MB   00:03
(36/36): python3-perf-4.18.0-305.17.1.el8_4.x86_64.rpm     | 6.0 MB   00:02

Thu Sep  9 11:58:55 MDT 2021
** File Download Process Completed **

Such that, I suspect that there ‘is’ an issue with the syncing of the ‘mirror’ sites, at least with the “repodata” contents…

I don’t want to ‘hammer’ the source Rocky repo site for all of my future updates, and hope that if there ‘is’ an issue with the ‘mirror’ site(s), that it’s identified and resolved, so I can bother ‘them’ instead…

Lothar · September 9, 2021, 6:18pm

AH ! I will acknowledge that those additional updates, were created within the past 24 hours… And may not have made their way through the ‘mirror’ list…