Filesystem-3.16-5 package broken?

Rocky Linux Tech Help
1

Hi

I did today dnf update but it looks like the filesystem-3.16-5.el9.x86_64 is broken?

dnf -y update
Last metadata expiration check: 3:04:58 ago on Thu 21 Nov 2024 06:15:46 AM CET.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                     Architecture                            Version                                        Repository                               Size
==================================================================================================================================================================================
Upgrading:
 filesystem                                  x86_64                                  3.16-5.el9                                     baseos                                  1.1 M

Transaction Summary
==================================================================================================================================================================================
Upgrade  1 Package

Total download size: 1.1 M
Downloading Packages:
filesystem-3.16-5.el9.x86_64.rpm                                                                                                                  6.5 MB/s | 1.1 MB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                             2.0 MB/s | 1.1 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-5.el9.x86_64                                                                                                                             1/1
  Preparing        :                                                                                                                                                          1/1
  Upgrading        : filesystem-3.16-5.el9.x86_64                                                                                                                             1/2
Error unpacking rpm package filesystem-3.16-5.el9.x86_64
  Verifying        : filesystem-3.16-5.el9.x86_64                                                                                                                             1/2
  Verifying        : filesystem-3.16-2.el9.x86_64                                                                                                                             2/2

Failed:
  filesystem-3.16-2.el9.x86_64                                                            filesystem-3.16-5.el9.x86_64

Error: Transaction failed

I also manually downloaded it from multiple different networks to check if it was somewhere wrong cached. But the SHA256 is always CABE98453B9517158C5A045CFEC0418B747C874DA2717D183F4296F240BDF8D9

Any idea?

2

Update:

Some workstation / servers can install it. Most of it not. Doesn’t matter if VM or physical machine

3

I ran into this problem as well when upgrading RL 9.4 to 9.5.

Any chance you are running OpenAFS? I would include a URL but this is my first post and I don’t want it to be sent to purgatory.

For me, the smoking gun was in /var/log/dnf.rpm.log:
2024-11-20T15:25:58-0500 INFO error: unpacking of archive failed on file /afs: cpio: chmod failed - Device or resource busy

The solution was to stop the OpenAFS client while doing the update:

systemctl stop openafs-client
dnf update
systemctl start openafs-client

Of course if you are running OpenAFS, you’ll also need to compile a new module to go with the new kernel that ships with RL 9.5 (and every time there is a new point release). Weak updates are fine in my experience when new kernels are pushed out between point releases.

4

Unfortunately not the case on my machines.

cat /var/log/dnf.rpm.log
2024-11-21T18:34:39+0100 ERROR Error unpacking rpm package filesystem-3.16-5.el9.x86_64

dnf.log is also not helpfull.

2024-11-21T18:34:37+0100 INFO --- logging initialized ---
2024-11-21T18:34:37+0100 DDEBUG timer: config: 3 ms
2024-11-21T18:34:37+0100 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, notify-packagekit, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
2024-11-21T18:34:37+0100 DEBUG DNF version: 4.14.0
2024-11-21T18:34:37+0100 DDEBUG Command: dnf -y --rpmverbosity=debug update
2024-11-21T18:34:37+0100 DDEBUG Installroot: /
2024-11-21T18:34:37+0100 DDEBUG Releasever: 9
2024-11-21T18:34:37+0100 DEBUG cachedir: /var/cache/dnf
2024-11-21T18:34:37+0100 DDEBUG Base command: update
2024-11-21T18:34:37+0100 DDEBUG Extra commands: ['-y', '--rpmverbosity=debug', 'update']
2024-11-21T18:34:37+0100 DEBUG User-Agent: constructed: 'libdnf (Rocky Linux 9.5; generic; Linux.x86_64)'
2024-11-21T18:34:37+0100 DEBUG repo: using cache for: epel
2024-11-21T18:34:37+0100 DEBUG epel: using metadata from Wed 20 Nov 2024 03:22:00 PM CET.
2024-11-21T18:34:37+0100 DEBUG repo: using cache for: epel-cisco-openh264
2024-11-21T18:34:37+0100 DEBUG epel-cisco-openh264: using metadata from Mon 22 May 2023 06:19:21 PM CEST.
2024-11-21T18:34:37+0100 DEBUG repo: using cache for: baseos
2024-11-21T18:34:38+0100 DEBUG baseos: using metadata from Sat 16 Nov 2024 10:34:11 PM CET.
2024-11-21T18:34:38+0100 DEBUG repo: using cache for: appstream
2024-11-21T18:34:38+0100 DEBUG appstream: using metadata from Wed 20 Nov 2024 07:26:25 PM CET.
2024-11-21T18:34:38+0100 DEBUG repo: using cache for: extras
2024-11-21T18:34:38+0100 DEBUG extras: using metadata from Thu 14 Nov 2024 06:50:01 AM CET.
2024-11-21T18:34:38+0100 DEBUG repo: using cache for: zabbix
2024-11-21T18:34:38+0100 DEBUG zabbix: using metadata from Tue 19 Nov 2024 06:29:35 PM CET.
2024-11-21T18:34:38+0100 DEBUG repo: using cache for: zabbix-non-supported
2024-11-21T18:34:38+0100 DEBUG zabbix-non-supported: using metadata from Wed 06 Jul 2022 01:03:24 PM CEST.
2024-11-21T18:34:38+0100 INFO Last metadata expiration check: 0:27:54 ago on Thu 21 Nov 2024 06:06:44 PM CET.
2024-11-21T18:34:38+0100 DDEBUG timer: sack setup: 657 ms
2024-11-21T18:34:38+0100 DEBUG Excludes in repo epel: zabbix*
2024-11-21T18:34:38+0100 DEBUG --> Starting dependency resolution
2024-11-21T18:34:38+0100 DEBUG ---> Package filesystem.x86_64 3.16-2.el9 will be upgraded
2024-11-21T18:34:38+0100 DEBUG ---> Package filesystem.x86_64 3.16-5.el9 will be an upgrade
2024-11-21T18:34:38+0100 DEBUG --> Finished dependency resolution
2024-11-21T18:34:38+0100 DDEBUG timer: depsolve: 30 ms
2024-11-21T18:34:38+0100 INFO Dependencies resolved.
2024-11-21T18:34:38+0100 INFO ================================================================================
 Package             Architecture    Version              Repository       Size
================================================================================
Upgrading:
 filesystem          x86_64          3.16-5.el9           baseos          1.1 M

Transaction Summary
================================================================================
Upgrade  1 Package

2024-11-21T18:34:38+0100 INFO Total download size: 1.1 M
2024-11-21T18:34:38+0100 INFO Downloading Packages:
2024-11-21T18:34:38+0100 INFO --------------------------------------------------------------------------------
2024-11-21T18:34:38+0100 INFO Total                                           4.2 MB/s | 1.1 MB     00:00
2024-11-21T18:34:38+0100 DEBUG Using rpmkeys executable at /usr/bin/rpmkeys to verify signatures
2024-11-21T18:34:38+0100 INFO Running transaction check
2024-11-21T18:34:38+0100 INFO Transaction check succeeded.
2024-11-21T18:34:38+0100 INFO Running transaction test
2024-11-21T18:34:38+0100 INFO Transaction test succeeded.
2024-11-21T18:34:38+0100 DDEBUG timer: transaction test: 125 ms
2024-11-21T18:34:38+0100 INFO Running transaction
2024-11-21T18:34:38+0100 DDEBUG RPM transaction start.
2024-11-21T18:34:39+0100 DDEBUG RPM transaction over.
2024-11-21T18:34:39+0100 DEBUG Errors occurred during transaction.
2024-11-21T18:34:52+0100 DDEBUG timer: verify transaction: 13344 ms
2024-11-21T18:34:52+0100 DDEBUG timer: transaction: 14105 ms
2024-11-21T18:34:52+0100 DEBUG Completion plugin: Generating completion cache...
2024-11-21T18:34:53+0100 DEBUG Failed: filesystem-3.16-2.el9.x86_64
2024-11-21T18:34:53+0100 DEBUG Failed: filesystem-3.16-5.el9.x86_64
2024-11-21T18:34:53+0100 DDEBUG Cleaning up.
2024-11-21T18:34:53+0100 DDEBUG /var/cache/dnf/baseos-522ed8e2b2f761ff/packages/filesystem-3.16-5.el9.x86_64.rpm removed
2024-11-21T18:34:53+0100 DDEBUG Plugins were unloaded.
2024-11-21T18:34:53+0100 SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 176, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.9/site-packages/dnf/cli/cli.py", line 264, in do_transaction
    raise dnf.exceptions.Error(_('Transaction failed'))
dnf.exceptions.Error: Transaction failed
2024-11-21T18:34:53+0100 CRITICAL Error: Transaction failed

I have issued dnf -y --rpmverbosity=debug update but i don’t see any obvious reason why it doenst work.

5

Your post title says 3.16.2, but that’s the old one, the new one is 3.16.5

It might be worth verify the old one anyway:

rpm --verify filesystem-3.16-2.el9.x86_64

no output means it’s ok

6

Your are right. I changed the title.

When i do that then i get a long list of paths like

missing     /usr/share/man/man9x
missing     /usr/share/man/mann
missing     /usr/share/mime-info
missing     /usr/share/omf
missing     /usr/share/sounds
missing     /usr/share/themes
missing     /usr/share/wayland-sessions
missing     /usr/share/xsessions
missing     /usr/src
missing     /usr/src/debug
missing     /usr/src/kernels
missing     /var/adm
missing     /var/cache/bpf
missing     /var/empty
missing     /var/ftp
missing     /var/games
missing     /var/lib/games
missing     /var/lib/misc
missing     /var/local
missing     /var/nis
missing     /var/opt
missing     /var/preserve
missing     /var/spool/lpd
missing     /var/yp

dnf reinstall filesystem
Last metadata expiration check: 2:48:30 ago on Thu 21 Nov 2024 07:02:31 PM CET.
Installed package filesystem-3.16-2.el9.x86_64 (from baseos) not available.
Error: No packages marked for reinstall.

wget https://dl.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/f/filesystem-3.16-2.el9.x86_64.rpm
dnf reinstall filesystem-3.16-2.el9.x86_64.rpm
Last metadata expiration check: 2:51:11 ago on Thu 21 Nov 2024 07:02:31 PM CET.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                    Architecture                           Version                                     Repository                                    Size
==================================================================================================================================================================================
Reinstalling:
 filesystem                                 x86_64                                 3.16-2.el9                                  @commandline                                 1.1 M

Transaction Summary
==================================================================================================================================================================================

Total size: 1.1 M
Installed size: 106
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-2.el9.x86_64                                                                                                                             1/1
  Preparing        :                                                                                                                                                          1/1
  Reinstalling     : filesystem-3.16-2.el9.x86_64                                                                                                                             1/2
Error unpacking rpm package filesystem-3.16-2.el9.x86_64
  Verifying        : filesystem-3.16-2.el9.x86_64                                                                                                                             1/2
  Verifying        : filesystem-3.16-2.el9.x86_64                                                                                                                             2/2

Failed:
  filesystem-3.16-2.el9.x86_64                                                            filesystem-3.16-2.el9.x86_64

Error: Transaction failed

wget https://mirror.init7.net/rockylinux/9.5/BaseOS/x86_64/os/Packages/f/filesystem-3.16-5.el9.x86_64.rpm
dnf install filesystem-3.16-5.el9.x86_64.rpm
Last metadata expiration check: 2:52:42 ago on Thu 21 Nov 2024 07:02:31 PM CET.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                    Architecture                           Version                                     Repository                                    Size
==================================================================================================================================================================================
Upgrading:
 filesystem                                 x86_64                                 3.16-5.el9                                  @commandline                                 1.1 M

Transaction Summary
==================================================================================================================================================================================
Upgrade  1 Package

Total size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-5.el9.x86_64                                                                                                                             1/1
  Preparing        :                                                                                                                                                          1/1
  Upgrading        : filesystem-3.16-5.el9.x86_64                                                                                                                             1/2
Error unpacking rpm package filesystem-3.16-5.el9.x86_64
  Verifying        : filesystem-3.16-5.el9.x86_64                                                                                                                             1/2
  Verifying        : filesystem-3.16-2.el9.x86_64                                                                                                                             2/2

Failed:
  filesystem-3.16-2.el9.x86_64                                                            filesystem-3.16-5.el9.x86_64

Error: Transaction failed

looks like i am stuck

7

Regarding the list of paths it says are “missing”, can you check if they really are missing? If you have more than one machine, can you check at least one of the other machines.

A quick check on my Rocky 9.4 indicates those paths do exist.

8

Yes they are really missing on systems with the issue:

ll /usr/share/xsessions
ls: cannot access '/usr/share/xsessions': No such file or directory

On systems where the filesystem package is not a problem the same command shows this:

ll /usr/share/xsessions
total 0
9

OK, so it sounds like those systems are broken in some way, any idea how those directories got deleted??

10

I can only imagine that it happen during the ansible update playbook when i stopped it in the middle.

11 
rpm -Uvh --force filesystem-3.16-2.el9.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:filesystem-3.16-2.el9            ################################# [ 50%]
error: unpacking of archive failed on file /home: cpio: chmod failed - Device or resource busy
error: filesystem-3.16-2.el9.x86_64: install failed
error: filesystem-3.16-2.el9.x86_64: erase skipped

My /home is redirected to a nfs server with autofs. Looks like that is a problem. After i stopped autofs i could reinstall the package and then

 dnf update
Last metadata expiration check: 1:39:55 ago on Fri 22 Nov 2024 03:33:51 PM CET.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                     Architecture                            Version                                        Repository                               Size
==================================================================================================================================================================================
Upgrading:
 filesystem                                  x86_64                                  3.16-5.el9                                     baseos                                  1.1 M

Transaction Summary
==================================================================================================================================================================================
Upgrade  1 Package

Total download size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
filesystem-3.16-5.el9.x86_64.rpm                                                                                                                   18 MB/s | 1.1 MB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                             4.8 MB/s | 1.1 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-5.el9.x86_64                                                                                                                             1/1
  Preparing        :                                                                                                                                                          1/1
  Upgrading        : filesystem-3.16-5.el9.x86_64                                                                                                                             1/2
  Cleanup          : filesystem-3.16-2.el9.x86_64                                                                                                                             2/2
  Running scriptlet: filesystem-3.16-5.el9.x86_64                                                                                                                             2/2
  Running scriptlet: filesystem-3.16-2.el9.x86_64                                                                                                                             2/2
  Verifying        : filesystem-3.16-5.el9.x86_64                                                                                                                             1/2
  Verifying        : filesystem-3.16-2.el9.x86_64                                                                                                                             2/2

Upgraded:
  filesystem-3.16-5.el9.x86_64

Complete!

Now i can go through all my system to check that :smiley:

12

It looks like ‘/home’ is one of the directories it tries to create when it’s first run, it then thinks your home is wrong and tries to fix it, using chmod. I’m guessing if verify had worked, it would not try to fix /home.