Doing a yum update with 5.14.0-70.13.1.el9_0.x86_64 attempts to install the new version 5.14.0-162.18.1.el9x86_64 and on reboot the boot breaks.
even making
grub2-mkconfig -o /boot/grub/grub.cfg
is still broken
This is on an AWS EC2.
Any ideas?
Doing a yum update with 5.14.0-70.13.1.el9_0.x86_64 attempts to install the new version 5.14.0-162.18.1.el9x86_64 and on reboot the boot breaks.
even making
grub2-mkconfig -o /boot/grub/grub.cfg
is still broken
This is on an AWS EC2.
Any ideas?
Need more info:
How far does the new kernel boot get?
Do you have multiple drives mounted via fstab?
Do you have an nvme / sata disk combination?
After rebooting from a failed boot what is the output of this command:
sudo jourmalctl -b -1 | tail -n 20
Note that yum is depricated and actually calls dnf to process the command. The new update command is:
sudo dnf upgrade
This is an official ami:
Rocky Linux 9 (Official) - x86_64 so no nvme / sata disk combination
in the boot logs goes as far as:
4.782856] systemd[1]: Starting Generate network units from Kernel command line...
[ 4.788702] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 4.800614] systemd[1]: Repartition Root Disk was skipped because all trigger condition checks failed.
[ 4.805827] systemd[1]: Starting Coldplug All udev Devices...
[ 4.812840] systemd[1]: Started Journal Service.
[ 4.839927] Adding 2072572k swap on /swapfile. Priority:-2 extents:70 across:3498964k SSFS
[ 4.857942] systemd-journald[619]: Received client request to flush runtime journal.
[ 5.159181] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[ 5.194258] parport_pc 00:03: reported by Plug and Play ACPI
[ 5.210818] input: PC Speaker as /devices/platform/pcspkr/input/input5
[ 5.560773] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[ 5.594576] ppdev: user-space parallel port driver
[ 8.022581] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
2023/03/10 10:37:42Z: Amazon SSM Agent v3.2.532.0 is running
2023/03/10 10:37:42Z: OsProductName: Rocky Linux
2023/03/10 10:37:42Z: OsVersion: 9.0
Rocky Linux 9.0 (Blue Onyx)
Kernel 5.14.0-70.26.1.el9_0.x86_64 on an x86_64
Activate the web console with: systemctl enable --now cockpit.socket
ip-10-245-0-210 login: [ 36.804218] systemd-rc-local-generator[1101]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 37.857573] systemd-rc-local-generator[1138]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 40.740484] systemd-rc-local-generator[1172]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 40.958938] systemd-rc-local-generator[1206]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 41.222629] systemd-rc-local-generator[1242]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 41.434038] systemd-rc-local-generator[1272]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 41.783110] systemd-rc-local-generator[1304]: /etc/rc.d/rc.local is not marked executable, skipping.
2023/03/10 10:38:13Z: Amazon SSM Agent v3.2.582.0 is running
2023/03/10 10:38:13Z: OsProductName: Rocky Linux
2023/03/10 10:38:13Z: OsVersion: 9.0
[ 275.595877] systemd[1]: systemd 250-6.el9_0.1 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 275.605021] systemd[1]: Detected virtualization amazon.
[ 275.606845] systemd[1]: Detected architecture x86-64.
[ 275.649470] systemd-rc-local-generator[1704]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 280.268763] systemd[1]: systemd 250-12.el9_1.3 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 280.278102] systemd[1]: Detected virtualization amazon.
[ 280.279910] systemd[1]: Detected architecture x86-64.
[ 280.329747] systemd-rc-local-generator[1773]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 306.151374] systemd-rc-local-generator[4899]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 306.538744] systemd-rc-local-generator[4936]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 306.775426] systemd-rc-local-generator[4972]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 308.627184] systemd-rc-local-generator[5040]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 308.809312] systemd-rc-local-generator[5068]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 309.022371] systemd-rc-local-generator[5098]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 309.200520] systemd-rc-local-generator[5127]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 309.412576] systemd-rc-local-generator[5156]: /etc/rc.d/rc.local is not marked executable, skipping.
in the fstab i have:
UUID=4ccc4686-cd36-4f83-8ad2-e852b63f8766 / xfs defaults 0 0
UUID=B466-AF54 /boot/efi vfat umask=0077,shortname=winnt 0 2
/swapfile none swap defaults 0 0```
i only have:
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1 vfat FAT32 B466-AF54 591.8M 1% /boot/efi
└─nvme0n1p2 xfs 4ccc4686-cd36-4f83-8ad2-e852b63f8766 3.2G 66% /
i use now dnf update but the result was the same
the update logs have some errors
473/473
Running scriptlet: kernel-core-5.14.0-162.18.1.el9_1.x86_64 473/473
/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "
dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.
Running scriptlet: kernel-modules-5.14.0-162.18.1.el9_1.x86_64 473/473
Running scriptlet: kexec-tools-2.0.24-5.el9.x86_64 473/473
Running scriptlet: rocky-logos-90.14-1.el9.x86_64 473/473
Running scriptlet: selinux-policy-targeted-34.1.43-1.el9_1.2.noarch 473/473
Running scriptlet: crypto-policies-scripts-20220815-1.git0fbe86f.el9.noarch 473/473
Running scriptlet: authselect-compat-1.2.5-2.el9_1.x86_64 473/473
Running scriptlet: microcode_ctl-4:20220809-1.el9.noarch 473/473
/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "
dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.
/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "
dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.
Running scriptlet: glibc-all-langpacks-2.34-40.el9_1.1.x86_64 473/473
Running scriptlet: libgcc-11.2.1-9.4.el9.x86_64 473/473
Failed to start jobs: Failed to enqueue some jobs, see logs for details: No such file or directory
Failed to start jobs: Failed to enqueue some jobs, see logs for details: No such file or directory
Upgraded:
NetworkManager-1:1.40.0-1.el9.x86_64 NetworkManager-libnm-1:1.40.0-1.el9.x86_64 NetworkManager-team-1:1.40.0-1.el9.x86_64
NetworkManager-tui-1:1.40.0-1.el9.x86_64 audit-3.0.7-103.el9.x86_64 audit-libs-3.0.7-103.el9.x86_64
authselect-1.2.5-2.el9_1.x86_64 authselect-compat-1.2.5-2.el9_1.x86_64 authselect-libs-1.2.5-2.el9_1.x86_64
bash-5.1.8-6.el9_1.x86_64 checkpolicy-3.4-1.el9.x86_64 chrony-4.2-1.el9.rocky.1.0.x86_64
cloud-init-22.1-7.el9_1.0.1.noarch cockpit-bridge-276.1-1.el9.x86_64 cockpit-system-276.1-1.el9.noarch
cockpit-ws-276.1-1.el9.x86_64 coreutils-8.32-32.el9.x86_64 coreutils-common-8.32-32.el9.x86_64
cronie-1.5.7-8.el9.x86_64 cronie-anacron-1.5.7-8.el9.x86_64 crypto-policies-20220815-1.git0fbe86f.el9.noarch
crypto-policies-scripts-20220815-1.git0fbe86f.el9.noarch cryptsetup-libs-2.4.3-5.el9_1.1.x86_64 curl-7.76.1-19.el9_1.1.x86_64
dbus-1:1.12.20-7.el9_1.x86_64 dbus-broker-28-7.el9.x86_64 dbus-common-1:1.12.20-7.el9_1.noarch
dbus-libs-1:1.12.20-7.el9_1.x86_64 dbus-tools-1:1.12.20-7.el9_1.x86_64 device-mapper-9:1.02.185-3.el9.x86_64
device-mapper-libs-9:1.02.185-3.el9.x86_64 dhcp-client-12:4.4.2-17.b1.el9.x86_64 dhcp-common-12:4.4.2-17.b1.el9.noarch
dnf-4.12.0-4.el9.noarch dnf-data-4.12.0-4.el9.noarch dnf-plugins-core-4.1.0-3.el9.noarch
dracut-057-13.git20220816.el9.x86_64 dracut-config-generic-057-13.git20220816.el9.x86_64 dracut-config-rescue-057-13.git20220816.el9.x86_64
dracut-network-057-13.git20220816.el9.x86_64 dracut-squash-057-13.git20220816.el9.x86_64 e2fsprogs-1.46.5-3.el9.x86_64
e2fsprogs-libs-1.46.5-3.el9.x86_64 elfutils-default-yama-scope-0.187-5.el9.noarch elfutils-libelf-0.187-5.el9.x86_64
elfutils-libs-0.187-5.el9.x86_64 expat-2.4.9-1.el9_1.1.x86_64 file-5.39-10.el9.x86_64
file-libs-5.39-10.el9.x86_64 freetype-2.10.4-9.el9.x86_64 fwupd-1.7.9-1.el9.x86_64
fwupd-plugin-flashrom-1.7.9-1.el9.x86_64 glibc-2.34-40.el9_1.1.x86_64 glibc-all-langpacks-2.34-40.el9_1.1.x86_64
glibc-common-2.34-40.el9_1.1.x86_64 glibc-gconv-extra-2.34-40.el9_1.1.x86_64 glibc-langpack-es-2.34-40.el9_1.1.x86_64
glibc-langpack-fr-2.34-40.el9_1.1.x86_64 glibc-minimal-langpack-2.34-40.el9_1.1.x86_64 gnutls-3.7.6-18.el9_1.x86_64
grub2-common-1:2.06-46.el9_1.3.rocky.0.2.noarch grub2-efi-x64-1:2.06-46.el9_1.3.rocky.0.2.x86_64 grub2-pc-1:2.06-46.el9_1.3.rocky.0.2.x86_64
grub2-pc-modules-1:2.06-46.el9_1.3.rocky.0.2.noarch grub2-tools-1:2.06-46.el9_1.3.rocky.0.2.x86_64 grub2-tools-efi-1:2.06-46.el9_1.3.rocky.0.2.x86_64
grub2-tools-extra-1:2.06-46.el9_1.3.rocky.0.2.x86_64 grub2-tools-minimal-1:2.06-46.el9_1.3.rocky.0.2.x86_64 grubby-8.40-61.el9.x86_64
gzip-1.12-1.el9.x86_64 harfbuzz-2.7.4-8.el9.x86_64 hwdata-0.348-9.5.el9.noarch
inih-49-6.el9.x86_64 initscripts-rename-device-10.11.5-1.el9.x86_64 initscripts-service-10.11.5-1.el9.noarch
iproute-5.18.0-1.el9.x86_64 iproute-tc-5.18.0-1.el9.x86_64 iptables-libs-1.8.8-6.el9_1.x86_64
iputils-20210202-8.el9_1.1.x86_64 irqbalance-2:1.9.0-3.el9.x86_64 jitterentropy-3.4.0-1.el9.x86_64
kernel-tools-5.14.0-162.18.1.el9_1.x86_64 kernel-tools-libs-5.14.0-162.18.1.el9_1.x86_64 kexec-tools-2.0.24-5.el9.x86_64
kpartx-0.8.7-12.el9_1.1.x86_64 krb5-libs-1.19.1-24.el9_1.x86_64 libarchive-3.5.3-3.el9.x86_64
libbasicobjects-0.1.1-53.el9.x86_64 libblkid-2.37.4-9.el9.x86_64 libblockdev-2.25-14.el9.x86_64
libblockdev-crypto-2.25-14.el9.x86_64 libblockdev-fs-2.25-14.el9.x86_64 libblockdev-loop-2.25-14.el9.x86_64
libblockdev-mdraid-2.25-14.el9.x86_64 libblockdev-part-2.25-14.el9.x86_64 libblockdev-swap-2.25-14.el9.x86_64
libblockdev-utils-2.25-14.el9.x86_64 libbpf-2:0.6.0-1.el9.x86_64 libcollection-0.7.0-53.el9.x86_64
libcom_err-1.46.5-3.el9.x86_64 libcurl-7.76.1-19.el9_1.1.x86_64 libdhash-0.5.0-53.el9.x86_64
libdnf-0.67.0-3.el9.x86_64 libestr-0.1.11-4.el9.x86_64 libfdisk-2.37.4-9.el9.x86_64
libgcc-11.3.1-2.1.el9.x86_64 libgcrypt-1.10.0-8.el9_0.x86_64 libgomp-11.3.1-2.1.el9.x86_64
libibverbs-41.0-3.el9.x86_64 libini_config-1.3.1-53.el9.x86_64 libjpeg-turbo-2.0.90-6.el9_1.x86_64
libksba-1.5.1-6.el9_1.x86_64 libldb-2.5.2-1.el9.x86_64 libmount-2.37.4-9.el9.x86_64
libnetfilter_conntrack-1.0.8-5.el9_1.x86_64 libnfsidmap-1:2.5.4-15.el9.x86_64 libnl3-3.7.0-1.el9.x86_64
libnl3-cli-3.7.0-1.el9.x86_64 libpath_utils-0.2.1-53.el9.x86_64 libref_array-0.1.5-53.el9.x86_64
librepo-1.14.2-3.el9.x86_64 libselinux-3.4-3.el9.x86_64 libselinux-utils-3.4-3.el9.x86_64
libsemanage-3.4-2.el9.x86_64 libsepol-3.4-1.1.el9.x86_64 libsmartcols-2.37.4-9.el9.x86_64
libsolv-0.7.22-1.el9.x86_64 libss-1.46.5-3.el9.x86_64 libsss_certmap-2.7.3-4.el9_1.3.x86_64
libsss_idmap-2.7.3-4.el9_1.3.x86_64 libsss_nss_idmap-2.7.3-4.el9_1.3.x86_64 libsss_sudo-2.7.3-4.el9_1.3.x86_64
libstdc++-11.3.1-2.1.el9.x86_64 libtasn1-4.16.0-8.el9_1.x86_64 libtdb-1.4.6-1.el9.x86_64
libteam-1.31-16.el9_1.x86_64 libtevent-0.12.0-0.el9.x86_64 libtirpc-1.3.3-0.el9.x86_64
libusbx-1.0.26-1.el9.x86_64 libuser-0.63-11.el9.x86_64 libuuid-2.37.4-9.el9.x86_64
libxml2-2.9.13-3.el9_1.x86_64 logrotate-3.18.0-7.el9.x86_64 lshw-B.02.19.2-9.el9.x86_64
lua-libs-5.4.4-2.el9_1.x86_64 mdadm-4.2-6.el9.x86_64 microcode_ctl-4:20220809-1.el9.noarch
mokutil-2:0.4.0-9.el9.x86_64 nfs-utils-1:2.5.4-15.el9.x86_64 nginx-1:1.20.1-13.el9.x86_64
nginx-filesystem-1:1.20.1-13.el9.noarch oddjob-0.34.7-6.el9.x86_64 oddjob-mkhomedir-0.34.7-6.el9.x86_64
openldap-2.6.2-3.el9.x86_64 openldap-compat-2.6.2-3.el9.x86_64 openssh-8.7p1-24.el9_1.x86_64
openssh-clients-8.7p1-24.el9_1.x86_64 openssh-server-8.7p1-24.el9_1.x86_64 openssl-1:3.0.1-47.el9_1.x86_64
openssl-libs-1:3.0.1-47.el9_1.x86_64 pam-1.5.1-12.el9.x86_64 parted-3.5-2.el9.x86_64
pcre2-10.40-2.el9.x86_64 pcre2-syntax-10.40-2.el9.noarch policycoreutils-3.4-4.el9.x86_64
policycoreutils-python-utils-3.4-4.el9.noarch procps-ng-3.3.17-8.el9.x86_64 python-unversioned-command-3.9.14-1.el9_1.2.noarch
python3-3.9.14-1.el9_1.2.x86_64 python3-audit-3.0.7-103.el9.x86_64 python3-dnf-4.12.0-4.el9.noarch
python3-dnf-plugins-core-4.1.0-3.el9.noarch python3-gobject-base-3.40.1-6.el9.x86_64 python3-hawkey-0.67.0-3.el9.x86_64
python3-libdnf-0.67.0-3.el9.x86_64 python3-libs-3.9.14-1.el9_1.2.x86_64 python3-libselinux-3.4-3.el9.x86_64
python3-libsemanage-3.4-2.el9.x86_64 python3-libxml2-2.9.13-3.el9_1.x86_64 python3-policycoreutils-3.4-4.el9.noarch
python3-rpm-4.16.1.3-19.el9_1.x86_64 python3-setools-4.4.0-5.el9.x86_64 python3-setuptools-53.0.0-10.el9_1.1.noarch
python3-setuptools-wheel-53.0.0-10.el9_1.1.noarch qemu-guest-agent-17:7.0.0-13.el9_1.2.x86_64 rng-tools-6.15-1.el9.x86_64
rocky-gpg-keys-9.1-1.11.el9.noarch rocky-logos-90.14-1.el9.x86_64 rocky-logos-httpd-90.14-1.el9.noarch
rocky-release-9.1-1.11.el9.noarch rocky-repos-9.1-1.11.el9.noarch rpcbind-1.2.6-5.el9.x86_64
rpm-4.16.1.3-19.el9_1.x86_64 rpm-build-libs-4.16.1.3-19.el9_1.x86_64 rpm-libs-4.16.1.3-19.el9_1.x86_64
rpm-plugin-audit-4.16.1.3-19.el9_1.x86_64 rpm-plugin-selinux-4.16.1.3-19.el9_1.x86_64 rpm-plugin-systemd-inhibit-4.16.1.3-19.el9_1.x86_64
rpm-sign-libs-4.16.1.3-19.el9_1.x86_64 rsync-3.2.3-18.el9.x86_64 rsyslog-8.2102.0-105.el9.x86_64
rsyslog-logrotate-8.2102.0-105.el9.x86_64 selinux-policy-34.1.43-1.el9_1.2.noarch selinux-policy-targeted-34.1.43-1.el9_1.2.noarch
setroubleshoot-server-3.3.28-4.el9.x86_64 setup-2.13.7-7.el9.noarch sg3_utils-1.47-9.el9.x86_64
sg3_utils-libs-1.47-9.el9.x86_64 shadow-utils-2:4.9-5.el9.x86_64 shim-x64-15.6-1.el9.rocky.0.1.x86_64
sos-4.3-5.el9_1.noarch sqlite-libs-3.34.1-6.el9_1.x86_64 sssd-client-2.7.3-4.el9_1.3.x86_64
sssd-common-2.7.3-4.el9_1.3.x86_64 sssd-kcm-2.7.3-4.el9_1.3.x86_64 sssd-nfs-idmap-2.7.3-4.el9_1.3.x86_64
sudo-1.9.5p2-7.el9_1.1.x86_64 systemd-250-12.el9_1.3.x86_64 systemd-libs-250-12.el9_1.3.x86_64
systemd-pam-250-12.el9_1.3.x86_64 systemd-rpm-macros-250-12.el9_1.3.noarch systemd-udev-250-12.el9_1.3.x86_64
tar-2:1.34-6.el9_1.x86_64 teamd-1.31-16.el9_1.x86_64 tpm2-tss-3.0.3-8.el9.x86_64
tzdata-2022g-1.el9_1.noarch util-linux-2.37.4-9.el9.x86_64 util-linux-core-2.37.4-9.el9.x86_64
vim-minimal-2:8.2.2637-20.el9_1.x86_64 webkit2gtk3-jsc-2.36.7-1.el9_1.2.x86_64 which-2.21-28.el9.x86_64
yum-4.12.0-4.el9.noarch yum-utils-4.1.0-3.el9.noarch zlib-1.2.11-35.el9_1.x86_64
Installed:
kernel-5.14.0-162.18.1.el9_1.x86_64 kernel-core-5.14.0-162.18.1.el9_1.x86_64 kernel-modules-5.14.0-162.18.1.el9_1.x86_64 nginx-core-1:1.20.1-13.el9.x86_64 python3-gobject-base-noarch-3.40.1-6.el9.noarch
Complete!
After rebooting from a failed boot I can no longer access the instance.
It turns out to be a resource error.
The machine was a t3.micro. I noticed that Rocky 8 forces you to boot with at least a t3.small but Rocky 9 lets you boot with a t3.micro.
And everything was fine until this update where it seems that it doesn’t work with t3.micro anymore.
I just stopped the machine and changed to t3.small and it started perfectly without errors.