Yum update in Rocky 9 upgrades to Rocky 9.1 and breaks startup

Doing a yum update with 5.14.0-70.13.1.el9_0.x86_64 attempts to install the new version 5.14.0-162.18.1.el9x86_64 and on reboot the boot breaks.
even making

grub2-mkconfig -o /boot/grub/grub.cfg

is still broken
This is on an AWS EC2.

Any ideas?

Need more info:
How far does the new kernel boot get?
Do you have multiple drives mounted via fstab?
Do you have an nvme / sata disk combination?
After rebooting from a failed boot what is the output of this command:

sudo jourmalctl -b -1 | tail -n 20

Note that yum is depricated and actually calls dnf to process the command. The new update command is:

sudo dnf upgrade

This is an official ami:
Rocky Linux 9 (Official) - x86_64 so no nvme / sata disk combination

in the boot logs goes as far as:

    4.782856] systemd[1]: Starting Generate network units from Kernel command line...
[    4.788702] systemd[1]: Starting Remount Root and Kernel File Systems...
[    4.800614] systemd[1]: Repartition Root Disk was skipped because all trigger condition checks failed.
[    4.805827] systemd[1]: Starting Coldplug All udev Devices...
[    4.812840] systemd[1]: Started Journal Service.
[    4.839927] Adding 2072572k swap on /swapfile.  Priority:-2 extents:70 across:3498964k SSFS
[    4.857942] systemd-journald[619]: Received client request to flush runtime journal.
[    5.159181] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[    5.194258] parport_pc 00:03: reported by Plug and Play ACPI
[    5.210818] input: PC Speaker as /devices/platform/pcspkr/input/input5
[    5.560773] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[    5.594576] ppdev: user-space parallel port driver
[    8.022581] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
2023/03/10 10:37:42Z: Amazon SSM Agent v3.2.532.0 is running
2023/03/10 10:37:42Z: OsProductName: Rocky Linux
2023/03/10 10:37:42Z: OsVersion: 9.0

Rocky Linux 9.0 (Blue Onyx)
Kernel 5.14.0-70.26.1.el9_0.x86_64 on an x86_64

Activate the web console with: systemctl enable --now cockpit.socket

ip-10-245-0-210 login: [   36.804218] systemd-rc-local-generator[1101]: /etc/rc.d/rc.local is not marked executable, skipping.
[   37.857573] systemd-rc-local-generator[1138]: /etc/rc.d/rc.local is not marked executable, skipping.
[   40.740484] systemd-rc-local-generator[1172]: /etc/rc.d/rc.local is not marked executable, skipping.
[   40.958938] systemd-rc-local-generator[1206]: /etc/rc.d/rc.local is not marked executable, skipping.
[   41.222629] systemd-rc-local-generator[1242]: /etc/rc.d/rc.local is not marked executable, skipping.
[   41.434038] systemd-rc-local-generator[1272]: /etc/rc.d/rc.local is not marked executable, skipping.
[   41.783110] systemd-rc-local-generator[1304]: /etc/rc.d/rc.local is not marked executable, skipping.
2023/03/10 10:38:13Z: Amazon SSM Agent v3.2.582.0 is running
2023/03/10 10:38:13Z: OsProductName: Rocky Linux
2023/03/10 10:38:13Z: OsVersion: 9.0
[  275.595877] systemd[1]: systemd 250-6.el9_0.1 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  275.605021] systemd[1]: Detected virtualization amazon.
[  275.606845] systemd[1]: Detected architecture x86-64.
[  275.649470] systemd-rc-local-generator[1704]: /etc/rc.d/rc.local is not marked executable, skipping.
[  280.268763] systemd[1]: systemd 250-12.el9_1.3 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[  280.278102] systemd[1]: Detected virtualization amazon.
[  280.279910] systemd[1]: Detected architecture x86-64.
[  280.329747] systemd-rc-local-generator[1773]: /etc/rc.d/rc.local is not marked executable, skipping.
[  306.151374] systemd-rc-local-generator[4899]: /etc/rc.d/rc.local is not marked executable, skipping.
[  306.538744] systemd-rc-local-generator[4936]: /etc/rc.d/rc.local is not marked executable, skipping.
[  306.775426] systemd-rc-local-generator[4972]: /etc/rc.d/rc.local is not marked executable, skipping.
[  308.627184] systemd-rc-local-generator[5040]: /etc/rc.d/rc.local is not marked executable, skipping.
[  308.809312] systemd-rc-local-generator[5068]: /etc/rc.d/rc.local is not marked executable, skipping.
[  309.022371] systemd-rc-local-generator[5098]: /etc/rc.d/rc.local is not marked executable, skipping.
[  309.200520] systemd-rc-local-generator[5127]: /etc/rc.d/rc.local is not marked executable, skipping.
[  309.412576] systemd-rc-local-generator[5156]: /etc/rc.d/rc.local is not marked executable, skipping.

in the fstab i have:

UUID=4ccc4686-cd36-4f83-8ad2-e852b63f8766 /                       xfs     defaults        0 0
UUID=B466-AF54          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
/swapfile none swap defaults 0 0```

i only have:

NAME        FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1 vfat   FAT32       B466-AF54                             591.8M     1% /boot/efi
└─nvme0n1p2 xfs                4ccc4686-cd36-4f83-8ad2-e852b63f8766    3.2G    66% /

i use now dnf update but the result was the same
the update logs have some errors


                                                                                                                                                473/473
  Running scriptlet: kernel-core-5.14.0-162.18.1.el9_1.x86_64                                                                                                                                             473/473
/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "

dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.


  Running scriptlet: kernel-modules-5.14.0-162.18.1.el9_1.x86_64                                                                                                                                          473/473
  Running scriptlet: kexec-tools-2.0.24-5.el9.x86_64                                                                                                                                                      473/473
  Running scriptlet: rocky-logos-90.14-1.el9.x86_64                                                                                                                                                       473/473
  Running scriptlet: selinux-policy-targeted-34.1.43-1.el9_1.2.noarch                                                                                                                                     473/473
  Running scriptlet: crypto-policies-scripts-20220815-1.git0fbe86f.el9.noarch                                                                                                                             473/473
  Running scriptlet: authselect-compat-1.2.5-2.el9_1.x86_64                                                                                                                                               473/473
  Running scriptlet: microcode_ctl-4:20220809-1.el9.noarch                                                                                                                                                473/473
/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "

dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.

/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "

dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.


  Running scriptlet: glibc-all-langpacks-2.34-40.el9_1.1.x86_64                                                                                                                                           473/473
  Running scriptlet: libgcc-11.2.1-9.4.el9.x86_64                                                                                                                                                         473/473
Failed to start jobs: Failed to enqueue some jobs, see logs for details: No such file or directory

Failed to start jobs: Failed to enqueue some jobs, see logs for details: No such file or directory


Upgraded:
  NetworkManager-1:1.40.0-1.el9.x86_64                                    NetworkManager-libnm-1:1.40.0-1.el9.x86_64                            NetworkManager-team-1:1.40.0-1.el9.x86_64
  NetworkManager-tui-1:1.40.0-1.el9.x86_64                                audit-3.0.7-103.el9.x86_64                                            audit-libs-3.0.7-103.el9.x86_64
  authselect-1.2.5-2.el9_1.x86_64                                         authselect-compat-1.2.5-2.el9_1.x86_64                                authselect-libs-1.2.5-2.el9_1.x86_64
  bash-5.1.8-6.el9_1.x86_64                                               checkpolicy-3.4-1.el9.x86_64                                          chrony-4.2-1.el9.rocky.1.0.x86_64
  cloud-init-22.1-7.el9_1.0.1.noarch                                      cockpit-bridge-276.1-1.el9.x86_64                                     cockpit-system-276.1-1.el9.noarch
  cockpit-ws-276.1-1.el9.x86_64                                           coreutils-8.32-32.el9.x86_64                                          coreutils-common-8.32-32.el9.x86_64
  cronie-1.5.7-8.el9.x86_64                                               cronie-anacron-1.5.7-8.el9.x86_64                                     crypto-policies-20220815-1.git0fbe86f.el9.noarch
  crypto-policies-scripts-20220815-1.git0fbe86f.el9.noarch                cryptsetup-libs-2.4.3-5.el9_1.1.x86_64                                curl-7.76.1-19.el9_1.1.x86_64
  dbus-1:1.12.20-7.el9_1.x86_64                                           dbus-broker-28-7.el9.x86_64                                           dbus-common-1:1.12.20-7.el9_1.noarch
  dbus-libs-1:1.12.20-7.el9_1.x86_64                                      dbus-tools-1:1.12.20-7.el9_1.x86_64                                   device-mapper-9:1.02.185-3.el9.x86_64
  device-mapper-libs-9:1.02.185-3.el9.x86_64                              dhcp-client-12:4.4.2-17.b1.el9.x86_64                                 dhcp-common-12:4.4.2-17.b1.el9.noarch
  dnf-4.12.0-4.el9.noarch                                                 dnf-data-4.12.0-4.el9.noarch                                          dnf-plugins-core-4.1.0-3.el9.noarch
  dracut-057-13.git20220816.el9.x86_64                                    dracut-config-generic-057-13.git20220816.el9.x86_64                   dracut-config-rescue-057-13.git20220816.el9.x86_64
  dracut-network-057-13.git20220816.el9.x86_64                            dracut-squash-057-13.git20220816.el9.x86_64                           e2fsprogs-1.46.5-3.el9.x86_64
  e2fsprogs-libs-1.46.5-3.el9.x86_64                                      elfutils-default-yama-scope-0.187-5.el9.noarch                        elfutils-libelf-0.187-5.el9.x86_64
  elfutils-libs-0.187-5.el9.x86_64                                        expat-2.4.9-1.el9_1.1.x86_64                                          file-5.39-10.el9.x86_64
  file-libs-5.39-10.el9.x86_64                                            freetype-2.10.4-9.el9.x86_64                                          fwupd-1.7.9-1.el9.x86_64
  fwupd-plugin-flashrom-1.7.9-1.el9.x86_64                                glibc-2.34-40.el9_1.1.x86_64                                          glibc-all-langpacks-2.34-40.el9_1.1.x86_64
  glibc-common-2.34-40.el9_1.1.x86_64                                     glibc-gconv-extra-2.34-40.el9_1.1.x86_64                              glibc-langpack-es-2.34-40.el9_1.1.x86_64
  glibc-langpack-fr-2.34-40.el9_1.1.x86_64                                glibc-minimal-langpack-2.34-40.el9_1.1.x86_64                         gnutls-3.7.6-18.el9_1.x86_64
  grub2-common-1:2.06-46.el9_1.3.rocky.0.2.noarch                         grub2-efi-x64-1:2.06-46.el9_1.3.rocky.0.2.x86_64                      grub2-pc-1:2.06-46.el9_1.3.rocky.0.2.x86_64
  grub2-pc-modules-1:2.06-46.el9_1.3.rocky.0.2.noarch                     grub2-tools-1:2.06-46.el9_1.3.rocky.0.2.x86_64                        grub2-tools-efi-1:2.06-46.el9_1.3.rocky.0.2.x86_64
  grub2-tools-extra-1:2.06-46.el9_1.3.rocky.0.2.x86_64                    grub2-tools-minimal-1:2.06-46.el9_1.3.rocky.0.2.x86_64                grubby-8.40-61.el9.x86_64
  gzip-1.12-1.el9.x86_64                                                  harfbuzz-2.7.4-8.el9.x86_64                                           hwdata-0.348-9.5.el9.noarch
  inih-49-6.el9.x86_64                                                    initscripts-rename-device-10.11.5-1.el9.x86_64                        initscripts-service-10.11.5-1.el9.noarch
  iproute-5.18.0-1.el9.x86_64                                             iproute-tc-5.18.0-1.el9.x86_64                                        iptables-libs-1.8.8-6.el9_1.x86_64
  iputils-20210202-8.el9_1.1.x86_64                                       irqbalance-2:1.9.0-3.el9.x86_64                                       jitterentropy-3.4.0-1.el9.x86_64
  kernel-tools-5.14.0-162.18.1.el9_1.x86_64                               kernel-tools-libs-5.14.0-162.18.1.el9_1.x86_64                        kexec-tools-2.0.24-5.el9.x86_64
  kpartx-0.8.7-12.el9_1.1.x86_64                                          krb5-libs-1.19.1-24.el9_1.x86_64                                      libarchive-3.5.3-3.el9.x86_64
  libbasicobjects-0.1.1-53.el9.x86_64                                     libblkid-2.37.4-9.el9.x86_64                                          libblockdev-2.25-14.el9.x86_64
  libblockdev-crypto-2.25-14.el9.x86_64                                   libblockdev-fs-2.25-14.el9.x86_64                                     libblockdev-loop-2.25-14.el9.x86_64
  libblockdev-mdraid-2.25-14.el9.x86_64                                   libblockdev-part-2.25-14.el9.x86_64                                   libblockdev-swap-2.25-14.el9.x86_64
  libblockdev-utils-2.25-14.el9.x86_64                                    libbpf-2:0.6.0-1.el9.x86_64                                           libcollection-0.7.0-53.el9.x86_64
  libcom_err-1.46.5-3.el9.x86_64                                          libcurl-7.76.1-19.el9_1.1.x86_64                                      libdhash-0.5.0-53.el9.x86_64
  libdnf-0.67.0-3.el9.x86_64                                              libestr-0.1.11-4.el9.x86_64                                           libfdisk-2.37.4-9.el9.x86_64
  libgcc-11.3.1-2.1.el9.x86_64                                            libgcrypt-1.10.0-8.el9_0.x86_64                                       libgomp-11.3.1-2.1.el9.x86_64
  libibverbs-41.0-3.el9.x86_64                                            libini_config-1.3.1-53.el9.x86_64                                     libjpeg-turbo-2.0.90-6.el9_1.x86_64
  libksba-1.5.1-6.el9_1.x86_64                                            libldb-2.5.2-1.el9.x86_64                                             libmount-2.37.4-9.el9.x86_64
  libnetfilter_conntrack-1.0.8-5.el9_1.x86_64                             libnfsidmap-1:2.5.4-15.el9.x86_64                                     libnl3-3.7.0-1.el9.x86_64
  libnl3-cli-3.7.0-1.el9.x86_64                                           libpath_utils-0.2.1-53.el9.x86_64                                     libref_array-0.1.5-53.el9.x86_64
  librepo-1.14.2-3.el9.x86_64                                             libselinux-3.4-3.el9.x86_64                                           libselinux-utils-3.4-3.el9.x86_64
  libsemanage-3.4-2.el9.x86_64                                            libsepol-3.4-1.1.el9.x86_64                                           libsmartcols-2.37.4-9.el9.x86_64
  libsolv-0.7.22-1.el9.x86_64                                             libss-1.46.5-3.el9.x86_64                                             libsss_certmap-2.7.3-4.el9_1.3.x86_64
  libsss_idmap-2.7.3-4.el9_1.3.x86_64                                     libsss_nss_idmap-2.7.3-4.el9_1.3.x86_64                               libsss_sudo-2.7.3-4.el9_1.3.x86_64
  libstdc++-11.3.1-2.1.el9.x86_64                                         libtasn1-4.16.0-8.el9_1.x86_64                                        libtdb-1.4.6-1.el9.x86_64
  libteam-1.31-16.el9_1.x86_64                                            libtevent-0.12.0-0.el9.x86_64                                         libtirpc-1.3.3-0.el9.x86_64
  libusbx-1.0.26-1.el9.x86_64                                             libuser-0.63-11.el9.x86_64                                            libuuid-2.37.4-9.el9.x86_64
  libxml2-2.9.13-3.el9_1.x86_64                                           logrotate-3.18.0-7.el9.x86_64                                         lshw-B.02.19.2-9.el9.x86_64
  lua-libs-5.4.4-2.el9_1.x86_64                                           mdadm-4.2-6.el9.x86_64                                                microcode_ctl-4:20220809-1.el9.noarch
  mokutil-2:0.4.0-9.el9.x86_64                                            nfs-utils-1:2.5.4-15.el9.x86_64                                       nginx-1:1.20.1-13.el9.x86_64
  nginx-filesystem-1:1.20.1-13.el9.noarch                                 oddjob-0.34.7-6.el9.x86_64                                            oddjob-mkhomedir-0.34.7-6.el9.x86_64
  openldap-2.6.2-3.el9.x86_64                                             openldap-compat-2.6.2-3.el9.x86_64                                    openssh-8.7p1-24.el9_1.x86_64
  openssh-clients-8.7p1-24.el9_1.x86_64                                   openssh-server-8.7p1-24.el9_1.x86_64                                  openssl-1:3.0.1-47.el9_1.x86_64
  openssl-libs-1:3.0.1-47.el9_1.x86_64                                    pam-1.5.1-12.el9.x86_64                                               parted-3.5-2.el9.x86_64
  pcre2-10.40-2.el9.x86_64                                                pcre2-syntax-10.40-2.el9.noarch                                       policycoreutils-3.4-4.el9.x86_64
  policycoreutils-python-utils-3.4-4.el9.noarch                           procps-ng-3.3.17-8.el9.x86_64                                         python-unversioned-command-3.9.14-1.el9_1.2.noarch
  python3-3.9.14-1.el9_1.2.x86_64                                         python3-audit-3.0.7-103.el9.x86_64                                    python3-dnf-4.12.0-4.el9.noarch
  python3-dnf-plugins-core-4.1.0-3.el9.noarch                             python3-gobject-base-3.40.1-6.el9.x86_64                              python3-hawkey-0.67.0-3.el9.x86_64
  python3-libdnf-0.67.0-3.el9.x86_64                                      python3-libs-3.9.14-1.el9_1.2.x86_64                                  python3-libselinux-3.4-3.el9.x86_64
  python3-libsemanage-3.4-2.el9.x86_64                                    python3-libxml2-2.9.13-3.el9_1.x86_64                                 python3-policycoreutils-3.4-4.el9.noarch
  python3-rpm-4.16.1.3-19.el9_1.x86_64                                    python3-setools-4.4.0-5.el9.x86_64                                    python3-setuptools-53.0.0-10.el9_1.1.noarch
  python3-setuptools-wheel-53.0.0-10.el9_1.1.noarch                       qemu-guest-agent-17:7.0.0-13.el9_1.2.x86_64                           rng-tools-6.15-1.el9.x86_64
  rocky-gpg-keys-9.1-1.11.el9.noarch                                      rocky-logos-90.14-1.el9.x86_64                                        rocky-logos-httpd-90.14-1.el9.noarch
  rocky-release-9.1-1.11.el9.noarch                                       rocky-repos-9.1-1.11.el9.noarch                                       rpcbind-1.2.6-5.el9.x86_64
  rpm-4.16.1.3-19.el9_1.x86_64                                            rpm-build-libs-4.16.1.3-19.el9_1.x86_64                               rpm-libs-4.16.1.3-19.el9_1.x86_64
  rpm-plugin-audit-4.16.1.3-19.el9_1.x86_64                               rpm-plugin-selinux-4.16.1.3-19.el9_1.x86_64                           rpm-plugin-systemd-inhibit-4.16.1.3-19.el9_1.x86_64
  rpm-sign-libs-4.16.1.3-19.el9_1.x86_64                                  rsync-3.2.3-18.el9.x86_64                                             rsyslog-8.2102.0-105.el9.x86_64
  rsyslog-logrotate-8.2102.0-105.el9.x86_64                               selinux-policy-34.1.43-1.el9_1.2.noarch                               selinux-policy-targeted-34.1.43-1.el9_1.2.noarch
  setroubleshoot-server-3.3.28-4.el9.x86_64                               setup-2.13.7-7.el9.noarch                                             sg3_utils-1.47-9.el9.x86_64
  sg3_utils-libs-1.47-9.el9.x86_64                                        shadow-utils-2:4.9-5.el9.x86_64                                       shim-x64-15.6-1.el9.rocky.0.1.x86_64
  sos-4.3-5.el9_1.noarch                                                  sqlite-libs-3.34.1-6.el9_1.x86_64                                     sssd-client-2.7.3-4.el9_1.3.x86_64
  sssd-common-2.7.3-4.el9_1.3.x86_64                                      sssd-kcm-2.7.3-4.el9_1.3.x86_64                                       sssd-nfs-idmap-2.7.3-4.el9_1.3.x86_64
  sudo-1.9.5p2-7.el9_1.1.x86_64                                           systemd-250-12.el9_1.3.x86_64                                         systemd-libs-250-12.el9_1.3.x86_64
  systemd-pam-250-12.el9_1.3.x86_64                                       systemd-rpm-macros-250-12.el9_1.3.noarch                              systemd-udev-250-12.el9_1.3.x86_64
  tar-2:1.34-6.el9_1.x86_64                                               teamd-1.31-16.el9_1.x86_64                                            tpm2-tss-3.0.3-8.el9.x86_64
  tzdata-2022g-1.el9_1.noarch                                             util-linux-2.37.4-9.el9.x86_64                                        util-linux-core-2.37.4-9.el9.x86_64
  vim-minimal-2:8.2.2637-20.el9_1.x86_64                                  webkit2gtk3-jsc-2.36.7-1.el9_1.2.x86_64                               which-2.21-28.el9.x86_64
  yum-4.12.0-4.el9.noarch                                                 yum-utils-4.1.0-3.el9.noarch                                          zlib-1.2.11-35.el9_1.x86_64
Installed:
  kernel-5.14.0-162.18.1.el9_1.x86_64  kernel-core-5.14.0-162.18.1.el9_1.x86_64  kernel-modules-5.14.0-162.18.1.el9_1.x86_64  nginx-core-1:1.20.1-13.el9.x86_64  python3-gobject-base-noarch-3.40.1-6.el9.noarch

Complete!

After rebooting from a failed boot I can no longer access the instance.

It turns out to be a resource error.
The machine was a t3.micro. I noticed that Rocky 8 forces you to boot with at least a t3.small but Rocky 9 lets you boot with a t3.micro.
And everything was fine until this update where it seems that it doesn’t work with t3.micro anymore.
I just stopped the machine and changed to t3.small and it started perfectly without errors.