Rocky 9.4 AWS ami not booting after kernel update

Hello

We have noticed that on our AWS environments since the 9.4 ami was published our servers do not boot up after a kernel/dnf update which is part of our provisioning

AMI id: ami-02d6d64456f57b99d
AMI Name: Rocky-9-EC2-Base-9.4-20240509.0.x86_64
Region: eu-west-2
Disk type: gp3
Instance type: t3.medium

The serial console snippet:

[   15.746077] nvme nvme0: pci function 0000:00:04.0
[   15.746470] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[   15.770394] nvme nvme0: 2/0/0 default/read/poll queues
[   15.775054]  nvme0n1: p1 p2 p3 p4
[  143.289253] dracut-initqueue[429]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[  143.336391] dracut-initqueue[429]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fmapper\x2floop0p4.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[  143.436561] dracut-initqueue[429]:     [ -e "/dev/mapper/loop0p4" ]
[  143.465303] dracut-initqueue[429]: fi"
[  143.482303] dracut-initqueue[429]: Warning: dracut-initqueue: starting timeout scripts
[  143.845722] dracut-initqueue[429]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[  143.892666] dracut-initqueue[429]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fmapper\x2floop0p4.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[  143.992287] dracut-initqueue[429]:     [ -e "/dev/mapper/loop0p4" ]
[  144.022260] dracut-initqueue[429]: fi"
[  144.038265] dracut-initqueue[429]: Warning: dracut-initqueue: starting timeout scripts
[  144.398278] dracut-initqueue[429]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[  144.444311] dracut-initqueue[429]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fmapper\x2floop0p4.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[  144.541977] dracut-initqueue[429]:     [ -e "/dev/mapper/loop0p4" ]

<snip - same message repeated>

[  209.332749] dracut-initqueue[429]: Warning: dracut-initqueue: starting timeout scripts
[  209.332777] dracut-initqueue[429]: Warning: Could not boot.
         Starting Dracut Emergency Shell...
Warning: /dev/mapper/loop0p4 does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


dracut:/# 

I am new to the forum so will try figure out how to attach the whole console log

Has anyone experienced anything similar?

Regards
Jared

2 Likes

The dnf update:

[root@ip-10-124-0-18 ~]# dnf update
Rocky Linux 9 - BaseOS                                                                                                                                                               1.4 MB/s | 2.2 MB     00:01    
Rocky Linux 9 - AppStream                                                                                                                                                            3.6 MB/s | 7.9 MB     00:02    
Rocky Linux 9 - Extras                                                                                                                                                                15 kB/s |  15 kB     00:00    
Dependencies resolved.
=====================================================================================================================================================================================================================
 Package                                                  Architecture                                Version                                                      Repository                                   Size
=====================================================================================================================================================================================================================
Installing:
 kernel                                                   x86_64                                      5.14.0-427.16.1.el9_4                                        baseos                                      5.8 M
 kernel-core                                              x86_64                                      5.14.0-427.16.1.el9_4                                        baseos                                       21 M
Upgrading:
 kernel-tools                                             x86_64                                      5.14.0-427.16.1.el9_4                                        baseos                                      6.0 M
 kernel-tools-libs                                        x86_64                                      5.14.0-427.16.1.el9_4                                        baseos                                      5.8 M
Installing dependencies:
 kernel-modules                                           x86_64                                      5.14.0-427.16.1.el9_4                                        baseos                                       39 M
 kernel-modules-core                                      x86_64                                      5.14.0-427.16.1.el9_4                                        baseos                                       33 M

Transaction Summary
=====================================================================================================================================================================================================================
Install  4 Packages
Upgrade  2 Packages

Total download size: 111 M
Is this ok [y/N]: y
Downloading Packages:
(1/6): kernel-5.14.0-427.16.1.el9_4.x86_64.rpm                                                                                                                                       5.8 MB/s | 5.8 MB     00:00    
(2/6): kernel-core-5.14.0-427.16.1.el9_4.x86_64.rpm                                                                                                                                   12 MB/s |  21 MB     00:01    
(3/6): kernel-tools-libs-5.14.0-427.16.1.el9_4.x86_64.rpm                                                                                                                             16 MB/s | 5.8 MB     00:00    
(4/6): kernel-tools-5.14.0-427.16.1.el9_4.x86_64.rpm                                                                                                                                  16 MB/s | 6.0 MB     00:00    
(5/6): kernel-modules-5.14.0-427.16.1.el9_4.x86_64.rpm                                                                                                                                14 MB/s |  39 MB     00:02    
(6/6): kernel-modules-core-5.14.0-427.16.1.el9_4.x86_64.rpm                                                                                                                           15 MB/s |  33 MB     00:02    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                 33 MB/s | 111 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                             1/1 
  Installing       : kernel-modules-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                            1/8 
  Installing       : kernel-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                    2/8 
  Running scriptlet: kernel-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                    2/8 
  Installing       : kernel-modules-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                 3/8 
  Running scriptlet: kernel-modules-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                 3/8 
  Upgrading        : kernel-tools-libs-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                              4/8 
  Running scriptlet: kernel-tools-libs-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                              4/8 
  Upgrading        : kernel-tools-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                   5/8 
  Running scriptlet: kernel-tools-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                   5/8 
  Installing       : kernel-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                         6/8 
  Running scriptlet: kernel-tools-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                                   7/8 
  Cleanup          : kernel-tools-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                                   7/8 
  Running scriptlet: kernel-tools-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                                   7/8 
  Cleanup          : kernel-tools-libs-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                              8/8 
  Running scriptlet: kernel-tools-libs-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                              8/8 
  Running scriptlet: kernel-modules-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                            8/8 
  Running scriptlet: kernel-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                    8/8 
  Running scriptlet: kernel-modules-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                 8/8 
  Running scriptlet: kernel-tools-libs-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                              8/8 
  Verifying        : kernel-modules-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                 1/8 
  Verifying        : kernel-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                    2/8 
  Verifying        : kernel-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                         3/8 
  Verifying        : kernel-modules-core-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                            4/8 
  Verifying        : kernel-tools-libs-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                              5/8 
  Verifying        : kernel-tools-libs-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                              6/8 
  Verifying        : kernel-tools-5.14.0-427.16.1.el9_4.x86_64                                                                                                                                                   7/8 
  Verifying        : kernel-tools-5.14.0-427.13.1.el9_4.x86_64                                                                                                                                                   8/8 

Upgraded:
  kernel-tools-5.14.0-427.16.1.el9_4.x86_64                                                              kernel-tools-libs-5.14.0-427.16.1.el9_4.x86_64                                                             
Installed:
  kernel-5.14.0-427.16.1.el9_4.x86_64           kernel-core-5.14.0-427.16.1.el9_4.x86_64           kernel-modules-5.14.0-427.16.1.el9_4.x86_64           kernel-modules-core-5.14.0-427.16.1.el9_4.x86_64          

Complete!

Full serial console log https://rpa.st/LZPQ

tested with gp2 volume also

We are in the process of investigating this. We’re not entirely sure why /dev/loop0 is being referenced (it starts at /etc/kernel/cmdline). Before updating an instance, you could change that file to make sure the root matches whatever is in /proc/cmdline at that time, which should technically get around that issue. However, we want to try to make sure you nor anyone else has to do that.

I’ll also point out we do provide LVM variants of our cloud images. The LVM variants of our images are not affected by this, if you wanted to try it out or use it.

Thank you for pointing this out, we’ll be working on it throughout the day and hopefully have an idea for a fix soon.

4 Likes

As an update, @neil and myself have drilled down to the primary issue that was causing this. Due to this, we will be rebuilding and reissuing all of the cloud images for 9.4 on all applicable architectures. This process can take a while, but when we have it all done, we will notify here when they’re good to go.

Thank you all for your patience and apologies for this unfortunate (and rather strange) inconvenience.

2 Likes

Thanks to you and Neil for your prompt investigation into this

We have been using the 9.3 ami in the interim

This is undoubtedly a “me” problem but I’m only seeing 9.3.20231113 in the AWS Marketplace. I might be doing it wrong.
edit: Remembered the old “search for 792107900819” so ignore me. Unless that is also sub-optimal.

Just to inform that I have the same issue after reboot on a different cloud-provider (openstack-based).

Hey Frank - Yeah all the non-LVM based images were affected and are being rebuilt. Should be out today!

Thanks everyone for your patience and reports.

I was able to update from 9.3 → 9.4 without any problems in my environment.

I found that the AWS account ID of the AMI provider of ami-02d6d64456f57b99d is 792107900819.

The official RockyLinux image distribution source is

The AWS accountID of the AMI provider of the 9.3 image booted from here is 679593333241.

Is the AWS account ID for the official AMI image variable…?

Looking forward to seeing the 9.4 images.

Indeed if you take the 9.3 AMI, then do a yum/dnf update, it will update to 9.4 and reboots without issues.

One thing that I noticed is that the 9.4 AMI is not mentionned on Download - Rocky Linux
It only exists on AWS public images with the Rocky Linux account “792107900819”
The “aws-marketplace” account owner “679593333241” doesn’t have the 9.4 AMI, probably because the 9.4 AMI provided by Rocky Linux team is faulty, thus not passing AWS validation.

I’m now thinking if I should use aws 679593333241 instead of rocky 792107900819 for my packer and terraform datasources, if that can free me from some troubles in the future.

Just to note (more for search probably, I see you say it affects all non-uefi cloud images) same issue on OpenStack using Rocky-9-GenericCloud-Base-9.4-20240509.0.x86_64.qcow2

@neil @nazunalika I have just realised your message saying new images should be out “today” is dated 7 days ago - any update please?

Apologies for the delays. We are also dealing with 8.10 work at the same time as this. We have built new images, we just need to upload, test, and publish. Our initial tests with the generic cloud images show that updates should no longer break the images.