AWS Instance status checks failing with "Instance reachability check failed"

Hi Team,

Recently we migrated from Cenos 7 to Rocky 8. All our instances are running on AWS instance type “r6i.4xlarge”.

From last 2-3 months we have been facing an issue where the Instance status checks is showing “Instance reachability check failed”. The servers are failing randomly and on restart it works fine for some random time.

We have investigated the System logs but did not find anything suspicious.
Here is the complete details of Linux for your reference.

This is an urgent issue as it is impacting all our production environments and causing disruptions to end customers. Would request you to please look into it on Priority.
Please let me know if any further info is required.

Command: cat /etc/*release
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: Rocky
Description: Rocky Linux release 8.10 (Green Obsidian)
Release: 8.10
Codename: GreenObsidian
[root@CHAEQ1DBST044V rocky]# cat /etc/*release
Rocky Linux release 8.10 (Green Obsidian)
NAME=“Rocky Linux”
VERSION=“8.10 (Green Obsidian)”
ID=“rocky”
ID_LIKE=“rhel centos fedora”
VERSION_ID=“8.10”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“Rocky Linux 8.10 (Green Obsidian)”
ANSI_COLOR=“0;32”
LOGO=“fedora-logo-icon”
CPE_NAME=“cpe:/o:rocky:rocky:8:GA”
HOME_URL=“https://rockylinux.org/
BUG_REPORT_URL=“https://bugs.rockylinux.org/
SUPPORT_END=“2029-05-31”
ROCKY_SUPPORT_PRODUCT=“Rocky-Linux-8”
ROCKY_SUPPORT_PRODUCT_VERSION=“8.10”
REDHAT_SUPPORT_PRODUCT=“Rocky Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=“8.10”
Rocky Linux release 8.10 (Green Obsidian)
Rocky Linux release 8.10 (Green Obsidian)
Rocky Linux release 8.10 (Green Obsidian)

Thanks in advance.

We are using a variety of EC2 instances; all are running with Rocky Linux 8.10 , did not faced Instance check failed so for.

Did you try to connect to Instance through “EC2 serial console” while it get disconnected?

if yes what is output of demsg -T ?

any log entry during disconnect time ?

Did you try to get the screenshot and “system log” through Ec2 Monitor and troubleshoot option ?

Have you checked with AWS support ?

We do not see any log entry during disconnect time and when we try to take the screenshot we are getting the below. Nothing in the System logs as well.

@linuxlover We also checked with AWS Support and they mentioned the issue lies at OS level.
Moreover, had discussions with MarkLogic Support but mentioned that there is no issue at the MarkLogic level.

I really do love such comments like this. Makes me laugh. If the system is so important to you why aren’t you purchasing a subscription/support contract when things go wrong? And instead, use a free operating system and expect priority support for free when things go wrong lol

Don’t get me wrong, I understand you have a problem, but when something is free, be it the system or support, there’s no place for making demands. You chose to use a system without a support contract/subscription, there are no guarantees or SLA’s for support.

That said, I would at first though try reinstalling the kernel and or reinstalling grub as if it’s just booting to the grub command prompt that looks like it hasn’t read the grub configuration for some reason. But from the screen showing an AWS logo, looks like it’s booted some other image or recovery mode or something, rather than booting Rocky.