Upgrading Rocky Linux 8.5 failed to boot

Hi All, today i upgraded rocky linux from 8.5 after the upgrade i reboot the system and failed to boot. heres the screenshot of the errors.


Uploading: 296884160_826974508472002_9126610545202497383_n.jpg…

The system ruined after the upgrade.
Can anyone give some input based the screenshot i provided?

If im going to use SystemrescueCD, do you think it can fixed the system?

thank you in advance

The kernel messages seem to refer to your disk having issues. Your disk may be starting to fail. I would suggest booting into a rescue media and checking the SMART data.

1 Like

like the systemRescueCd sir ?

Kind of, but don’t forget you can use the Rocky install media as a bootable rescue cd, the advantage being that every package will be compatible with the “broken” installatoin.

1 Like

Thank very much sir! 'still struggling to revive the disk here :slight_smile:

Hi all, :pray:
Im trying to understand the log result here, but i need really your expertise mates
:pray:

# smartctl -a /dev/sdc
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1160.15.2.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 3.5
Device Model:     ST2000DM006-2DM164
Serial Number:    Z4Z91ZSD
LU WWN Device Id: 5 000c50 0a474e5a3
Firmware Version: CC26
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 11 09:32:52 2022 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(   89) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 232) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x1085)	SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   093   093   006    Pre-fail  Always       -       136969175
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       181
  5 Reallocated_Sector_Ct   0x0033   079   079   010    Pre-fail  Always       -       26688
  7 Seek_Error_Rate         0x000f   083   060   030    Pre-fail  Always       -       226872390
  9 Power_On_Hours          0x0032   091   085   000    Old_age   Always       -       8326
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       89
183 Runtime_Bad_Block       0x0032   095   095   000    Old_age   Always       -       5
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       247
188 Command_Timeout         0x0032   099   099   000    Old_age   Always       -       1 3 4
189 High_Fly_Writes         0x003a   092   092   000    Old_age   Always       -       8
190 Airflow_Temperature_Cel 0x0022   071   057   045    Old_age   Always       -       29 (Min/Max 29/29)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       127
193 Load_Cycle_Count        0x0032   087   087   000    Old_age   Always       -       26630
194 Temperature_Celsius     0x0022   029   043   000    Old_age   Always       -       29 (0 20 0 0 0)
197 Current_Pending_Sector  0x0012   001   001   000    Old_age   Always       -       23800
198 Offline_Uncorrectable   0x0010   001   001   000    Old_age   Offline      -       23800
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       21157h+50m+33.352s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       7822365613
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       35518177192

SMART Error Log Version: 1
ATA Error Count: 250 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 250 occurred at disk power-on lifetime: 8325 hours (346 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:00:09.537  READ FPDMA QUEUED
  60 00 90 ff ff ff 4f 00      00:00:09.534  READ FPDMA QUEUED
  60 00 38 ff ff ff 4f 00      00:00:09.534  READ FPDMA QUEUED
  60 00 18 ff ff ff 4f 00      00:00:09.534  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:00:09.534  READ FPDMA QUEUED

Error 249 occurred at disk power-on lifetime: 8325 hours (346 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:27:35.773  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:27:35.770  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:27:35.764  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:27:35.761  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:27:35.748  READ FPDMA QUEUED

Error 248 occurred at disk power-on lifetime: 8325 hours (346 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:01:53.813  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:01:53.810  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:01:53.804  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:01:53.801  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:01:53.788  READ FPDMA QUEUED

Error 247 occurred at disk power-on lifetime: 8325 hours (346 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 53 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:00:55.340  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00      00:00:55.340  SET FEATURES [Enable SATA feature]
  27 00 00 00 00 00 e0 00      00:00:55.339  READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
  ec 00 00 00 00 00 a0 00      00:00:55.339  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      00:00:55.339  SET FEATURES [Set transfer mode]

Error 246 occurred at disk power-on lifetime: 8325 hours (346 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 53 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:00:51.698  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:00:51.690  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:00:51.681  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:00:51.681  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      00:00:51.676  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Mybe try these simpified commands

smartctl --health /dev/sdc
smartctl --test=short /dev/sdc

wait, wait, wait

smartctl --log=error /dev/sdc
smartctl --log=selftest /dev/sdc
1 Like

Thanks mate.
Now my challenge on mounting gpt. still googling for possible guide.

The GPT is a format of partition table. One does not mount partition table.
One does mount filesystem that usually is in partition.

gdisk -l /dev/sdc
lsblk
lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,MOUNTPOINTS
blkid
1 Like

Thanks mate, i am able to mount /dev/sdc1 the file system(the one failed to boot).
my question is , Is it safe to delete everything inside EFI and replace it from a copy of a fresh-new installed filesystem from Rocky?

You havn’t shown the results of the smart tests above yet, nor the output of the gdisk and lsblk commands, nor have you made it clear if you’ve understood why it was showing disk errors.

Regarding the EFI partition, you should be able to do this from within the GUI installer, there’s an option to re-use and re-format, but it’s better to start with a completely clean disk, install, restore from backups.

1 Like

Heres the result with smartctl -H /dev/sdc

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.18.0-2-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

and heres the Errlog with smartctl

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.18.0-2-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.18.0-2-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
ATA Error Count: 2541 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2541 occurred at disk power-on lifetime: 8366 hours (348 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:07:10.138  READ FPDMA QUEUED
  60 00 08 00 f0 32 40 00      00:07:10.120  READ FPDMA QUEUED
  60 00 48 ff ff ff 4f 00      00:07:10.118  READ FPDMA QUEUED
  60 00 50 ff ff ff 4f 00      00:07:10.118  READ FPDMA QUEUED
  60 00 30 ff ff ff 4f 00      00:07:10.118  READ FPDMA QUEUED

Error 2540 occurred at disk power-on lifetime: 8356 hours (348 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      00:00:09.965  READ FPDMA QUEUED
  60 00 48 b8 0d 29 42 00      00:00:09.951  READ FPDMA QUEUED
  60 00 50 60 0d 29 42 00      00:00:09.951  READ FPDMA QUEUED
  60 00 30 28 0d 29 42 00      00:00:09.951  READ FPDMA QUEUED
  60 00 a8 78 0c 29 42 00      00:00:09.951  READ FPDMA QUEUED

Error 2539 occurred at disk power-on lifetime: 8355 hours (348 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 53 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      14:40:32.271  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00      14:40:32.270  SET FEATURES [Enable SATA feature]
  27 00 00 00 00 00 e0 00      14:40:32.270  READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
  ec 00 00 00 00 00 a0 00      14:40:32.270  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      14:40:32.269  SET FEATURES [Set transfer mode]

Error 2538 occurred at disk power-on lifetime: 8355 hours (348 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 53 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      14:40:28.573  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      14:40:28.572  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      14:40:28.565  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      14:40:28.564  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00      14:40:28.559  READ FPDMA QUEUED

Error 2537 occurred at disk power-on lifetime: 8355 hours (348 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 53 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00      14:40:24.672  READ FPDMA QUEUED
  ef 10 02 00 00 00 a0 00      14:40:24.672  SET FEATURES [Enable SATA feature]
  27 00 00 00 00 00 e0 00      14:40:24.672  READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
  ec 00 00 00 00 00 a0 00      14:40:24.672  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      14:40:24.671  SET FEATURES [Set transfer mode]

In your earlier post:

  • 26688 (original) sectors have failed a write and for each of them a new one has been taken into use from reserve. There are only a limited number of reserve sectors that can be allocated.
  • 23800 sectors should be “reallocated”, i.e. remapped but for some reason haven’t. Perhaps the reserve has already exhausted?

I’d replace the disk ASAP, since it has had so many reallocation events (and attempts) already.

Do you have backups of the data?

One last ditch backup procedure is:

  1. Do not boot from the failed disk. Don’t even mount it anywhere
  2. Attach new disk of sufficient size
  3. Boot from something else: Installer image, Live image, etc
  4. dd entire content of the faulty disk to new disk. E.g. ‘dd if=/dev/sdc of=/dev/sdd’ (if sdd is the new, empty disk)
  5. Remove faulty disk
  6. Fresh install OS to (another) new disk
  7. Mount filesystems from the copy (the former sdd) and restore data to installed system
  8. The “sdd” can then be used as backup media or rearranged to be a data volume

The main point is that access to faulty is minimized; only one read (with dd)

1 Like

Thank you for this very detailed/useful reply mate. i follow this Thanks again!

Maybe I’m missing something, but aren’t the errors in your original screenshot referring to sda? Why are smartctl commands being run against sdc?

that was an image taken when the subject drive failed to boot. The errLogs was reproduced using gparted live cd right after.