Rocky Linux 8.5 Secure Boot Error

Badou_Dream · December 1, 2021, 5:04pm

Hi,

I have a fresh install of RockyLinux 8.5 with UEFI and Secure Boot. My hypervisor is Esxi 7.0U2 and I have configured VM profile to CentOs 8 (64bits).

With bootSecure desactivate, kdump running. But if I activate Secure Boot, kdump failed.

systemctl status kdump
● kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-12-01 17:49:43 CET; 28s ago
  Process: 1276 ExecStart=/usr/bin/kdumpctl start (code=exited, status=1/FAILURE)
 Main PID: 1276 (code=exited, status=1/FAILURE)

déc. 01 17:49:42 srv-proxy-era.codradmz.local systemd[1]: Starting Crash recovery kernel arming...
déc. 01 17:49:43 srv-proxy-era.codradmz.local kdumpctl[1276]: kdump: Secure Boot is enabled. Using kexec file based syscall.
déc. 01 17:49:43 srv-proxy-era.codradmz.local kdumpctl[1276]: kdump: kexec: failed to load kdump kernel
déc. 01 17:49:43 srv-proxy-era.codradmz.local kdumpctl[1276]: kdump: Starting kdump: [FAILED]
déc. 01 17:49:43 srv-proxy-era.codradmz.local systemd[1]: kdump.service: Main process exited, code=exited, status=1/FAILURE
déc. 01 17:49:43 srv-proxy-era.codradmz.local systemd[1]: kdump.service: Failed with result 'exit-code'.
déc. 01 17:49:43 srv-proxy-era.codradmz.local systemd[1]: Failed to start Crash recovery kernel arming.

kdump.log

+ 2021-12-01 15:24:52 /usr/bin/kdumpctl@708: /sbin/kexec -s -s -d -p '--command-line=BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-348.2.1.el8_5.x86_64 ro resume=/dev/mapper/rl-swap irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable disable_cpu_apicid=0' --initrd=/boot/initramfs-4.18.0-348.2.1.el8_5.x86_64kdump.img /boot/vmlinuz-4.18.0-348.2.1.el8_5.x86_64
Try gzip decompression.
Try LZMA decompression.
lzma_decompress_file: read on /boot/vmlinuz-4.18.0-348.2.1.el8_5.x86_64 of 65536 bytes failed
kexec_file_load failed: Operation not permitted
+ 2021-12-01 15:24:52 /usr/bin/kdumpctl@712: ret=255
+ 2021-12-01 15:24:52 /usr/bin/kdumpctl@713: set +x

Thanks for your help

nazunalika · December 2, 2021, 4:13pm

This is a known issue. https://bugs.rockylinux.org/show_bug.cgi?id=174

Badou_Dream · December 2, 2021, 5:09pm

Oh Thank’s.

I test some version to replace CentOs on my production server. I can’t choose a distribution woth No UEFI or no SecureBoot. I hope this problem will be solved quickly, otherwise I would have to turn to AlmaLinux.

Thanks

nazunalika · December 3, 2021, 1:00am

From the bug report:

Looks like it is a shim bug: mok: delete the existing RT variables only when only_first=TRUE by lcp · Pull Request #387 · rhboot/shim · GitHub and hopefully will be fixed when shim 15.5 released
Also someone else did test some versions of ESXi 7.0.2 and kdump was working fine with SB enabled, so not all ESXi version are affected by this.

When shim 15.5 is released, it will be fixed. I don’t have an ETA on that though (that’s more of an upstream thing). According to Sherif though, there are some versions of ESXi that are working just fine with kdump. @Sherif are you able to give a bit more info here?

Sherif · December 3, 2021, 9:11am

Secureboot is working fine on RockyLinux 8.5, you will be able to boot the machine in secureboot and verify that everything is signed with correct certs. We are using shim 15.4 + few critical patches that didn’t make it yet to the upstream distro, more info here about our review and the patches that we included based on the shim-review committee Shim 15.4 for Rocky Linux 8 · Issue #194 · rhboot/shim-review · GitHub

However, there was a bug that seems to be solved with 15.5 which still in RC and didn’t make it to the upstream as well, we did some tests as far as we can since we can’t load certs into EXSi UEFI firmware and we do have an internal ticket open with vmware and they recommend us to include the patch @nazunalika mentioned, which will be included with shim 15.5 once it is released by the upstream distro. The bug is only effecting kexec which is needed to load kdump, doesn’t effect the secureboot verify and booting process. We still have no ETA regarding when 15.5 will be released by the upstream vendor.

Some members managed to get Rocky with secrureboot running without kdump on ESXi 7.0.3 and some other members managed to get secureboot running with kdump on esxi 7.0.2.

joebeasley3 · December 3, 2021, 6:37pm

I can confirm Rocky 8.5 with secure boot and kdump working with esxi 6.7.

shim 15.4-2.el8_5.2.rocky
kexec-tools 2.0.20.el8

Sherif · December 6, 2021, 11:13pm

Thank you so much for the update @joebeasley3

Badou_Dream · December 7, 2021, 10:51am

Thank.

I do to update every esxi and Vcenter, but I need time for this. If I have some news, i answer in this topic. For the moment, with Esxi 7.0.2build17867351, shim 15.4-2.el8_5.2.rocky and kexec-tools 2.0.20.el8 this is not good.

Many thanks

reindan · December 7, 2021, 11:46am

I just stumbled across this thread and can confirm the same issue with Rocky 8.5 on ESXi 7.0.1, 17168206

roc · December 26, 2021, 3:11am

Secure boot appears to be working, but kdump is not working with secure boot, in Rocky 8.5 with ESXi 6.7U3 (18828794) , kexec-tools-2.0.20-57.el8_5.1.x86_64 and shim-x64-15.4-2.el8_5.2.rocky.x86_64.

grydan · April 6, 2022, 1:02pm

I have the same problem with the latest version of Rocky look the image, any idea how to fix? The machine is under VMWARE ESXi Version 6.7 P05… I have also changed the /etc/default/grub in the line from auto to 256M but don’t solve
GRUB_CMDLINE_LINUX=“crashkernel=256M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap”

gerry666uk · April 7, 2022, 6:38pm

Are you saying that Rocky 8.5 works perfectly on VMWARE ESXi Version 6.7 P05, and it’s just the kdump service that doesn’t work?

grydan · April 7, 2022, 8:31pm

Yes if you disable “secure boot” into VMware setting all work without any problem.
Kdump seem don’t work when “secure boot” option is enable into VMware settings.

Opa114 · May 31, 2022, 2:53pm

Any news about this issue with secure boot enables in VMWare ESXi?

Badou_Dream · June 9, 2022, 8:08am

Hi everybody,

All my server are upgrade in 8.6 GreenObside and kdump is Ok.

Thanks everybody

Opa114 · June 9, 2022, 1:56pm

@Badou_Dream interesting. Are you using VMware ESX (vcenter). Secure Boot enabled in vm options?
I’m using 8.6, too and kdump there did not start (secure boot enabled in vm options)

Badou_Dream · June 14, 2022, 8:15am

Sorry Somebody has deactivate the Secure Boot.

No solution for this problem ?

simple_user · June 15, 2022, 1:52pm

@Opa114 and @Badou_Dream, as I understand it, there’s a bug in the secure boot process that most vendors ignore, which means secure boot works for most people. VMWare doesn’t ignore it, and so secure boot is broken.

shim 15.5 fixes this. It was released in the upstream 8.6 edition, but is not yet in the Rocky one. That’s because MS needs to sign the shim to help secure the entire secure boot chain. Rocky put in a request to sign that shim in March.

github.com/rhboot/shim-review

Shim 15.5 for Rocky Linux 8 "ia32/x64"

opened 01:29PM - 25 Mar 22 UTC

closed 09:57AM - 14 Jun 22 UTC

SherifNagy

Make sure you have provided the following information: - [x] link to your co…de branch cloned from rhboot/shim-review in the form user/repo@tag - [x] completed README.md file with the necessary information - [x] shim.efi to be signed - [x] public portion of your certificate(s) embedded in shim (the file passed to VENDOR_CERT_FILE) - [x] binaries, for which hashes are added to vendor_db ( if you use vendor_db and have hashes allow-listed ) - [x] any extra patches to shim via your own git tree or as files - [x] any extra patches to grub via your own git tree or as files - [x] build logs - [x] a Dockerfile to reproduce the build of the provided shim EFI binaries Review avilable at https://github.com/rocky-linux/shim-review/tree/rockylinux-8-shim-x86_64-20220325 ------------------------------------------------------------------------------- ### What organization or people are asking to have this signed? ------------------------------------------------------------------------------- Rocky Enterprise Software Foundation ------------------------------------------------------------------------------- ### What product or service is this for? ------------------------------------------------------------------------------- Rocky Linux 8 ------------------------------------------------------------------------------- ### Please create your shim binaries starting with the 15.4 shim release tar file: https://github.com/rhboot/shim/releases/download/15.4/shim-15.4.tar.bz2 ### This matches https://github.com/rhboot/shim/releases/tag/15.4 and contains the appropriate gnu-efi source. ### Please confirm this as the origin your shim. ------------------------------------------------------------------------------- shim 15.5 from https://github.com/rhboot/shim/tree/15.5 ------------------------------------------------------------------------------- ### What's the justification that this really does need to be signed for the whole world to be able to boot it? ------------------------------------------------------------------------------- Rocky Linux is a community enterprise operating system designed to be 100% bug-for-bug compatible with RHEL ------------------------------------------------------------------------------- ### How do you manage and protect the keys used in your SHIM? ------------------------------------------------------------------------------- Keys stored in FIPS-140-2 level 2 certified HSM and managed by our security team ------------------------------------------------------------------------------- ### Do you use EV certificates as embedded certificates in the SHIM? ------------------------------------------------------------------------------- No ------------------------------------------------------------------------------- ### If you use new vendor_db functionality, are any hashes allow-listed? ### If yes: for what binaries? ------------------------------------------------------------------------------- We don't use vendor_db functionality in this build ------------------------------------------------------------------------------- ### Is kernel upstream commit 75b0cea7bf307f362057cc778efe89af4c615354 present in your kernel, if you boot chain includes a Linux kernel ? ------------------------------------------------------------------------------- We have validated that all those commits present: ``` 475fb4e8b2f4444d1d7b406ff3a7d21bc89a1e6f 1957a85b0032a81e6482ca4aab883643b8dae06e 612bd01fc6e04c3ce9eb59587b4a7e4ebd6aff35 75b0cea7bf307f362057cc778efe89af4c615354 435d1a471598752446a72ad1201b3c980526d869 ``` And the configuration setting CONFIG_EFI_CUSTOM_SSDT_OVERLAYS is disabled. ------------------------------------------------------------------------------- ### if SHIM is loading GRUB2 bootloader, are CVEs CVE-2020-14372, CVE-2020-25632, CVE-2020-25647, CVE-2020-27749, CVE-2020-27779, CVE-2021-20225, CVE-2021-20233, CVE-2020-10713, CVE-2020-14308, CVE-2020-14309, CVE-2020-14310, CVE-2020-14311, CVE-2020-15705, ( July 2020 grub2 CVE list + March 2021 grub2 CVE list ) and if you are shipping the shim_lock module CVE-2021-3418 fixed ? ------------------------------------------------------------------------------- We never built any shim before SBAT support, we only started building and signing shim from shim 15.4 with SBAT support, so the cert that shim uses never used with any grub2 / kernel that effected with those vulnerabilities. ------------------------------------------------------------------------------- ### "Please specifically confirm that you add a vendor specific SBAT entry for SBAT header in each binary that supports SBAT metadata ( grub2, fwupd, fwupdate, shim + all child shim binaries )" to shim review doc ? ### Please provide exact SBAT entries for all SBAT binaries you are booting or planning to boot directly through shim ### Where your code is only slightly modified from an upstream vendor's, please also preserve their SBAT entries to simplify revocation. ------------------------------------------------------------------------------- For shim we have the following SBAT: ``` sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md shim,1,UEFI shim,shim,1,https://github.com/rhboot/shim shim.rocky,1,Rocky Linux,shim,15.5,security@rockylinux.org ``` For grub: ``` sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md grub,1,Free Software Foundation,grub,2.02,https://www.gnu.org/software/grub/ grub.rhel8,1,Red Hat Enterprise Linux 8,grub2,@@VERSION@@,mail:secalert@redhat.com grub.rocky8,1,Rocky Linux 8,grub2,@@VERSION@@,mail:security@rockylinux.org ``` For fwupd binaries will have the following entries: ``` sbat,1,UEFI shim,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md fwupd,1,Firmware update daemon,fwupd,1.5.9,https://github.com/fwupd/fwupd fwupd.rocky,1,Rocky Linux,fwupd,1.5.9,mail:security@rockylinux.org ``` ------------------------------------------------------------------------------- ### Were your old SHIM hashes provided to Microsoft ? ------------------------------------------------------------------------------- yes, they were provided during signing ------------------------------------------------------------------------------- ### Did you change your certificate strategy, so that affected by CVE-2020-14372, CVE-2020-25632, CVE-2020-25647, CVE-2020-27749, CVE-2020-27779, CVE-2021-20225, CVE-2021-20233, CVE-2020-10713, CVE-2020-14308, CVE-2020-14309, CVE-2020-14310, CVE-2020-14311, CVE-2020-15705 ( July 2020 grub2 CVE list + March 2021 grub2 CVE list ) grub2 bootloaders can not be verified ? ------------------------------------------------------------------------------- No, since we never signed any shim , grub2 or kernel before shim 15.4 with SBAT support ------------------------------------------------------------------------------- ### What exact implementation of Secureboot in grub2 ( if this is your bootloader ) you have ? ### * Upstream grub2 shim_lock verifier or * Downstream RHEL/Fedora/Debian/Canonical like implementation ? ------------------------------------------------------------------------------- RHEL downstream like implementation ------------------------------------------------------------------------------- ### Which modules are built into your signed grub image? ------------------------------------------------------------------------------- all_video boot blscfg btrfs cat configfile cryptodisk echo ext2 fat font gcry_rijndael gcry_rsa gcry_serpent gcry_sha256 gcry_twofish gcry_whirlpool gfxmenu gfxterm gzio halt hfsplus http increment iso9660 jpeg loadenv loopback linux lvm luks mdraid09 mdraid1x minicmd net normal part_apple part_msdos part_gpt password_pbkdf2 png reboot regexp search search_fs_uuid search_fs_file search_label serial sleep syslinuxcfg test tftp video xfs efi_netfs efifwsetup efinet lsefi lsefimmap connectefi backtrace chain usb usbserial_common usbserial_pl2303 usbserial_ftdi usbserial_usbdebug keylayouts at_keyboard ------------------------------------------------------------------------------- ### What is the origin and full version number of your bootloader (GRUB or other)? ------------------------------------------------------------------------------- RHEL downstream, grub2-2.02-106.el8.0.2 https://git.rockylinux.org/staging/rpms/grub2 ------------------------------------------------------------------------------- ### If your SHIM launches any other components, please provide further details on what is launched. ------------------------------------------------------------------------------- It launches fwupd ------------------------------------------------------------------------------- ### If your GRUB2 launches any other binaries that are not the Linux kernel in SecureBoot mode, please provide further details on what is launched and how it enforces Secureboot lockdown. ------------------------------------------------------------------------------- Grub validates signatures using shim's protocol for boot kernel. fwupd only loads UEFI updates ------------------------------------------------------------------------------- ### If you are re-using a previously used (CA) certificate, you will need to add the hashes of the previous GRUB2 binaries exposed to the CVEs to vendor_dbx in shim in order to prevent GRUB2 from being able to chainload those older GRUB2 binaries. If you are changing to a new (CA) certificate, this does not apply. ### Please describe your strategy. ------------------------------------------------------------------------------- We only started signing shim after 15.4 with SBAT support ------------------------------------------------------------------------------- ### How do the launched components prevent execution of unauthenticated code? ------------------------------------------------------------------------------- Grub validates signatures using shim's protocol for boot kernel. fwupd only loads UEFI updates ------------------------------------------------------------------------------- ### Does your SHIM load any loaders that support loading unsigned kernels (e.g. GRUB)? ------------------------------------------------------------------------------- No ------------------------------------------------------------------------------- ### What kernel are you using? Which patches does it includes to enforce Secure Boot? ------------------------------------------------------------------------------- RHEL kernel 4.18.0 and all signed kernels are all patched for the kernel boothole CVEs and have the CONFIG_EFI_CUSTOM_SSDT_OVERLAYS config option disabled. ------------------------------------------------------------------------------- ### What changes were made since your SHIM was last signed? ------------------------------------------------------------------------------- Our latest signed shim was shim 15.4 + cherry picked patches based on the shim review board recommendation, we dropped all that and using latest shim 15.5 ------------------------------------------------------------------------------- ### What is the SHA256 hash of your final SHIM binary? ------------------------------------------------------------------------------- ``` 945511223c674a99cbf2fe459f6b5d1643d92e352342d52b9560ec2397ff6c0d shimx64.efi 465e03b7af50b04bb92ed708c0997c9c728ff45e7153d664b1d6905f79678089 shimia32.efi ```

They responded to a few questions and up until yesterday, all of the recent posts have been Rocky asking if there are any updates or any more information needed.

That request was closed yesterday without signing the shim because of new CVE’s and Rocky needs to open a new request.

So, there is a fix for this. The ability to implement is out of the Rocky Linux’ team hands. I wouldn’t expect any solution soon, because of both the time this request languished and the need to stay bug for bug compatible with upstream, which already has a signed 15.5 shim.

windacplay · July 30, 2022, 4:39pm

Please has anyone solved this problem?
Use environment:
Use the latest version rocky8.6
dnf update to the latest version
Esxi7.0.3f 20036589
After a fresh install, the problem still occurs
And try to change the crashkernel=auto value in grub2-efi.cfg
128/192/256M still can’t start