We have exact same ESXi and vSphere Client versions and builds. I did a test install for Rocky 8.5 with matching specs from your screenshots and I was able to install the ‘Server without gui’. It boots up fine. See attached screenshot.
We have Dell PE R630 servers with 20 cores, 192GB RAM, and 2TB raid.
Were you able to install RedHat 8/CentOS 8 before?