LetsEncrypt provides correct certificate, but doesn't work

hartings · October 14, 2022, 5:04pm

Hi again!
I run a fresh RL) on a new ,still not running server (mail & web server), called “server.hartings.se”. I have installed certbot and it installed very nicely a certificate from LetsEncrypt for my domain “hartings.se”:

[root@server ~]# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: hartings.se
    Serial Number: 49d2631d66b5055a8bc69d59c3cd44c5895
    Key Type: RSA
    Domains: hartings.se www.hartings.se
    Expiry Date: 2023-01-12 10:14:10+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/hartings.se/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/hartings.se/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[root@server ~]#

But when I do a test as running server, with the server acting as domain hartings.se, my browser shows for https://hartings.se, that the site is not secure: and t shows the certificate is for “server.hartings.se” it shows nothing else, see picture below. When my (old) server (server1.hartings.se) acts as main domain server, it shows in the browser that the certificate is from LetsEncrypt and all other details - as it is expected. all is OK!
But this is shown on the (new) server (server.hartings.se):

What is wrong here? Can anyone point me to where I should start looking?
Thanks!

sspencerwire · October 14, 2022, 5:45pm

Greetings @hartings … What does your httpd or nginx configuration look like? Also, what exactly is showing when you try to load the front page, can you include a screenshot of that? What exactly doesn’t work?

hartings · October 14, 2022, 7:39pm

Many thanks @sspencerwire for your reply! Sorry for a lengthy reply

What is not running is:

the LetsEncrypt certificate is not seen by the web browser, so the connection is regarded as not secure. It looks almost like if there is another general certificate somewhere?
some of the links I use on my running server, like Webmail - hartings.se doesn’t work. This is normally linked to:

ProxyPass /webmail/ http://hartings.se:7080/surgeweb
ProxyPassReverse /webmail/ http://hartings.se:7080/surgeweb

Such strange behaviour points to problems in the httpd.conf file, but I cannot really find any…
I have a longer httpd.conf file as I run my mailserver through apache. The “same” httpd.conf is used on my running server, which has no problems. The differences are:
The differences between the httpd.conf on the new, not running server with the original httpd_hartings.conf from the still running and working server are:

[root@server conf]# diff httpd.conf httpd_hartings.conf
47c47
< #isten 80
---
> Listen 80
359,360c359,360
< 
< #start extra lines in httpd.conf to run Surgemail WebMail through port 80 or ssl
---
> LoadModule rewrite_module modules/mod_rewrite.so
> #start extra lines to run Surgemail WebMail through port 80 or ssl
487d486
< Include /etc/httpd/conf/httpd-le-ssl.conf
[root@server conf]#

I have not tried yet to activate the line “Listen 80”.

I also ran

[root@server ~]# apachectl configtest
Syntax OK
[root@server ~]#

So no syntax errors at least in htpd.conf. I don’t see any means here to upload the file itself. How do I do that here?
Here are anyhow some important parts:

[root@server conf]# tail httpd.conf
<VirtualHost *:80>
    ServerName hartings.se
    Redirect / https://hartings.se/
    ServerAlias www.hartings.se
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.hartings.se [OR]
RewriteCond %{SERVER_NAME} =hartings.se
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
Include /etc/httpd/conf/httpd-le-ssl.conf
[root@server conf]# more /etc/httpd/conf/httpd-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName hartings.se
    Redirect / https://hartings.se/
    ServerAlias www.hartings.se
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =www.hartings.se [OR]
# RewriteCond %{SERVER_NAME} =hartings.se
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/hartings.se/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/hartings.se/privkey.pem
</VirtualHost>
</IfModule>
[root@server conf]# more /etc/letsencrypt/options-ssl-apache.conf
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file. Contents are based on https://ssl-config.mozilla.org

SSLEngine on

# Intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AE
S256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA38
4
SSLHonorCipherOrder     off
SSLSessionTickets       off

SSLOptions +StrictRequire

# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
[root@server conf]#

I am gratefull for any help as where to start looking, as I need to switch servers!

rockandroller · October 14, 2022, 7:59pm

Dunno if this is related, but: I recently did a big migration, and found a problem with one domain not getting SSL (even though certbot certificates showed the cert existing) … and it was with a domain that was a server alias of another domain.
The /etc/httpd/conf/httpd-le-ssl.conf virtual host entry for it pointed to the key file for the “main” domain .
I split it out of the original vhost and made it a standalone vhost in both the httpd.conf and the httpd-le-ssl.conf (new entry pointing to the correct key-folder) (and vhost pointing to the same web directory) and then all was well…

gerry666uk · October 14, 2022, 8:21pm

The screenshot thinks you are browsing to a site that starts with the word “server”, but your cert does not support such a site.

You could test with curl -v to get the exact names.

sspencerwire · October 14, 2022, 9:20pm

In your original message, the path to the SSL is shown by cerbot certificates as:

Certificate Path: /etc/letsencrypt/live/hartings.se/fullchain.pem
Private Key Path: /etc/letsencrypt/live/hartings.se/privkey.pem

What is the path in your httpd.conf? There should be a path to the certificates there. Note that by default, cerbot puts the certificates in the configuration found in the default location. If you’ve got a custom location for the configuration file, it didn’t find that configuration… You’d simply need to place the SSL certificate lines in the httpd.conf. Do a search for SSLCertificate and SSLCertificateChainFile to see what is there. If nothing, you need to place the path to both certificates (again from your original message) in those lines. You’ll also need a line for the private key using SSLCertificateKeyFile. The SSLCertificate and the SSLCertificateChainFile will use the same .pem.

hartings · October 15, 2022, 10:30am

Thanks @rockandroller !

“… and it was with a domain that was a server alias of another domain.
The /etc/httpd/conf/httpd-le-ssl.conf virtual host entry for it pointed to the key file for the “main” domain .”

I don’t think this is the case here. The certificate is for one domain hartings.se with its subdomain www.hartings.se. Or do I misread your comment?

Thanks @gerry666uk . Yes, this is the problem I have and try to solve.
The question is why does it points to “server.hartings.se” instead of the domain hartings.se as is stated in the LetsEncrypt certificate.

Thanks @sspencerwire !

“What is the path in your httpd.conf? There should be a path to the certificates there. Note that by default, cerbot puts the certificates in the configuration found in the default location. If you’ve got a custom location for the configuration file, it didn’t find that configuration… You’d simply need to place the SSL certificate lines in the httpd.conf.”

Good point. However, the httpd.conf does have this line:

Include /etc/httpd/conf/httpd-le-ssl.conf

and this file includes:

[root@server conf]# more /etc/httpd/conf/httpd-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName hartings.se
    Redirect / https://hartings.se/
    ServerAlias www.hartings.se
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =www.hartings.se [OR]
# RewriteCond %{SERVER_NAME} =hartings.se
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/hartings.se/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/hartings.se/privkey.pem
</VirtualHost>
</IfModule>
[root@server conf]#

So indirectly the location of the LetsEncrypt cert is included. Or am I wrong? Should they physically/directly be included in httpd.conf?

For what it is worth: I have updated the httpd.conf on the new server (server.hartings.se) to be exactly the same as in the old runing server (server1.hartings.se). Exactly meaning: the command lines are exactly identical. Some outcomented lines with “#” are different. As far as I can see te other files:

/etc/httpd/conf/httpd-le-ssl.conf
/etc/letsencrypt/options-ssl-apache.conf

are 100% identical on both servers.

I restarted httpd, but the certificate shown in the browser is still the same it only shows server.hartings.se as i the picture in my first post. I t should say that the certificate is not valid as it is valid for hartings.se, but is doesn’t…

sspencerwire · October 15, 2022, 12:30pm

In that included file, I don’t see the SSLCertificateChainFile which should be:

SSLCertificateChainFile /etc/letsencrypt/live/hartings.se/fullchain.pem

Yes. It is exactly the same as the SSLCertificateFile, because the .pem includes both.

Provided that your include statement is in the correct spot in your httpd.conf file, that should work. I’m going to be away for a few days and won’t have access to electronics. Hope you get this figured out!

hartings · October 15, 2022, 3:19pm

thanks again @sspencerwire !

I copied the line SSLCertificateChainFile /etc/letsencrypt/live/hartings.se/fullchain.pem directly into the main httpd.conf file and restarted httpd.
At first, I didn’t see any difference. The browser does not respond to localhost or 127.0.0.1 In those cases it redirects to hartings.se which is active hartings.se server in my LAN.
Then I went to https://192.168.1.221 , which is the server.hartings.se temporary IP address for tweaking before launching it. It showed the website I run (a copy is already installed on this new server).

When I now look at what the browser says about the certificate, see the attached screendump:

And there the hartings.se certificate is, but the one with server.hartings.se is still there…
I wonder if this is created by my built-in mailserver program Surgemail, or by Rocky Linux as standard during the fresh installation.
I need to find out where that certificate is and then just remove it, I guess.
I searched for “ertificat” on the server and found some (apart from some responses like binary’s and /usr/lib/…files):

/var/www/html/wp-includes/certificates
/var/www/html/wp-includes/certificates/ca-bundle.crt
/var/www/html_old/open/ralf/wordpress/wp-includes/certificates
/var/www/html_old/open/ralf/wordpress/wp-includes/certificates/ca-bundle.crt

These are from my website. The ca-bundle.crt file contains: (first part only):

Bundle of CA Root Certificates
##
## Certificate data from Mozilla as of: Wed Jul 22 03:12:14 2020 GMT
## Includes a WordPress Modification - We include the 'legacy' 1024bit certificates
## for backward compatibility. See https://core.trac.wordpress.org/ticket/34935#comment:10
##
## This is a bundle of X.509 certificates of public Certificate Authorities
## (CA). These were automatically extracted from Mozilla's root certificates
## file (certdata.txt).  This file can be found in the mozilla source tree:
## https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt
##
## It contains the certificates in PEM format and therefore
## can be directly used with curl / libcurl / php_curl, or with
## an Apache+mod_ssl webserver for SSL client authentication.
## Just configure this file as the SSLCACertificateFile.
##
## Conversion done with mk-ca-bundle.pl version 1.27.
## SHA256: fffa309937c3be940649293f749b8207fabc6eb224e50e4bb3f2c5e44e0d6a6b
##

EE Certification Centre Root CA
===============================
-----BEGIN CERTIFICATE-----
MIIEAzCCAuugAwIBAgIQVID5oHPtPwBMyonY43HmSjANBgkqhkiG9w0BAQUFADB1MQswCQYDVQQG
EwJFRTEiMCAGA1UECgwZQVMgU2VydGlmaXRzZWVyaW1pc2tlc2t1czEoMCYGA1UEAwwfRUUgQ2

I also checked the old (running) server for hartings.se and the same certificate files are on the old running server, which doesn’t have the issue… So this certificate is not the real problem? So what causes the problem?

Thanks for any comments! This is a great forum!

rockandroller · October 15, 2022, 9:22pm

Looks like your issue here was a little different than mine. I have had some weird oddities with DNS servers caching old server IPs during migration. With Certbot, I like to make sure that the domain is actually ‘real-world-global’ directed to the NEW server (‘migrated’ site responding http:// !) before I use certbot to load an SSL cert onto it. I usually do a slight edit to the site title l so I know for sure which one I am really viewing…
Generally its a situation where I monitor closely as DNS can ripple through very quick - or sometimes not!

hartings · October 16, 2022, 2:15pm

Thanks @rockandroller ! I did install the cerbot certificate during a first live-trial, when it was acting as "offical " server. The installed certificate from LetsEncrypt is indeed OK. The issue is the other certificate, which I strongly suspect is coming from my mailserver Surgemail, which apparently created its own certificate (“server.hartings.se”). This is the one I need to remove, but I haven’t found out yet how… I have contacted Surgemail support which normally provides a fantastic service!

sspencerwire · October 17, 2022, 1:33pm

This whole thing is haunting me as in my old $dayjob (which I left in 2021) I ran into issues with an SSL certificate once and ended up putting the certificate redirects into the SSL.conf… (/etc/httpd/conf.d/ssl.conf)… I don’t specifically remember the entire issue, but I do remember that it was similar.

What is in this file for you? Does it by chance have a direct to the certificate that is arguing with you?

Look for this part way down in the file:

##
## SSL Virtual Host Context
##

hartings · October 17, 2022, 3:17pm

Thanks again @sspencerwire !

This is what I see regarding references to such certificates (see screenshot):

These are clearly not related to the Letsencrypt certificates!

I am checking this from my phone as I am out of office, and this is a bit uncomfortable, but I suspect this is the certificate that is bothering me. I’ll check this on Thursday more in detail!
Thanks for the tip!

pajamian · October 18, 2022, 1:27pm

This is not the correct way to check a certificate. The problem is that httpd uses the hostname you connect with to figure out which certificate to send to the browser, and the browser checks that same hostname against the CN and Subject Alternative Names in the certificate itself and none of these will match 192.168.1.221. What you need to do is add an entry to your hosts file pointing that IP address to the appropriate domain name that you’re testing, dump cache in your browser then the browser will correctly test the certificate on that server.

hartings · October 21, 2022, 9:23am

Thanks again @sspencerwire .
I have now looked at that file /etc/httpd/conf.d/ssl.conf you mentioned and I found the lines which refer to:

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

I commented out those two lines and restarted httpd, but it resulted in a failure. Those lines seems to be required.
Renaming /etc/httpd/conf.d/ssl.conf to something else, not ending with .conf, didn’t make a difference.

hartings · October 21, 2022, 10:03am

Thanks @pajamian !

My /etc/hosts file looked like:

[root@server conf.d]# more /etc/hosts
127.0.0.1	localhost server.hartings.se localhost4 localhost4.localdomain4
::1	localhost server.hartings.se localhost6 localhost6.localdomain6
[root@server conf.d]#

and I changed it to:

[root@server conf.d]# more /etc/hosts
127.0.0.1	localhost server.hartings.se localhost4 localhost4.localdomain4
::1	localhost server.hartings.se localhost6 localhost6.localdomain6
192.168.1.221	hartings.se
[root@server conf.d]#

I cleared the cache in the browser and now I don’t see any certificate at all when browsing to https://192.168.1.221 It says the site is not secure, I can click on “view certificate” in the browser, but nothing is shown (blank list).

Adding 192.168.1.221 in the hosts file and connect it to “hartings.se” .should not cause a conflict with the DNS server om my active server 192.168.1.94 for “hartings.se”, when I enter in the browser https://hartings.se as this should point to its own IP adress 192.168.1.221, following the info in the hosts file…The indicated order of resolving is 1. hosts file, 2 DNS
I also stopped the DNS server on my LAN so only the host file with “192.168.1.221 hartings.se” should be seen by the new test server. In that case I still see the certificate “server.hartings.se”.
I also removed “server.hartings.se” from the /etc/hosts file. Same result.

I also don’t think anymore that this certificate issue is due to my mail server Surgemail, as it isn’t related to “server.hartings.se”, but only to the domain it is serving, hartings.se

How do I get rid of that certificate for “server.hartings.se”? How is this controlled?
What am I doing wrong?
Am I testing the certificate for “hartings.se” on this test-server in the right way?

I am lost now…
Any help is really appreciated!!

sspencerwire · October 21, 2022, 8:41pm

They are required… I think what I meant was for you to try inserting your SSL certificate values in the bottom of that file and then restarting. “Yes” they should load from your configuration file properly, but again, I’ve seen weird issues like this before.

Leave the values in the ssl.conf, then add your Let’s encrypt values at the bottom of the file and restart.

sspencerwire · October 21, 2022, 8:44pm

With your hosts file in place, from the same machine that you have the host file set:

ping hartings.se

Does it resolve to your host file IP (it should) or the DNS values?

gerry666uk · October 21, 2022, 9:56pm

apache web server “out-of-the-box” will have those two lines, but when you configure it for a real website, those lines will be gone, and you’ll have directives hand crafted for your site, which will include paths to the sitename certificate, the intermed, and sitename key.

hartings · October 22, 2022, 1:42pm

Thanks @gerry666uk !
Yes, this is how it normally works and works on my running server.
However in this case it doesn’t seem to be the case. That is why I have a problem. The httpd.conf on my new (not active) server is a 100% identical copy of the corresponding file on the running server, apart from that I also here needed to add the path to the Letsencrypt certicifate (chain and private) at the very bottom of the file to get the certificates visable… This should not be required if all is OK, as I understood it.

[root@server ~]# tail /etc/httpd/conf/httpd.conf
    ServerName hartings.se
    Redirect / https://hartings.se/
    ServerAlias www.hartings.se
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.hartings.se [OR]
RewriteCond %{SERVER_NAME} =hartings.se
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
Include /etc/httpd/conf/httpd-le-ssl.conf
SSLCertificateChainFile /etc/letsencrypt/live/hartings.se/fullchain.pem
[root@server ~]#

Thanks @sspencerwire for the detailed help!

Yes, the ping points indeed to the same machine (192.168.1.221):

[root@server ~]# ping -c 5 hartings.se
PING hartings.se (192.168.1.221) 56(84) bytes of data.
64 bytes from hartings.se (192.168.1.221): icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from hartings.se (192.168.1.221): icmp_seq=2 ttl=64 time=0.019 ms
64 bytes from hartings.se (192.168.1.221): icmp_seq=3 ttl=64 time=0.014 ms
64 bytes from hartings.se (192.168.1.221): icmp_seq=4 ttl=64 time=0.017 ms
64 bytes from hartings.se (192.168.1.221): icmp_seq=5 ttl=64 time=0.018 ms

--- hartings.se ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4088ms
rtt min/avg/max/mdev = 0.014/0.020/0.035/0.007 ms
[root@server ~]#

So that is OK! But I still see in the browser first the certificate from “server.hartings.se” before the real ones from “hartings.se” WITHOUT adding the LetsEncrypt certificate location to the bottom of ssl.conf.

Now I am uncertain what you mean by

The ssl.conf file is a configuration file, not a certificate itself. Adding the lines
SSLCertificateChainFile /etc/letsencrypt/live/hartings.se/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/hartings.se/privkey.pem

to the ssl.conf file, doesn’t make a difference. So I removed them again.

So I assume you mean the text inside the certificate files below?

Meaning the certificate text content from (SSLCertificateChainFile)

/etc/letsencrypt/live/hartings.se/fullchain.pem       into the file
/etc/pki/tls/certs/localhost.crt

And also the certificate text content from (SSLCertificateKeyFile )

/etc/letsencrypt/live/hartings.se/privkey.pem       into the file 
/etc/pki/tls/private/localhost.key

Have I understood this correctly?

Topic		Replies	Views
Dovecot - 10-ssl.conf - Letsencrypt Certificates Rocky Linux General	1	241	August 25, 2023
Problem certificate ssl Rocky Linux General	2	563	August 24, 2023
Problem with Logwatch since the last update Rocky Linux General	5	129	January 25, 2024
Certbot --apache SSL Cert no longer trusted Rocky Linux General	1	395	August 25, 2023
TLS errors in unbound Rocky Linux General	8	940	August 25, 2023

LetsEncrypt provides correct certificate, but doesn't work

Related Topics