Citrix Workspace App vs. certificates in /etc/pki/tls/certs

microlinux · December 2, 2022, 8:05am

Hi,

I’m managing a handful of Linux installations for a client in Belgium. His main workstation is currently running OpenSUSE Tumbleweed, and I’d like to move it to my personal blend of Rocky Linux 8 and KDE. So far I’ve found viable solutions for all the applications he needs. I’m having one last issue with the Citrix Workspace App. He needs this one to connect online to his accountant’s server.

I’ve documented the installation procedure for Citrix Workspace on OpenSUSE here. It’s in French, but the Linux bits are universal:

In a nutshell, the Citrix website offers the application as an RPM package to download. After installing it using rpm -ivh, everything is in /opt/Citrix.

One tweak I had to perform under OpenSUSE to avoid an SSL connection error was to replace the shipped certificates with those already present in the system:

# cd /opt/Citrix/ICAClient/keystore/
# rm -rf cacerts/
# ln -s /etc/ssl/certs cacerts

On a vanilla Rocky Linux 8 installation I got the same SSL error in the default setup of Citrix Workspace, so I tried to tweak it as I had done previously on OpenSUSE:

# cd /opt/Citrix/ICAClient/keystore/
# rm -rf cacerts/
# ln -s /etc/pki/tls/certs cacerts

Only this time I still got the same SSL error. One thing that worked though was to take all the content of /etc/ssl/certs from an OpenSUSE installation and move it to /opt/Citrix/ICAClient/keystore/cacerts on my Rocky Linux installation. But this is a bit of an ugly hack.

Since I’m curious and I like to know why things work or don’t work, I thought I’d post this here.

I vaguely know these files are certificates, but what exactly does the stuff in /etc/ssl/certs (OpenSUSE) or /etc/pki/tls/certs do ? Where do all these files come from ?
What would be an orthodox way to make my Rocky Linux installation behave like OpenSUSE (as described above) ?

I’m not a lamer for RTFM, so feel free to send me a curt link to the relevant documentation.

Thanks & cheers,

Niki

lumarel · December 2, 2022, 8:35am

Hi Niki,

great to see that the Citrix Workspace app also works on Rocky 8, we are running it on Rocky 9, so good to know

And so, this is basically the shared certificate store in the whole Fedora / EL world, I don’t really understand why everyone (Ubuntu, Debian, Alpine, OpenSuSE, …) put them where ever they think is the best place and not all of them at the same one. But here is basically the documentation from Fedora about this topic:

https://docs.fedoraproject.org/en-US/quick-docs/using-shared-system-certificates/

And the same thing in the RHEL docs:

Cheers Lukas

jlehtone · December 2, 2022, 9:27am

The shared certs is mostly about CAs:

Copy certificate authority to trusted ca path: ‘/etc/pki/ca-trust/source/anchors/’
Run ‘update-ca-trust’

That adds your CA certs to the list of CAs that we do trust. The distro applications
do use that list. There seem to be just the “Ca bundle” in /etc/pki/tls/certs:

$ ll /etc/pki/tls/certs
total 0
lrwxrwxrwx. 1 root root 49 Sep 21 17:36 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root 55 Sep 21 17:36 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

Citrix, Apache, dirsrv are services. They have certificate signed by some CA.

For example, the Apache might have in its config some of:

SSLCertificateFile /etc/pki/tls/certs/my_fqdn.pem
SSLCertificateKeyFile /etc/pki/tls/private/my_fqdn.key
# SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
# SSLCertificateChainFile /etc/pki/tls/certs/something.pem
SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt

Assumedly the Citrix config points to CA (chain/bundle), key, and the public part of its certificate.
The CA bundle includes the CA that did sign the services certificate (key&public).

Does the config of Citrix use the correct names of the files?
Are (all) the certificates where they are expected to be?

iwalker · December 2, 2022, 10:03am

Rocky8 and Rocky9, assuming also RHEL, do have /etc/ssl/certs as well, which is symlinked to the appropriate location under /etc/pki.

So in theory you can still potentially use what you did up until now.

[root@rocky9 ~]# ls -lha /etc/ssl/
total 12K
drwxr-xr-x.  2 root root   77 Sep 26 08:34 .
drwxr-xr-x. 83 root root 8.0K Nov 30 23:34 ..
lrwxrwxrwx.  1 root root   49 Sep 20 18:35 cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx.  1 root root   18 Sep 20 18:35 certs -> /etc/pki/tls/certs
lrwxrwxrwx.  1 root root   28 Sep 20 18:35 ct_log_list.cnf -> /etc/pki/tls/ct_log_list.cnf
lrwxrwxrwx.  1 root root   24 Sep 20 18:35 openssl.cnf -> /etc/pki/tls/openssl.cnf

microlinux · December 2, 2022, 2:05pm

I know, and this is how I found out about /etc/pki/tls/certs in the first place. But the contents of these respective directories is quite different under OpenSUSE and Rocky Linux. And I wonder how I could make Rocky Linux behave without copying the whole load of files over from OpenSUSE.

iwalker · December 4, 2022, 12:03pm

Not sure of how it looks in OpenSUSE or the included cacerts in the package, but I know on Debian the /etc/ssl/certs directory has all the files with pem extensions. In Rocky these have .crt extensions. So I’m unsure if that would make a difference or not, if for example Citrix Workspace is looking for .pem files. You could always try symlinking all the .crt files in Rocky to the same name but with a pem extension and see if that helps? In Debian everything in /etc/ssl/certs are pem symlinks to crt files in /usr/share/ca-certificates.

Just an idea maybe.

microlinux · December 4, 2022, 3:14pm

Looks like we’re coming closer. I gave it another spin in a fresh VM. Installed the latest Citrix Receiver, initiated a connection to a Windows server, and got the following SSL error:

Now I took a look in the /etc/ssl/certs directory of my OpenSUSE installation, and found the following two files (among many others):

Go_Daddy_Class_2_CA.pem
Go_Daddy_Root_Certificate_Authority_-_G2.pem

I copied these two files over to /opt/Citrix/ICAClient/keystore/cacerts on my Rocky Linux installation, and now everything works perfectly.

Now I’d like to know two things:

Where do these two files come from on my OpenSUSE installation ? I mean what is their “upstream source” if I can say so ?
How comes this kind of stuff is missing in Rocky Linux ? Or is there an “orthodox” way to install it (besides copying it over from an OpenSUSE) ?

I’m aware my questions may seem a bit naive, but I’m doing my best to describe my ignorance here.

Cheers,

Niki

microlinux · December 4, 2022, 3:28pm

I’ll answer that last question myself, since I just found out the answer

# cd /opt/Citrix/ICAClient/keystore/cacerts
# wget -c https://certs.godaddy.com/repository/gdig2.crt.pem

And now everything works perfectly.

Again, thanks very much everybody for your contributions and your patience.

Cheers,

Niki

jlehtone · December 4, 2022, 3:41pm

The “Go Daddy” is one vendor that sells certificates, isn’t it?
On another system certificates were bought from “GEANT” (or smh).

Certificates created by vendor X are verified by comparing with (public part of) CA of X.
We have the bundle of CAs so that we can verify certificates originating from those “trusted vendors”.

Both Go Daddy Class 2 CA and Go Daddy Root Certificate Authority - G2 CAs are probably included in the ca-bundle of Rocky. If they are, it is issue with Citrix that it does not use the ca-bundle.

Furthermore, within the server certificate there are identifiers and if the service that we try to access does not have those same names, then we don’t trust it. For example, the certificate of this Forum appears to be issued by vendor “Let’s Encrypt” and is valid for accessing HTTPS server that has DNS name forums.rockylinux.org.

Red Hat probably assumes that each site procures their own valid certificates. For example, the Apache package includes only a “self-signed certificate” {localhost.key, localhost.crt}. (Browser does note about it that its CA is not in the bundle.)

Since you don’t seem to have such server-specific certificates, the Citrix does not use them like Apache does?

citrix_user · May 20, 2023, 2:29pm

Just wanted to explain what is going on and what worked for me.

Basically Citrix have decided not to use Mozilla’s bundle of certificates, which is the default bundle on every Linux distribution (and like most other bundles e.g. Apple’s, Microsoft’s, or Chrome’s, the Mozilla bundle contains an enormous amount of certificates from companies you have never heard of, and they are used interchangeably to encrypt connections to email providers, website providers, etc.).
Instead, Citrix have decided to only trust the following companies’ certificates:
-Amazon
-Digicert
-Geotrust
And a few others.

As a result, if you are trying to connect to a server that does not use a certificate from one of these companies, you have to manually add a .pem file to /opt/Citrix/ICAClient/keystore/cacerts and then run /opt/Citrix/ICAClient/util/ctx_rehash from a user which has write permissions to the directory /opt/Citrix/ICAClient/keystore/cacerts.

So, get the company name of the certificate you are looking for (this will show up in your SSL warnings when trying to launch a Citrix Workspace session). Then find the company cert name within the file /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (not /etc/pki/ca-trust/extracted/openssl/ca-bundle.crt.trusted as the certs in this file didn’t work for me) and copy the single certificate into a new file, e.g. alternate.pem, then move it into /opt/Citrix/ICAClient/keystore/cacerts and run ctx_rehash.