How can we upgrade openssl 3.0.7 to 3.2.1 on rocky linux 9.4

techgeek93 · September 10, 2024, 11:22am

We want to upgrade openssl 3.0.7 to 3.2.1 on rocky linux 9.4. By default the openssl version is 3.0.7 and upgrade does not show new version directly. So, what could be the best way to upgrade?

iwalker · September 10, 2024, 11:27am

You cannot upgrade it. The version of OpenSSL provided with RHEL and Rocky cannot be changed. If RHEL release a newer version in the future, only at this point will you be able to use it. The system depends on it, and if you attempt to upgrade it yourself, you will break the server. Same applies to glibc.

Any fixes required for OpenSSL are backported by RHEL so there should be no need to do anything.

See this for info on backporting: What is backporting and how does it affect Red Hat Enterprise Linux? - Red Hat Customer Portal

remi · September 10, 2024, 11:46am

OpenSSL 3.2.2 is already in CentOS Stream 9 (and 10)

So “should” be part of Rocky 9.5 (and 10.0)

hortimech · September 10, 2024, 12:12pm

Centos stream is upstream from RHEL, so something that is in Centos Stream may get into the next version of RHEL, it doesn’t mean that it will or should be in the current RHEL. Rocky Linux is now where the old Centos used to be, down stream from RHEL, so if it isn’t in RHEL, it will not be in Rocky Linux.

jlehtone · September 10, 2024, 12:38pm

Not quite. As explained in the backporting page, the version is 3.0.7-27.el9.
That is something in RHEL 9 and Rocky 9 that “has evolved from upstream 3.0.7” and thus no longer 3.0.7.

If one replaces packages In Rocky with something that is not Rocky, then the system is not quite Rocky any more. If it breaks, then you can keep the pieces …

In other words, the “best way” is to not mess with the system.

That is scary. It is very uncommon for Red Hat to rebase core components. Although, IIRC, they have rebased OpenSSL before and it was not painless. If there is (security) reason that warrants such rebase …
Perhaps it is easier with OpenSSL 3?

sghatuwa · September 10, 2024, 3:59pm

If 3.2 not supported, then is upgrade to openssl 3.1 possible? or can we downgrade to openssl 1.1?

iwalker · September 10, 2024, 4:00pm

Same answer as before to the question about 3.2. Downgrade to 1.1 not possible, Rocky 9 and RHEL 9 has OpenSSL 3.

Read the backporting link, and then understand why the version number is practically irrelevant at this point. Accept the fact that whatever version it is, it gets fixes, and that is all that is important.

raven · September 11, 2024, 3:37am

@techgeek93 what is the goal of this update? If you want to update OpenSSL for the WHOLE operating system, this is not possible because so many dependencies with other parts of the system will be broken. So you will most likely end up with a broken system.
If you need a different version of OpenSSL for a particular software, you can use one of compat-openssl11 or libressl packages or build your own version of OpenSSL prefixed somewhere in /opt and link your software to it.

techgeek93 · September 11, 2024, 5:35am

@raven we are seeing communication link failures randomly on MariaDB while using ssl enabled users on the application. With ssl disabled users there isn’t any issue related to link failures. With some research found that the openssl version 3.0 has some performance degradation over 1.1. We were using 1.1 before on centos7 and did not face any issues with ssl enabled users there.

github.com/openssl/openssl

OpenSSL 3.0 performance degradation due to bottleneck in the `ossl_gcm_get_ctx_params` and `ossl_gcm_set_ctx_params` functions

opened 10:43AM - 30 Jan 23 UTC

colmenero

triaged: feature triaged: performance

**Context**: We are observing some performance degradation when migrating to Ope…nSSL 3.0.7 from OpenSSL 1.1.1s. Looking at a profiler in a Linux architecture revealed that the calls to [`EVP_CipherInit_ex`](https://github.com/openssl/openssl/blob/19cc035b6c6f2283573d29c7ea7f7d675cf750ce/crypto/evp/evp_enc.c#L408), [`EVP_CIPHER_CTX_set_params`](https://github.com/openssl/openssl/blob/19cc035b6c6f2283573d29c7ea7f7d675cf750ce/crypto/evp/evp_enc.c#L1266), and [`EVP_CIPHER_CTX_get_params`](https://github.com/openssl/openssl/blob/19cc035b6c6f2283573d29c7ea7f7d675cf750ce/crypto/evp/evp_enc.c#L1275) are now taking more cycles. **Problem:** These APIs end up in the [`ossl_gcm_get_ctx_params`](https://github.com/openssl/openssl/blob/19cc035b6c6f2283573d29c7ea7f7d675cf750ce/providers/implementations/ciphers/ciphercommon_gcm.c#L143) and [`ossl_gcm_set_ctx_params`](https://github.com/openssl/openssl/blob/19cc035b6c6f2283573d29c7ea7f7d675cf750ce/providers/implementations/ciphers/ciphercommon_gcm.c#L230) functions, which seem to be the bottleneck. These functions call [`OSSL_PARAM_LOCATE`](https://github.com/openssl/openssl/blob/19cc035b6c6f2283573d29c7ea7f7d675cf750ce/crypto/params.c#L27) for every possible parameter. For every call to `OSSL_PARAM_LOCATE`, we are iterating over the parameters and doing a string comparison that checks if the parameter key matches the one we are looking for. Summing up, every time we set a parameter, we end up doing a bunch of `strcmp`. **Alternative**: The performance would be greatly improved if the key were an integer instead of a string. **Side note**: Another way that performance could be improved is by skipping the call to `OPENSSL_PARAM_LOCATER`. Instead, iterating (once) over all parameters, checking for each parameter its key.

github.com/openssl/openssl

3.0 performance degraded due to locking

opened 05:39PM - 14 Feb 23 UTC

richsalz

branch: master help wanted triaged: bug triaged: refactor branch: 3.0 branch: 3.1 branch: 3.2 triaged: performance

Using some internal QUIC tests, we see this timing in 1.1: ``` $ USE_GQUIC_VER…SIONS=1 perf record -F 999 -g ./quic_lib_tests --gtest_repeat=100 --gtest_filter=*ZeroRttDisabled* $ perf report + 31.73% 31.55% quic_lib_tests quic_lib_tests [.] bn_sqr8x_internal + 9.28% 9.28% quic_lib_tests quic_lib_tests [.] mul4x_internal + 4.91% 4.91% quic_lib_tests quic_lib_tests [.] sha256_block_data_order_avx ``` In 3.0 we see this: ``` $ USE_GQUIC_VERSIONS=1 perf record -F 999 -g ./quic_lib_tests --gtest_repeat=100 --gtest_filter=*ZeroRttDisabled* $ perf report + 11.02% 10.99% quic_lib_tests quic_lib_tests [.] bn_sqr8x_internal + 8.38% 8.08% quic_lib_tests libpthread-2.31.so [.] __pthread_rwlock_rdlock + 7.65% 7.51% quic_lib_tests libpthread-2.31.so [.] __pthread_rwlock_unlock + 4.98% 4.78% quic_lib_tests quic_lib_tests [.] getrn + 4.14% 4.11% quic_lib_tests quic_lib_tests [.] mul4x_internal + 3.37% 2.57% quic_lib_tests quic_lib_tests [.] ossl_tolower 3.30% 3.30% quic_lib_tests quic_lib_tests [.] ossl_lh_strcasehash + 2.72% 2.13% quic_lib_tests quic_lib_tests [.] OPENSSL_strcasecmp + 2.29% 2.05% quic_lib_tests quic_lib_tests [.] ossl_lib_ctx_get_data + 1.93% 1.93% quic_lib_tests quic_lib_tests [.] sha256_block_data_order_avx ``` This seems to be part of `OSSL_DECODER_CTX_new_for_pkey` 16% of the time is spent in locking, on a *single threaded binary.* And 10% is in a string hashtable lookup. If anyone on the project is going to look at this, I will try to get a small reproducer. But our the time for our QUIC tests is doubling.

And openssl 3.2 suggests significant performance improvement over 3.0. So, we wanted to try that out and see if it resolves the link failure issues with ssl enabled users on MariaDB.

jlehtone · September 11, 2024, 5:56am

In other words, you can create an instance of CentOS Stream 9 now. Add copy of database there and test. If that solves your issue, then you know that future Rocky 9 probably improves your production system. If Stream 9 with openssl 3.2.2 is as bad as current, then tell Red Hat that they have an issue.

raven · September 11, 2024, 7:41am

Well, it sounds like you have to wait for 9.5

gerry666uk · September 11, 2024, 9:44pm

Is this a web application where the “user” is part of a connection attempt to MariaDB?

What exactly happens at the point of the “link failure”, does the web application give an error, does the database give an error, what exact errors do they give?

iwalker · September 12, 2024, 6:35am

Other than what @gerry666uk asked, MariaDB installed from where? Rocky repositories or the community version provided by MariaDB?

techgeek93 · September 13, 2024, 7:36am

@iwalker Yes, we are using repository provided by MariaDB (10.5.18) since we are migrating from the same version in centos.
baseurl = Index of /mariadb-10.5.18/yum/rhel8-amd64/

@gerry666uk this is a java application using grails to connect to mariadb. The page shows an error and in log we see communication link failure error when there is heavy request on the application for the ssl enabled user.
Errors like below are seen:Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
…
Caused by: java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848

iwalker · September 13, 2024, 7:51am

Wouldn’t it be better to use the version of MariaDB included in Rocky?

Name         : mariadb-server
Epoch        : 3
Version      : 10.5.22
Release      : 1.el9_2
Architecture : x86_64
Size         : 9.6 M
Source       : mariadb-10.5.22-1.el9_2.src.rpm
Repository   : Rocky_Linux_9_AppStream

this version is 10.5.22 so slightly newer than your 10.5.18 that you are using from MariaDB community.

If MariaDB is the problem and you are using their community version, it might be prudent to ask your questions on the MariaDB forum if something is not working when SSL is enabled. SSL should work irrespective of whether it’s OpenSSL 3.0 or 3.2 - it most likely could be a problem with MariaDB configuration when SSL is enabled.

If needing a newer version of OpenSSL you would need to find a Linux distro that has that by default. Even Debian Stable 12.x only has OpenSSL 3.0.14 currently.

techgeek93 · September 16, 2024, 4:05am

We tried with rocky linux 8 with the same mariadb version 10.5.18. Since, rocky linux 8 has openssl 1.1 we did not face any communication link failure issue there. So, the openssl 3.0 on rocky linux 9 seems to be have issues when using ssl in mariadb.
In centos as well there is openssl 1.0 so we did not face any issues there as well.
When we switched again to rocky linux 9.4 we started to see those link failures again with ssl enabled users. Hence, we were looking for a way to upgrade/downgrade openssl on rocky 9 to see if that solves the issue.

nazunalika · September 16, 2024, 5:15am

I’m going to mirror what @iwalker asked, which is, wouldn’t it be better to use the MariaDB we provide? Have you tested using our MariaDB? I would really recommend doing this on a test machine, using the same configuration and perhaps a copy of the data you have.

Long and the short: If you experience the same issues using the current openssl version, then yes, it will be our openssl version. If you are feeling adventurous, you can install CentOS Stream 9 in a test machine also and try to replicate your setup there to see if openssl 3.2.2 is the “cure”.

If you must use a MariaDB version that we do not provide, you may want to ask that community for advice. We do not know their package build roots or what they’re doing to build their packages.

While I understand testing this way helps prove the case to you because you can see that it works “fine” on an older version of Rocky Linux, it does not prove the case to everyone else (even with github issues that point to similar behavior). This is why I ask that you use the MariaDB that we provide in our repos and see if openssl really is causing an issue here.

With all of this being said, after careful reading and looking through PR’s, branches, and other github issues while monitoring the activity of this thread, I’ll give you the benefit of the doubt that our openssl version may be related.

And let’s say our openssl version is the issue regardless of where you got mariadb from. If it is, you can either wait for Rocky Linux 9.5 to be released in November or you can install CentOS Stream 9 (assuming in your tests the issue is not there).

In conclusion: Just like the others have stated: upgrading openssl will never be recommended. This will break your system. As such, I recommend standing up test machines using the parameters I have presented earlier. Changing the openssl version we provide in any way will never be recommended and doing so against any advice provided here, you risk breaking your production system in which you get to keep the pieces.