Possible SSL issue with AWS S3?

Hi everyone,

Happy new year.
Thanks for proposing such a good distro.
I’m running RL9 and face some SSL issues.
They happened when I was using ansible so I’ve started a thread here : Ansible Galaxy installation fail on some AWS IP / OpenSSL3.X issue · Issue #79640 · ansible/ansible · GitHub

Then I’ve dig in the openssl repo and here is the result of my tests below :
I’ve tested with the native openssl version then build 3.0.7 and also compile the 3.1 release.
I’ve got alwas the following message :

OpenSSL 3.1.0-beta1 21 Dec 2022 (Library: OpenSSL 3.1.0-beta1 21 Dec 2022)
[root@rundeck bin]# ./openssl s_client -debug -connect ansible-galaxy.s3.amazonaws.com:443 
CONNECTED(00000003)
write to 0x24e32b0 [0x24f6ba0] (333 bytes => 333 (0x14D))
0000 - 16 03 01 01 48 01 00 01-44 03 03 15 5c fb b1 45   ....H...D...\..E
0010 - a8 74 1c bd 0a b0 19 29-48 ba 36 0a 18 c7 0d 62   .t.....)H.6....b
0020 - 3e dd db 89 36 c8 f6 ff-85 7c 14 20 da e8 c1 e0   >...6....|. ....
0030 - bf 8e af db b3 72 61 e3-c2 be fb 73 4a 1e 01 b8   .....ra....sJ...
0040 - 4c 38 67 fd ce 6f fe da-16 89 e6 0e 00 3e 13 02   L8g..o.......>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa   .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27   .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d   .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 bd   ...=.<.5./......
0090 - 00 00 00 24 00 22 00 00-1f 61 6e 73 69 62 6c 65   ...$."...ansible
00a0 - 2d 67 61 6c 61 78 79 2e-73 33 2e 61 6d 61 7a 6f   -galaxy.s3.amazo
00b0 - 6e 61 77 73 2e 63 6f 6d-00 0b 00 04 03 00 01 02   naws.com........
00c0 - 00 0a 00 16 00 14 00 1d-00 17 00 1e 00 19 00 18   ................
00d0 - 01 00 01 01 01 02 01 03-01 04 00 23 00 00 00 16   ...........#....
00e0 - 00 00 00 17 00 00 00 0d-00 2a 00 28 04 03 05 03   .........*.(....
00f0 - 06 03 08 07 08 08 08 09-08 0a 08 0b 08 04 08 05   ................
0100 - 08 06 04 01 05 01 06 01-03 03 03 01 03 02 04 02   ................
0110 - 05 02 06 02 00 2b 00 05-04 03 04 03 03 00 2d 00   .....+........-.
0120 - 02 01 01 00 33 00 26 00-24 00 1d 00 20 4e 0c bd   ....3.&.$... N..
0130 - 21 d4 5a 11 b3 0d de 4a-fc ac 25 7c 4c a3 bd b4   !.Z....J..%|L...
0140 - 8e ea 65 df 8f 5f f4 15-21 a9 8c 38 66            ..e.._..!..8f
read from 0x24e32b0 [0x24ed983] (5 bytes => 0)
write to 0x24e32b0 [0x24f6ba0] (7 bytes => 7 (0x7))
0000 - 15 03 01 00 02 02 32                              ......2
805B9E32657F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:321:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 340 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x24e32b0 [0x249e970] (8192 bytes => 0)

And finally I’ve made some really interessant tests here : Ansible Galaxy installation fail on some AWS IP / OpenSSL3.X issue · Issue #79640 · ansible/ansible · GitHub

The main Idea is that I’m facing unexpected EOF while reading SSL issue randomly on something that could look like specific IPs from AWS S3.

I suppose that something regarding the way IP are resolved could cause teh trouble and wondering if there were any setup in RL9 that could prevent this.

Any advice and guidance for additionnal tests will be appreciated.

Regards

Testing the exact same command from Rocky 9.1, I am able to connect straight away without any errors.

Hi @gerry666uk thanks for your test.
It happens to work but not with all IPs.
For example, it fail when IP returned is 54.231.199.193
and succeed when IP is 3.5.1.131 Ansible Galaxy installation fail on some AWS IP / OpenSSL3.X issue · Issue #79640 · ansible/ansible · GitHub
If you try wget https://galaxy.ansible.com/download/community-postgresql-2.3.2.tar.gz multiple times you’ll probably face the same issue and share the IP.

I think you’re saying that:

can resolve to more that one IP, and that some work and some don’t?

If that’s the case, it should be possible to use the exact ip address in the openssl client command…

I added this to /etc/hosts and did the openssl command from above thus forcing it to a particular IP and it works on Rocky 9 today. Perhaps there was infrastructure problems at Amazon.

thanks to all of you.
The point is that I made cross tests.
On my old fedora with openssl 1.1.1k and on RL9 openssl3.0.x
I fixed IP alternatively on both OS
It succeed on F32 and fail on RL9 with the same IPs
Any chance there is a kernel option or something that prevent openssl to keep a proper connection with the IP.
I’m able to make additionnal tests if you lead me.

Anyway thanks for the time already spent to help me on this