I would like to set up glusterfs for /home and have secure communication. Two (or maybe later three) systems (server-011, server-033), both need to be simultaneously server and client.
The problem seems to be the SSL connection, what I have done so far:
I looked at these two FAQs:
As a starting point:
[root@server-011 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─rl-root 253:0 0 70G 0 lvm /
├─rl-swap 253:1 0 3G 0 lvm [SWAP]
└─rl-home 253:2 0 126G 0 lvm /home
sdb 8:16 0 100G 0 disk
└─sdb1 8:17 0 100G 0 part
└─rl-home 253:2 0 126G 0 lvm /home
sr0 11:0 1 1024M 0 rom
I want to use the sdb drive for Glusterfs, so the partitions are adjusted accordingly:
[root@server-033 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─rl-root 253:0 0 70G 0 lvm /
├─rl-swap 253:1 0 3G 0 lvm [SWAP]
└─rl-home 253:2 0 26G 0 lvm /home
sdb 8:16 0 100G 0 disk
└─sdb1 8:17 0 100G 0 part
└─rl-ghome 253:3 0 100G 0 lvm
sr0 11:0 1 1024M 0 rom
glusterfs:
yum install centos-release-gluster -y
yum --enablerepo=crb install python3-pyxattr -y
yum install glusterfs-server -y
fstab:
/dev/mapper/rl-home /homeBackup xfs defaults 0 0
/dev/mapper/rl-ghome /mnt/glusterfs-data xfs defaults 0 0
localhost:/home /home glusterfs defaults,_netdev 0 0
firewall
firewall-cmd --zone=public --add-service=glusterfs --permanent
I needed to use force by creating the home volume (“parent directory is already part of a volume”):
gluster volume create home replica 2 transport tcp server-011:/mnt/glusterfs-data/gluster-brick server-033:/mnt/glusterfs-data/gluster-brick force
At this point, the volume is working, but I need a secure communication and only authorized servers should communicate to each other.
SSL setup:
cd /etc/ssl/
openssl genrsa -out glusterfs.key 2048
openssl req -new -x509 -key glusterfs.key -subj "/CN=${HOSTNAME}" -out glusterfs.pem -days 3650
only on server-033:
scp root@server-011:/etc/ssl/glusterfs.pem /etc/ssl/server-011.pem
cat /etc/ssl/glusterfs.pem /etc/ssl/server-011.pem > /etc/ssl/glusterfs.ca
only on server-011:
scp root@server-033:/etc/ssl/glusterfs.pem /etc/ssl/server-033.pem
cat /etc/ssl/glusterfs.pem /etc/ssl/server-033.pem > /etc/ssl/glusterfs.ca
touch /var/lib/glusterd/secure-access
systemctl restart glusterfsd.service
systemctl restart glusterd.service
Somehow I need the glusterfs.key on another path, than in the FAQ. glusterd.log:
E [socket.c:4241:ssl_setup_connection_params] 0-management: could not load private key at /etc/pki/tls/glusterfs.key
I created:
ln -s /etc/ssl/glusterfs.pem /etc/pki/tls/glusterfs.pem
ln -s /etc/ssl/glusterfs.key /etc/pki/tls/glusterfs.key
The peers are still not connected:
[root@server-033 glusterfs]# gluster peer status
Number of Peers: 1
Hostname: server-011
Uuid: 342181d2-0a34-4e96-b75c-3d4048ba6529
State: Peer in Cluster (Disconnected)
New error from the gluster logfile:
[2024-03-26 14:45:42.475243 +0000] I [socket.c:4107:ssl_setup_connection_params] 0-management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.56.104:24007
[2024-03-26 14:45:42.486170 +0000] E [socket.c:4249:ssl_setup_connection_params] 0-management: could not load CA list
[2024-03-26 14:45:42.486828 +0000] I [socket.c:4107:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.56.104:49151
[2024-03-26 14:45:42.492996 +0000] E [socket.c:4249:ssl_setup_connection_params] 0-socket.management: could not load CA list
[2024-03-26 14:45:42.493142 +0000] E [socket.c:222:ssl_dump_error_stack] 0-socket.management: error:0A000126:SSL routines::unexpected eof while reading
[2024-03-26 14:45:45.495282 +0000] I [socket.c:4107:ssl_setup_connection_params] 0-management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.56.104:24007
[2024-03-26 14:45:45.498217 +0000] E [socket.c:4249:ssl_setup_connection_params] 0-management: could not load CA list
If I check the ssl cert:
openssl s_client -showcerts -connect server-011:24007 -servername server-011
...
read R BLOCK
80CBF0E6997F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:320:
What did I do wrong in the configuration?