Changing the ETCD cluster from HTTP to HTTPS connections fails

[sharbich]

Hello,
Changing the ETCD cluster from HTTP to HTTPS connections is failing. With HTTP connections, I see three clusters. With HTTPS, I only see one cluster.

Here is my ETCD HTTPS configuration:

`ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_NAME="sme01"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER="sme01=http://sme01.intern.example.com:2380,sme02=http://sme02.intern.example.com:2380,sme03=http://sme03.intern.example.com:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sme01.intern.example.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://sme01.intern.example.com:2379"
ETCD_HEARTBEAT_INTERVAL="250"
ETCD_ELECTION_TIMEOUT="2500"

ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_NAME="sme02"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER="sme01=http://sme01.intern.example.com:2380,sme02=http://sme02.intern.example.com:2380,sme03=http://sme03.intern.example.com:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sme02.intern.example.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://sme02.intern.example.com:2379"
ETCD_HEARTBEAT_INTERVAL="250"
ETCD_ELECTION_TIMEOUT="2500"

ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_NAME="sme03"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER="sme01=http://sme01.intern.example.com:2380,sme02=http://sme02.intern.example.com:2380,sme03=http://sme03.intern.example.com:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://sme03.intern.example.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://sme03.intern.example.com:2379"
ETCD_HEARTBEAT_INTERVAL="250"
ETCD_ELECTION_TIMEOUT="2500"`

`etcdctl endpoint status --write-out=table
127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.697377ms

1a73e33e519f66c3, started, sme03.intern.example.com, http://192.168.20.170:2380, http://sme03.intern.example.com:2379, false

42e3e5160eb1a1c6, started, sme02.intern.example.com, http://192.168.20.110:2380, http://sme02.intern.example.com:2379, false

98dbc7dab8368fbb, started, sme01.intern.example.com, http://192.168.20.70:2380, http://sme01.intern.example.com:2379, false
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 98dbc7dab8368fbb | 3.5.16 | 143 kB | true | false | 371 | 463 | 463 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+`

I have a self-signed CA. My ETCD version is 3.5.16. I've tried creating the certificates in several variations using the "Subject Alternative Name" (with DNS / one IP address / three cluster IP addresses). I keep getting the same result:

`etcdctl endpoint status --write-out=table

127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.058356ms
8e9e05c52164694d, started, pveme01, http://localhost:2380, http://localhost:2379, false
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 8e9e05c52164694d | 3.5.16 | 20 kB | true | false | 21 | 58 | 58 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+`

Here is my ETCD HTTPS configuration:

`ETCD_NAME: sme01
ETCD_DATA_DIR: /var/lib/etcd
ETCD_DEBUG: 1
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: sme01=https://sme01.intern.example.com:2380,sme02=https://sme02.intern.example.com:2380,sme03=https://sme03.intern.example.com:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster
ETCD_LISTEN_PEER_URLS: https://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: https://sme01.intern.example.com:2380
ETCD_ADVERTISE_CLIENT_URLS: http://sme01.intern.example.com:2379
ETCD_CERT_FILE: /etc/ssl/etcd/etcd.intern.example.com.crt
ETCD_KEY_FILE: /etc/ssl/etcd/etcd.intern.example.com.key
ETCD_PEER_TRUSTED_CA_FILE: /etc/ssl/etcd/HarbichCA.crt
ETCD_PEER_CERT_FILE: /etc/ssl/etcd/sme01.intern.example.com.crt
ETCD_PEER_KEY_FILE: /etc/ssl/etcd/sme01.intern.example.com.key
ETCD_PEER_CLIENT_CERT_AUTH: 1

ETCD_NAME: sme02
ETCD_DATA_DIR: /var/lib/etcd
ETCD_DEBUG: 1
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: sme01=https://sme01.intern.example.com:2380,sme02=https://sme02.intern.example.com:2380,sme03=https://sme03.intern.example.com:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster
ETCD_LISTEN_PEER_URLS: https://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: https://sme02.intern.example.com:2380
ETCD_ADVERTISE_CLIENT_URLS: http://sme02.intern.example.com:2379
ETCD_CERT_FILE: /etc/ssl/etcd/etcd.intern.example.com.crt
ETCD_KEY_FILE: /etc/ssl/etcd/etcd.intern.example.com.key
ETCD_PEER_TRUSTED_CA_FILE: /etc/ssl/etcd/HarbichCA.crt
ETCD_PEER_CERT_FILE: /etc/ssl/etcd/sme02.intern.example.com.crt
ETCD_PEER_KEY_FILE: /etc/ssl/etcd/sme02.intern.example.com.key
ETCD_PEER_CLIENT_CERT_AUTH: 1
ETCD_HEARTBEAT_INTERVAL="250"
ETCD_ELECTION_TIMEOUT="2500"

ETCD_NAME: sme03
ETCD_DATA_DIR: /var/lib/etcd
ETCD_DEBUG: 1
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: sme01=https://sme01.intern.example.com:2380,sme02=https://sme02.intern.example.com:2380,sme03=https://sme03.intern.example.com:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster
ETCD_LISTEN_PEER_URLS: https://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: https://sme03.intern.example.com:2380
ETCD_ADVERTISE_CLIENT_URLS: http://sme03.intern.example.com:2379
ETCD_CERT_FILE: /etc/ssl/etcd/etcd.intern.example.com.crt
ETCD_KEY_FILE: /etc/ssl/etcd/etcd.intern.example.com.key
ETCD_PEER_TRUSTED_CA_FILE: /etc/ssl/etcd/HarbichCA.crt
ETCD_PEER_CERT_FILE: /etc/ssl/etcd/sme03.intern.example.com.crt
ETCD_PEER_KEY_FILE: /etc/ssl/etcd/sme03.intern.example.com.key
ETCD_PEER_CLIENT_CERT_AUTH: 1
ETCD_HEARTBEAT_INTERVAL="250"
ETCD_ELECTION_TIMEOUT="2500"`

I'm also attaching my self-signed certificates.

All three virtual machines have the same hosts file. Only the hostname differs, being "pveme01|2|3".

127.0.0.1 localhost
BEGIN POSTGRESQL
192.168.20.70 sme01.intern.example.com sme01
192.168.20.110 sme02.intern.example.com sme02
192.168.20.170 sme03.intern.example.com sme03
END POSTGRESQL
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
BEGIN PVE
127.0.1.1 pveme03.intern.example.com pveme03
END PVE

I don't understand where my mistake is?
Greetings Stefan Harbich

[sharbich]

Hello, i was able to solve the problem. Thank you very much for your support.