From 3f3f0ee78aeef41337f781a7ad66921475a9d57b Mon Sep 17 00:00:00 2001 From: Brian Cunnie Date: Sun, 17 Apr 2022 17:08:00 -0700 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9E=20TLS=20for=20etcd:=20add=20GKE=20?= =?UTF-8?q?Node=20IPs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ns-gce is unable to join the cluster because its source IP address is the node on which its running, 34.72.45.206, and that's not included in the SANs. This commit updates the etcd certificate to one which includes the three GKE nodes' IP addresses in its SANs. This commit also includes instruction to update the certificates in the event of an IP address change. Fixes: ``` Apr 16 14:15:34 ns-aws etcd[500]: rejected connection from "34.72.45.206:43080" (error "tls: \"34.72.45.206\" does not match any of DNSNames [\"ns-aws.sslip.io\" \"ns-azure.sslip.io\" \"ns-gce.sslip.io\" \"ns-aws\" \"ns-azure\" \"ns-gce\"] (lookup ns-gce: Temporary failure in name resolution)", ServerName "ns-aws.sslip.io", IPAddresses ["127.0.0.1" "52.0.56.137" "52.187.42.158" "104.155.144.4" "::1" "2600:1f18:aaf:6900::a"], DNSNames ["ns-aws.sslip.io" "ns-azure.sslip.io" "ns-gce.sslip.io" "ns-aws" "ns-azure" "ns-gce"]) ``` --- etcd/README.md | 40 +++++++++++++++++++++++++++++++++++----- etcd/etcd.pem | 20 ++++++++++---------- 2 files changed, 45 insertions(+), 15 deletions(-) diff --git a/etcd/README.md b/etcd/README.md index 9bc6c70..2bc2f21 100644 --- a/etcd/README.md +++ b/etcd/README.md @@ -13,30 +13,60 @@ communicate over TLS, but our clients won't). - `ca-csr.json`. Again, 30 years. ```shell -cfssl gencert -initca ca-csr.json | cfssljson -bare ca +cfssl gencert -initca ca-csr.json | cfssljson -bare etcd-ca ``` -The key is saved in LastPass as `etcd-ca-key.pem` +The key is saved in LastPass as `etcd-ca-key.pem`. Let's use our newly-created CA to generate the etcd certificates. Note that we throw almost every IP address/hostname we can think of into the SANs field (why not?): ```shell +GKE_NODE_PUBLIC_IPv4=$(gcloud compute instances list --format=json | + jq -r '[.[].networkInterfaces[0].accessConfigs[0].natIP] | join(",")') PUBLIC_HOSTNAMES=ns-aws.sslip.io,ns-azure.sslip.io,ns-gce.sslip.io HOSTNAMES=ns-aws,ns-azure,ns-gce -IPv4=127.0.0.1,52.0.56.137,52.187.42.158,104.155.144.4 +IPv4=127.0.0.1,52.0.56.137,52.187.42.158,104.155.144.4,$GKE_NODE_PUBLIC_IPv4 IPv6=::1,2600:1f18:aaf:6900::a cfssl gencert \ -ca=ca.pem \ - -ca-key=ca-key.pem \ + -ca-key=etcd-ca-key.pem \ -config=ca-config.json \ -hostname=${PUBLIC_HOSTNAMES},${HOSTNAMES},${IPv4},${IPv6} \ -profile=etcd \ etcd-csr.json | cfssljson -bare etcd ``` -The key is saved in LastPass as `etcd-key.pem` +The key is saved in LastPass as `etcd-key.pem`. + +#### Generating a New Cert for a New etcd Node + +Let's say you've introduced _new_ IPv4 addresses, or that you've recreated your +GKE clusters, and all the addresses have changed, then you'll need to +regenerate the certificates: + +``` +lpass show --note etcd-ca-key.pem > etcd-ca-key.pem +lpass show --note etcd-key.pem > etcd-key.pem +GKE_NODE_PUBLIC_IPv4=$(gcloud compute instances list --format=json | + jq -r '[.[].networkInterfaces[0].accessConfigs[0].natIP] | join(",")') +PUBLIC_HOSTNAMES=ns-aws.sslip.io,ns-azure.sslip.io,ns-gce.sslip.io +HOSTNAMES=ns-aws,ns-azure,ns-gce +IPv4=127.0.0.1,52.0.56.137,52.187.42.158,104.155.144.4,$GKE_NODE_PUBLIC_IPv4 +IPv6=::1,2600:1f18:aaf:6900::a + +cfssl gencsr \ + -key=etcd-key.pem \ + -hostname=${PUBLIC_HOSTNAMES},${HOSTNAMES},${IPv4},${IPv6} \ + -cert=etcd.pem | cfssljson -bare etcd +cfssl sign \ + -ca=ca.pem \ + -ca-key=etcd-ca-key.pem \ + -config=ca-config.json \ + -profile=etcd \ + etcd.csr | cfssljson -bare etcd +``` #### Configure ns-aws.sslip.io & ns-azure.sslip.io diff --git a/etcd/etcd.pem b/etcd/etcd.pem index eaa3cf2..f3a3617 100644 --- a/etcd/etcd.pem +++ b/etcd/etcd.pem @@ -1,18 +1,18 @@ -----BEGIN CERTIFICATE----- -MIIC0TCCAnegAwIBAgIULqgqHhpeTcE8fB0LJXo4xGr284UwCgYIKoZIzj0EAwIw +MIIC5DCCAougAwIBAgIUKb3EmR8U9BN+ocPgdCFyPDxFAjIwCgYIKoZIzj0EAwIw ajELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNh biBGcmFuY2lzY28xDTALBgNVBAoTBGV0Y2QxEDAOBgNVBAsTB25vbm8uaW8xDTAL -BgNVBAMTBGV0Y2QwIBcNMjExMjMxMjIzMzAwWhgPMjA1MTEyMjQyMjMzMDBaMGox +BgNVBAMTBGV0Y2QwIBcNMjIwNDE3MjM1MTAwWhgPMjA1MjA0MDkyMzUxMDBaMGox CzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4g RnJhbmNpc2NvMQ0wCwYDVQQKEwRldGNkMRAwDgYDVQQLEwdub25vLmlvMQ0wCwYD VQQDEwRldGNkMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE9n3v0f+CsUAS0spI Hhsd/hnVoS0oyONpe5ow/zSKSdM6F0e0T1W9ZDMkfy/QyDOmSSza9Sfz0DqDLkly -xObn8qOB+DCB9TAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEG -CCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFCnmehh+oSYc2iTkIRso -TH0OMw9qMIGWBgNVHREEgY4wgYuCD25zLWF3cy5zc2xpcC5pb4IRbnMtYXp1cmUu -c3NsaXAuaW+CD25zLWdjZS5zc2xpcC5pb4IGbnMtYXdzgghucy1henVyZYIGbnMt -Z2NlhwR/AAABhwQ0ADiJhwQ0uyqehwRom5AEhxAAAAAAAAAAAAAAAAAAAAABhxAm -AB8YCq9pAAAAAAAAAAAKMAoGCCqGSM49BAMCA0gAMEUCIEq4FoOJJWE6JQa0iD0B -hPkvhfvzDKH6nDPaCCXPLyPLAiEAowpAm1yKRr5kxdxxuc9p4PQGoYxAtlA/+CjA -pxYEW7s= +xObn8qOCAQswggEHMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcD +AQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUKeZ6GH6hJhzaJOQh +GyhMfQ4zD2owgagGA1UdEQSBoDCBnYIPbnMtYXdzLnNzbGlwLmlvghFucy1henVy +ZS5zc2xpcC5pb4IPbnMtZ2NlLnNzbGlwLmlvggZucy1hd3OCCG5zLWF6dXJlggZu +cy1nY2WHBH8AAAGHBDQAOImHBDS7Kp6HBGibkASHBCJILc6HBCPAdeqHBCPoeM2H +EAAAAAAAAAAAAAAAAAAAAAGHECYAHxgKr2kAAAAAAAAAAAowCgYIKoZIzj0EAwID +RwAwRAIgb0eNfkmfwV9ws5lJ4nvKscPNvgwlZH4zb1qe3+BHmdYCIBh9J7E05UmB +MibxDG9fnsaiABUMOGtHR5o5ZdMTSw6K -----END CERTIFICATE-----