Kubernetes证书过期测试

由于工作需要，需要验证Kubernetes集群证书过期后的续期流程。因为之前没有做过，特此记录。

修改并编译 kubeadm 源码

在构建环境中安装 go 和 git，并将 go 的 bin 路径加入 PATH。
下载 Kubernetes 源码，这里以 v1.18.18 为例。
git clone -b v1.18.18 https://github.com/kubernetes/kubernetes

修改以下文件，将证书有效期设置为 10 分钟：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
diff --git a/cmd/kubeadm/app/constants/constants.go b/cmd/kubeadm/app/constants/constants.go
index b56891ca908..eed934280e7 100644
--- a/cmd/kubeadm/app/constants/constants.go
+++ b/cmd/kubeadm/app/constants/constants.go
@@ -46,7 +46,8 @@ const (
TempDirForKubeadm = "tmp"

// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
- CertificateValidity = time.Hour * 24 * 365
+ //CertificateValidity = time.Hour * 24 * 365
+ CertificateValidity = time.Second * 600

// CACertAndKeyBaseName defines certificate authority base name
CACertAndKeyBaseName = "ca"
diff --git a/staging/src/k8s.io/client-go/util/cert/cert.go b/staging/src/k8s.io/client-go/util/cert/cert.go
index 9fd097af5e3..64e1dd90f43 100644
--- a/staging/src/k8s.io/client-go/util/cert/cert.go
+++ b/staging/src/k8s.io/client-go/util/cert/cert.go
@@ -35,7 +35,8 @@ import (
"k8s.io/client-go/util/keyutil"
)

-const duration365d = time.Hour * 24 * 365
+//const duration365d = time.Hour * 24 * 365
+const duration365d = time.Second * 600

// Config contains the basic fields required for creating a certificate
type Config struct {
@@ -93,7 +94,8 @@ func GenerateSelfSignedCertKey(host string, alternateIPs []net.IP, alternateDNS
// Certs/keys not existing in that directory are created.
func GenerateSelfSignedCertKeyWithFixtures(host string, alternateIPs []net.IP, alternateDNS []string, fixtureDirectory string) ([]byte, []byte, error) {
validFrom := time.Now().Add(-time.Hour) // valid an hour earlier to avoid flakes due to clock skew
- maxAge := time.Hour * 24 * 365 // one year self-signed certs
+ //maxAge := time.Hour * 24 * 365 // one year self-signed certs
+ maxAge := time.Second * 600 // one year self-signed certs

baseName := fmt.Sprintf("%s_%s_%s", host, strings.Join(ipsToStrings(alternateIPs), "-"), strings.Join(alternateDNS, "-"))
certFixturePath := path.Join(fixtureDirectory, baseName+".crt")

使用如下命令仅编译 kubeadm：
make WHAT=cmd/kubeadm GOFLAGS=-v
编译完成后会生成 _output 文件夹，kubeadm 二进制文件在 bin 子目录下。使用该二进制部署集群时，证书有效期为 10 分钟。

检查与续期证书

搭建 3 节点集群，确认证书剩余有效期约 4 分钟：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
root@wenhan-adm-cp:~# ./kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 24, 2021 06:01 UTC   4m                                      no
apiserver                  Jun 24, 2021 06:01 UTC   4m              ca                      no
apiserver-etcd-client      Jun 24, 2021 06:01 UTC   4m              etcd-ca                 no
apiserver-kubelet-client   Jun 24, 2021 06:01 UTC   4m              ca                      no
controller-manager.conf    Jun 24, 2021 06:01 UTC   4m                                      no
etcd-healthcheck-client    Jun 24, 2021 06:01 UTC   4m              etcd-ca                 no
etcd-peer                  Jun 24, 2021 06:01 UTC   4m              etcd-ca                 no
etcd-server                Jun 24, 2021 06:01 UTC   4m              etcd-ca                 no
front-proxy-client         Jun 24, 2021 06:01 UTC   4m              front-proxy-ca          no
scheduler.conf             Jun 24, 2021 06:01 UTC   4m                                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 24, 2021 07:31 UTC   1h              no
etcd-ca                 Jun 24, 2021 07:31 UTC   1h              no
front-proxy-ca          Jun 24, 2021 07:31 UTC   1h              no

root@wenhan-adm-cp:~# kubectl get node
NAME             STATUS   ROLES    AGE     VERSION
wenhan-adm-cp    Ready    master   4m35s   v1.18.18
wenhan-adm-wk1   Ready    <none>   3m32s   v1.18.18
wenhan-adm-wk2   Ready    <none>   3m28s   v1.18.18

证书过期后，kubectl get node 命令会失败：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
root@wenhan-adm-cp:~# ./kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

W0624 06:01:22.231290   13746 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 24, 2021 06:01 UTC   <invalid>                               no
apiserver                  Jun 24, 2021 06:01 UTC   <invalid>       ca                      no
apiserver-etcd-client      Jun 24, 2021 06:01 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Jun 24, 2021 06:01 UTC   <invalid>       ca                      no
controller-manager.conf    Jun 24, 2021 06:01 UTC   <invalid>                               no
etcd-healthcheck-client    Jun 24, 2021 06:01 UTC   <invalid>       etcd-ca                 no
etcd-peer                  Jun 24, 2021 06:01 UTC   <invalid>       etcd-ca                 no
etcd-server                Jun 24, 2021 06:01 UTC   <invalid>       etcd-ca                 no
front-proxy-client         Jun 24, 2021 06:01 UTC   <invalid>       front-proxy-ca          no
scheduler.conf             Jun 24, 2021 06:01 UTC   <invalid>                               no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 24, 2021 07:31 UTC   1h              no
etcd-ca                 Jun 24, 2021 07:31 UTC   1h              no
front-proxy-ca          Jun 24, 2021 07:31 UTC   1h              no

root@wenhan-adm-cp:~# kubectl get node
Unable to connect to the server: x509: certificate has expired or is not yet valid

续期证书并确认新的有效期：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
root@wenhan-adm-cp:~# ./kubeadm alpha certs renew all --config=kubeadm.yaml
W0624 06:04:00.447612   2860 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

续期后，确认各证书的有效期已更新：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
root@wenhan-adm-cp:~# ./kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

W0624 06:04:21.176808   3230 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE         EXPIRES          RESIDUAL TIME  CERTIFICATE AUTHORITY  EXTERNALLY MANAGED
admin.conf         Jun 24, 2021 06:14 UTC  9m                    no
apiserver          Jun 24, 2021 06:14 UTC  9m        ca            no
apiserver-etcd-client    Jun 24, 2021 06:14 UTC  9m        etcd-ca         no
apiserver-kubelet-client  Jun 24, 2021 06:14 UTC  9m        ca            no
controller-manager.conf   Jun 24, 2021 06:14 UTC  9m                    no
etcd-healthcheck-client   Jun 24, 2021 06:14 UTC  9m        etcd-ca         no
etcd-peer          Jun 24, 2021 06:14 UTC  9m        etcd-ca         no
etcd-server         Jun 24, 2021 06:14 UTC  9m        etcd-ca         no
front-proxy-client     Jun 24, 2021 06:14 UTC  9m        front-proxy-ca      no
scheduler.conf       Jun 24, 2021 06:14 UTC  9m                    no

CERTIFICATE AUTHORITY  EXPIRES          RESIDUAL TIME  EXTERNALLY MANAGED
ca            Jun 24, 2021 07:31 UTC  1h        no
etcd-ca         Jun 24, 2021 07:31 UTC  1h        no
front-proxy-ca      Jun 24, 2021 07:31 UTC  1h        no

续期证书后，重启控制平面：

1
root@wenhan-adm-cp:~# reboot

证书续期完成。

重启后可用新的认证文件访问集群：

1
2
3
4
5
6
7
8
root@wenhan-adm-cp:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? yes

root@wenhan-adm-cp:~# kubectl get node
NAME       STATUS  ROLES   AGE  VERSION
wenhan-adm-cp   Ready   master  14m  v1.18.18
wenhan-adm-wk1  Ready   <none>  13m  v1.18.18
wenhan-adm-wk2  Ready   <none>  13m  v1.18.18

修改并编译 kubeadm 源码#

检查与续期证书#

修改并编译 kubeadm 源码

检查与续期证书