潇湘夜雨移动版

kubernetes运行一段时间后出现如下故障：

 

May  2 11:15:12 master systemd: Started Kubernetes systemd probe.

May  2 11:15:12 master systemd: Starting Kubernetes systemd probe.

May  2 11:15:12 master kubelet: I0502 11:15:12.773791    3858 server.go:407] Version: v1.13.0

May  2 11:15:12 master kubelet: I0502 11:15:12.774404    3858 plugins.go:103] No cloud provider specified.

May  2 11:15:12 master kubelet: E0502 11:15:12.779283    3858 bootstrap.go:209] Part of the existing bootstrap client certificate is expired: 2019-12-12 15:11:57 +0000 UTC

May  2 11:15:12 master kubelet: F0502 11:15:12.779320    3858 server.go:261] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such

 file or directoryMay  2 11:15:12 master systemd: kubelet.service: main process exited, code=exited, status=255/n/a

 

原因:kubeadm 部署集群默认证书有效期为一年

 

[root@master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '

            Not Before: Dec 12 15:11:55 2018 GMT

            Not After : Dec 12 15:11:56 2019 GMT

 

The connection to the server 192.168.0.20:6443 was refused - did you specify the right host or port?

[root@master ~]# ls /etc/kubernetes/pki/

apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  etcd                front-proxy-ca.key      front-proxy-client.key  sa.pub

apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key

[root@master ~]# cp -r /etc/kubernetes /etc/kubernetes-bak

 

rm -rf $HOME/.kube

[root@master ~]# mkdir -p $HOME/.kube

[root@master ~]# cp -i /etc/kubernetes/admin.conf /root/.kube/config

 

[root@master ~]# rm -rf /etc/kubernetes/*.conf

[root@master ~]# kubeadm init phase kubeconfig all

I0502 11:45:07.945775    1717 version.go:94] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleap

is.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)I0502 11:45:07.945934    1717 version.go:95] falling back to the local client version: v1.13.0

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

[kubeconfig] Writing "admin.conf" kubeconfig file

[kubeconfig] Writing "kubelet.conf" kubeconfig file

[kubeconfig] Writing "controller-manager.conf" kubeconfig file

[kubeconfig] Writing "scheduler.conf" kubeconfig file

 

更新节点证书

 

[root@master ~]# kubeadm token create --print-join-command

kubeadm join 192.168.0.20:6443 --token lasfnx.fppsi34dk6iu6hdd --discovery-token-ca-cert-hash sha256:079253fc46bf505da4732ef306a73c4dd0d954eff2d0ca52f3ade4f4e930a6dd

 

cp -r /etc/kubernetes /etc/kubernetes-bak

rm -rf /etc/kubernetes/*.conf

rm -rf /etc/kubernetes/pki/ca.crt

 

[root@node2 ~]# kubeadm join 192.168.0.20:6443 --token lasfnx.fppsi34dk6iu6hdd --discovery-token-ca-cert-hash sha256:079253fc46bf505da4732ef306a73c4dd0d954eff2d0ca52f3ade4f

4e930a6dd[preflight] Running pre-flight checks

[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06

[WARNING Hostname]: hostname "node2.k8s" could not be reached

[WARNING Hostname]: hostname "node2.k8s": lookup node2.k8s on 114.114.114.114:53: no such host

[discovery] Trying to connect to API Server "192.168.0.20:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.20:6443"

[discovery] Requesting info from "https://192.168.0.20:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.0.20:6443"

[discovery] Successfully established connection with API Server "192.168.0.20:6443"

[join] Reading configuration from the cluster...

[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Activating the kubelet service

[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...

[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node2.k8s" as an annotation

 

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.

* The Kubelet was informed of the new secure connection details.

 

Run 'kubectl get nodes' on the master to see this node join the cluster.