Kubernetes,  Linux

在kubernetes cluster上部署NGINX服务流程及错误解决

一创建deployment

[root@master-node ~]# kubectl get nodes
NAME         STATUS   ROLES                 AGE   VERSION
master-node   Ready   control-plane,master   18h   v1.22.3
node-1       Ready   <none>                 17h   v1.22.3
node-2       Ready   <none>                 17h   v1.22.3
[root@master-node ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master-node ~]# kubectl get deployments
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/1     1            0           9s
[root@master-node ~]#

二创建service

[root@master-node ~]# kubectl get svc
NAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1   <none>        443/TCP   19h
[root@master-node ~]# kubectl create service nodeport nginx --tcp=80:80
service/nginx created
[root@master-node ~]# kubectl get svc
NAME         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)       AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP       19h
nginx       NodePort    10.109.243.29   <none>        80:32201/TCP   8s
[root@master-node ~]#

三访问报错

[root@master-node ~]# kubectl get svc
NAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)       AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP       3h50m
nginx       NodePort    10.97.11.68   <none>        80:32610/TCP   5s
[root@master-node ~]# curl cluster-node:32610
curl: (6) Could not resolve host: cluster-node; 未知的错误
[root@master-node ~]#

四排查pods信息

kubectl get pods;

kubectl describe pods nginx;

[root@master-node ~]# kubectl get pods
NAME                     READY   STATUS             RESTARTS   AGE
nginx-6799fc88d8-qmwcv   0/1     ContainerCreating   0         3h45m
[root@master-node ~]#
[root@master-node ~]# kubectl describe pods nginx
Name:           nginx-6799fc88d8-qmwcv
Namespace:     default
Priority:       0
Node:           node-2/172.16.11.161
Start Time:     Thu, 04 Nov 2021 11:44:46 +0800
Labels:         app=nginx
               pod-template-hash=6799fc88d8
Annotations:   <none>
Status:         Pending
IP:            
IPs:           <none>
Controlled By: ReplicaSet/nginx-6799fc88d8
Containers:
nginx:
  Container ID:  
  Image:         nginx
  Image ID:      
  Port:           <none>
  Host Port:     <none>
  State:         Waiting
    Reason:       ContainerCreating
  Ready:         False
  Restart Count:  0
  Environment:   <none>
  Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2p27l (ro)
Conditions:
Type             Status
Initialized       True
Ready             False
ContainersReady   False
PodScheduled     True
Volumes:
kube-api-access-2p27l:
  Type:                   Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds:  3607
  ConfigMapName:           kube-root-ca.crt
  ConfigMapOptional:       <nil>
  DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:             <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                            node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type     Reason                 Age                       From     Message
 ----     ------                  ----                      ----     -------
Warning FailedCreatePodSandBox 11m (x12195 over 3h45m)   kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f7f19384ea91a37dee5d7314b9899c456685ab8b8c29a7075926c779e60bbd66" network for pod "nginx-6799fc88d8-qmwcv": networkPlugin cni failed to set up pod "nginx-6799fc88d8-qmwcv_default" network: open /run/flannel/subnet.env: no such file or directory
Normal   SandboxChanged         6m4s (x12477 over 3h46m) kubelet Pod sandbox changed, it will be killed and re-created.

[root@master-node ~]#

Warning FailedCreatePodSandBox 11m (x12195 over 3h45m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “f7f19384ea91a37dee5d7314b9899c456685ab8b8c29a7075926c779e60bbd66” network for pod “nginx-6799fc88d8-qmwcv”: networkPlugin cni failed to set up pod “nginx-6799fc88d8-qmwcv_default” network: open /run/flannel/subnet.env: no such file or directory Normal SandboxChanged 6m4s (x12477 over 3h46m) kubelet Pod sandbox changed, it will be killed and re-created.

分析原因,网络哪儿配置错误,导致的。具体本质原因,我还不是很清楚,根据网络上找的资料,重新在2个worker节点,执行:kubeadm reset;重置节点,最后再在master节点上执行该命令。

接下来,通过下述命令:重新初始化master cluster,并把2个worker节点加入:

[root@master-node ~]# kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks
....
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.11.106:6443 --token lq88wp.ynfo1ulqxelz5r0b \
       --discovery-token-ca-cert-hash sha256:7cfb1263ac6e73bf790a300e314b37a5ac7ef006ff7d799ff44e566b3e0d3db8
[root@master-node ~]#

依然报同样的错误!!!

五解决错误

于是,重新再次kubeadm reset;先在2个worker节点执行,以及master节点执行。

并再次,重新初始化master node:

[root@master-node ~]# kubeadm init --image-repository registry.aliyuncs.com/google_containers  --pod-network-cidr=10.244.0.0/16
..
...

然后,在worker节点上,执行join操作。

看到节点正常:

[root@master-node ~]# kubectl get nodes
NAME         STATUS   ROLES                 AGE   VERSION
master-node   Ready   control-plane,master   76s   v1.22.3
node-1       Ready   <none>                 28s   v1.22.3
node-2       Ready   <none>                 26s   v1.22.3
[root@master-node ~]#

但是,没有/run/flannel/subnet.env文件,通过重新执行:kubectl apply -f flannel.yml ,然后该命令会自动在master和所有的worker节点上,创建并生成/run/flannel/subnet.env文件。该文件的创建是自动的,但是需要稍微经过一点儿时间,才可以从文件系统上看到该文件。

[root@master-node ~]# ll /run/flannel/
总用量 0
[root@master-node ~]# kubectl apply -f flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@master-node ~]# ll /run/flannel/
总用量 0
[root@master-node ~]# ll /run/flannel/
总用量 4
-rw-r--r-- 1 root root 96 11月  4 15:52 subnet.env
[root@master-node ~]#

六重新创建deployment和service

创建deployment,并查看其状态:

[root@master-node ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master-node ~]# kubectl get deployment
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/1     1            0           10s
[root@master-node ~]# kubectl get pods
NAME                     READY   STATUS   RESTARTS   AGE
nginx-6799fc88d8-dg6d9   1/1     Running   0         32s
[root@master-node ~]# kubectl get deployment
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           45s
[root@master-node ~]#

可以看到,创建完deployment之后,并不能马上看到其是ready的状态,稍后几秒钟,才能看到其状态变为ready。

创建service,及查看其状态:

[root@master-node ~]# kubectl get svc
NAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1   <none>        443/TCP   9m15s
[root@master-node ~]# kubectl create service nodeport nginx --tcp=80:80
service/nginx created
[root@master-node ~]# kubectl get svc
NAME         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)       AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP       9m53s
nginx       NodePort    10.108.50.59   <none>        80:30769/TCP   5s
[root@master-node ~]#

七访问服务:

从上,可以看到对应的NGINX服务运行在30769端口上了。

[root@master-node ~]# curl master-node:30769
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master-node ~]#

同时:

curl master-node:30769
curl node-1:30769
curl node-2:30769

都可以正常访问。再次describe pods的信息,不再报错:

[root@master-node ~]# kubectl describe pods nginx
Name:         nginx-6799fc88d8-dg6d9
Namespace:   default
Priority:     0
Node:         node-2/172.16.11.161
Start Time:   Thu, 04 Nov 2021 15:56:49 +0800
Labels:       app=nginx
             pod-template-hash=6799fc88d8
Annotations: <none>
Status:       Running
IP:           10.244.2.2
IPs:
IP:           10.244.2.2
Controlled By: ReplicaSet/nginx-6799fc88d8
Containers:
nginx:
  Container ID:   docker://0693af01c2b7246e25394f458b12a530f93543b42d22cf4d5984f00fe1661f71
  Image:         nginx
  Image ID:       docker-pullable://docker.io/nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36
  Port:           <none>
  Host Port:     <none>
  State:         Running
    Started:     Thu, 04 Nov 2021 15:57:10 +0800
  Ready:         True
  Restart Count:  0
  Environment:   <none>
  Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8hk5t (ro)
Conditions:
Type             Status
Initialized       True
Ready             True
ContainersReady   True
PodScheduled     True
Volumes:
kube-api-access-8hk5t:
  Type:                   Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds:  3607
  ConfigMapName:           kube-root-ca.crt
  ConfigMapOptional:       <nil>
  DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:             <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                            node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type   Reason     Age   From               Message
 ----    ------     ----  ----               -------
Normal Scheduled 20m   default-scheduler Successfully assigned default/nginx-6799fc88d8-dg6d9 to node-2
Normal Pulling   20m   kubelet           Pulling image "nginx"
Normal Pulled     20m   kubelet           Successfully pulled image "nginx" in 19.846688268s
Normal Created   20m   kubelet           Created container nginx
Normal Started   20m   kubelet           Started container nginx

[root@master-node ~]#

当然,通过browser,也都可以正常访问:

http://172.16.11.106:30769/ http://172.16.11.148:30769/ http://172.16.11.161:30769/

八 小结

  • 上一章节里安装的kubernetes cluster是存在问题的,因为创建的服务不能正常使用;
  • 解决办法是重新初始化并创建kubernetes cluster并带上–pod-network-cidr=10.244.0.0/16选项;
  • 然后重新配置网络组件kubectl apply -f flannel.yml;该命令在master节点执行,并且会自动在所有节点创建/run/flannel/subnet.env文件;
  • 查看pods和deployment是否可以正常使用,可以通过kubectl get pods|deployments,查看READY列状态数字是不是正常,如果是0/x,则肯定是用不了的;

kubernetes把服务暴露出来的几种方式:

Kubernetes offers several options when exposing your service based on a feature called Kubernetes Service-types and they are:

  1. ClusterIP – This Service-type generally exposes the service on an internal IP, reachable only within the cluster, and possibly only within the cluster-nodes.
  2. NodePort – This is the most basic option of exposing your service to be accessible outside of your cluster, on a specific port (called the NodePort) on every node in the cluster. We will illustrate this option shortly.
  3. LoadBalancer – This option leverages on external Load-Balancing services offered by various providers to allow access to your service. This is a more reliable option when thinking about high availability for your service, and has more feature beyond default access.
  4. ExternalName – This service does traffic redirect to services outside of the cluster. As such the service is thus mapped to a DNS name that could be hosted out of your cluster. It is important to note that this does not use proxying.

The default Service-type is ClusterIP.

九 参考链接

How to Deploy Nginx on a Kubernetes Cluster

Pods are not starting. NetworkPlugin cni failed to set up pod