如何在CentOS 7安装配置kubernetes cluster v1.22
1环境说明
配置1个master node,2个worker node的kubernetes cluster：
| IP address | role | ||
|---|---|---|---|
| 172.16.11.106 | master node | ||
| 172.16.11.148 | worker node | ||
| 172.16.11.161 | worker node |
二 配置步骤
1 3个机器分别修改/etc/hosts:
172.16.11.106 master-node
172.16.11.148 node-1 worker-node-1
172.16.11.161 node-2 worker-node-2
2 3个机器分别关闭SELINUX
sed -i 表示原地修改文件，–follow-symlinks表示连带软连接文件一起修改，’s/xxx/yy/g’，通过正在，表示匹配到xx,则就将其修改为yy。
[root@localhost ~]# setenforce 0
[root@localhost ~]# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
[root@localhost ~]#
3 3个机器分别关闭防火墙
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]#
具体是实际情况而定，可能需要开启Firewalls，同时开放对应的端口。
4 3个机器分别添加kubernetes Repo
由于国内网络原因，使用阿里云镜像地址：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
5 3个机器分别安装kubeadm和docker
[root@localhost ~]# yum install kubeadm docker -y
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* extras: mirrors.cn99.com
* updates: mirrors.cn99.com
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
kubernetes/signature | 844 B 00:00:00
从 https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 检索密钥
导入 GPG key 0x307EA071:
用户ID : "Rapture Automatic Signing Key (cloud-rapture-signing-key-2021-03-01-08_01_09.pub)"
指纹 : 7f92 e05b 3109 3bef 5a3c 2d38 feea 9169 307e a071
来自 : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
从 https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg 检索密钥
kubernetes/signature | 1.4 kB 00:00:05 !!!
updates | 2.9 kB 00:00:00
(1/3): extras/7/x86_64/primary_db | 243 kB 00:00:00
(2/3): kubernetes/primary | 99 kB 00:00:00
(3/3): updates/7/x86_64/primary_db | 12 MB 00:00:02
kubernetes
...
...
[root@localhost ~]# rpm -qa|grep kube
kubernetes-cni-0.8.7-0.x86_64
kubectl-1.22.3-0.x86_64
kubelet-1.22.3-0.x86_64
kubeadm-1.22.3-0.x86_64
[root@localhost ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@localhost ~]#
安装kubeadm之后，发现自动安装了kubectl,kubelet,kubenetes-cni。
6 3个机器分别启动kubelet和docker服务
如果去启动docker服务的话，在后面初始化master节点时，可能遭遇类似下述错误：
[root@localhost ~]# kubeadm init
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 3.10.0-862.el7.x86_64
....
CGROUPS_HUGETLB: enabled
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR CRI]: container runtime is not running: output: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
, error: exit status 1
[ERROR Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[ERROR Swap]: running with swap on is not supported. Please disable swap
[ERROR SystemVerification]: failed executing "docker info --format '{{json .}}'"\noutput: \nstderr: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
\nerror: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[root@localhost ~]#
启动docker服务：
systemctl start docker
这里在启动kubelet服务时，报错：
[root@master-node ~]# systemctl start kubelet
[root@master-node ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since 三 2021-11-03 15:31:58 CST; 4s ago
Docs: https://kubernetes.io/docs/
Process: 2946 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 2946 (code=exited, status=1/FAILURE)
11月 03 15:31:58 master-node systemd[1]: Unit kubelet.service entered failed state.
11月 03 15:31:58 master-node systemd[1]: kubelet.service failed.
[root@master-node ~]#
解决办法，暂时略过该错误。等后面kubelet init 初始化之后，kubelet服务会自动拉起来。
7 3个机器分别关闭交换分区
swapoff -a
8Initialize Kubernetes Master and Setup Default User
该步骤只在主节点执行：
[root@master-node ~]# kubeadm init
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
注意，此时会一直挂在这边不动，如果我们按照其提示，执行：kubeadm config images pull
[root@master-node ~]# kubeadm config images pull
failed to pull image "k8s.gcr.io/kube-apiserver:v1.22.3": output: Trying to pull repository k8s.gcr.io/kube-apiserver ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.189.82:443: i/o timeout
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher
[root@master-node ~]#
则，默认会到https://k8s.gcr.io/v1/_ping下载对应的images，而默认情况下，国内机器的网络是无法直接访问的。
解决办法是，指定阿里云的镜像地址：
[root@master-node ~]# kubeadm init --image-repository registry.aliyuncs.com/google_containers
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-node] and IPs [10.96.0.1 172.16.11.106]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master-node] and IPs [172.16.11.106 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master-node] and IPs [172.16.11.106 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.502268 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master-node as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master-node as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: krj5pn.7ss2hpvqiz56er6d
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.11.106:6443 --token krj5pn.7ss2hpvqiz56er6d \
--discovery-token-ca-cert-hash sha256:46de29a95c709b4b9d08aa764f033d9291c55ddbd2c44ddfb3b0ffbf76f72da9
[root@master-node ~]#
此时，可以看到master节点上的kubelet服务已正常，且，看到从阿里云镜像站点下载的几个images:
[root@master-node ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since 三 2021-11-03 15:43:22 CST; 2min 26s ago
Docs: https://kubernetes.io/docs/
Main PID: 4600 (kubelet)
Tasks: 14
Memory: 31.7M
CGroup: /system.slice/kubelet.service
└─4600 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yam...
11月 03 15:45:27 master-node kubelet[4600]: I1103 15:45:27.181096 4600 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
11月 03 15:45:27 master-node kubelet[4600]: E1103 15:45:27.416673 4600 kubelet.go:2337] "Container runtime network not ready" networkReady="NetworkReady=false r...nitialized"
11月 03 15:45:32 master-node kubelet[4600]: I1103 15:45:32.181441 4600 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
11月 03 15:45:32 master-node kubelet[4600]: E1103 15:45:32.418604 4600 kubelet.go:2337] "Container runtime network not ready" networkReady="NetworkReady=false r...nitialized"
11月 03 15:45:37 master-node kubelet[4600]: I1103 15:45:37.181784 4600 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
11月 03 15:45:37 master-node kubelet[4600]: E1103 15:45:37.420749 4600 kubelet.go:2337] "Container runtime network not ready" networkReady="NetworkReady=false r...nitialized"
11月 03 15:45:42 master-node kubelet[4600]: I1103 15:45:42.182572 4600 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
11月 03 15:45:42 master-node kubelet[4600]: E1103 15:45:42.423431 4600 kubelet.go:2337] "Container runtime network not ready" networkReady="NetworkReady=false r...nitialized"
11月 03 15:45:47 master-node kubelet[4600]: I1103 15:45:47.182739 4600 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
11月 03 15:45:47 master-node kubelet[4600]: E1103 15:45:47.425584 4600 kubelet.go:2337] "Container runtime network not ready" networkReady="NetworkReady=false r...nitialized"
Hint: Some lines were ellipsized, use -l to show in full.
[root@master-node ~]#
[root@master-node ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-apiserver v1.22.3 53224b502ea4 6 days ago 128 MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.22.3 0aa9c7e31d30 6 days ago 52.7 MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.22.3 05c905cef780 6 days ago 122 MB
registry.aliyuncs.com/google_containers/kube-proxy v1.22.3 6120bd723dce 6 days ago 104 MB
registry.aliyuncs.com/google_containers/etcd 3.5.0-0 004811815584 4 months ago 295 MB
registry.aliyuncs.com/google_containers/coredns v1.8.4 8d147537fb7d 5 months ago 47.6 MB
registry.aliyuncs.com/google_containers/pause 3.5 ed210e3e4a5b 7 months ago 683 kB
[root@master-node ~]#
根据提示，配置默认用户:
如果想以非root用户启动kubernetes服务的话，执行：
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
我这里想直接以root运行kubernetes，所以配置个环境变量到.bash_profile即可。
[root@master-node ~]# vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
KUBECONFIG=/etc/kubernetes/admin.conf
export PATH KUBECONFIG
...
[root@master-node ~]# source .bash_profile
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node NotReady control-plane,master 61m v1.22.3
[root@master-node ~]#
此时，看到master-node的状态还是NotReady的，其主要原因是还没有配置Pod的network。从kubeadm init 的执行结果也看到有明确提示：
根据提示页，我们选择配置Flannel：
- Flannel is an overlay network provider that can be used with Kubernetes.
9 配置pod network
注意：该步骤只在主节点执行。
打开Flannel的GitHub页面：
看到安装命令为：
For Kubernetes v1.17+ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
这里的1个问题就是：https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml我们无法访问，并且该文件配置的内容为：
[root@master-node ~]# cat flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.14.0
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.14.0
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
[root@master-node ~]#
执行安装：
[root@master-node ~]# kubectl apply -f flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,master 87m v1.22.3
[root@master-node ~]#
至此，看到master节点已经是Ready状态了。
10 添加worker节点
Join the Worker Node to the Kubernetes Cluster
根据前面步骤8中的提示，分别在2个worker节点上执行：
worker-1:172.16.11.148
[root@node-1 ~]# kubeadm join 172.16.11.106:6443 --token krj5pn.7ss2hpvqiz56er6d \
> --discovery-token-ca-cert-hash sha256:46de29a95c709b4b9d08aa764f033d9291c55ddbd2c44ddfb3b0ffbf76f72da9
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@node-1 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since 三 2021-11-03 17:16:27 CST; 11min ago
Docs: https://kubernetes.io/docs/
Main PID: 9539 (kubelet)
Tasks: 13
Memory: 54.1M
CGroup: /system.slice/kubelet.service
└─9539 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yam...
11月 03 17:27:23 node-1 kubelet[9539]: E1103 17:27:23.409220 9539 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-flannel\" ...
11月 03 17:27:37 node-1 kubelet[9539]: I1103 17:27:37.408362 9539 scope.go:110] "RemoveContainer" containerID="7bffcb305147552c9cf5f27da9410b3810b15f2a7e888a283...5b986ceb67"
11月 03 17:27:37 node-1 kubelet[9539]: E1103 17:27:37.409424 9539 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-flannel\" ...
11月 03 17:27:49 node-1 kubelet[9539]: I1103 17:27:49.408006 9539 scope.go:110] "RemoveContainer" containerID="7bffcb305147552c9cf5f27da9410b3810b15f2a7e888a283...5b986ceb67"
11月 03 17:27:49 node-1 kubelet[9539]: E1103 17:27:49.409468 9539 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-flannel\" ...
11月 03 17:28:01 node-1 kubelet[9539]: I1103 17:28:01.407251 9539 scope.go:110] "RemoveContainer" containerID="7bffcb305147552c9cf5f27da9410b3810b15f2a7e888a283...5b986ceb67"
11月 03 17:28:01 node-1 kubelet[9539]: E1103 17:28:01.407888 9539 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-flannel\" ...
11月 03 17:28:13 node-1 kubelet[9539]: I1103 17:28:13.408296 9539 scope.go:110] "RemoveContainer" containerID="7bffcb305147552c9cf5f27da9410b3810b15f2a7e888a283...5b986ceb67"
11月 03 17:28:13 node-1 kubelet[9539]: E1103 17:28:13.409702 9539 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-flannel\" ...
11月 03 17:28:24 node-1 kubelet[9539]: I1103 17:28:24.408292 9539 scope.go:110] "RemoveContainer" containerID="7bffcb305147552c9cf5f27da9410b3810b15f2a7e888a283...5b986ceb67"
Hint: Some lines were ellipsized, use -l to show in full.
[root@node-1 ~]#
添加之后，看到worker节点上的kubectl服务自动成功运行了。
稍等片刻，可以从主节点上看到节点状态：
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,master 93m v1.22.3
node-1 NotReady <none> 22s v1.22.3
[root@master-node ~]#
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,master 93m v1.22.3
node-1 Ready <none> 48s v1.22.3
[root@master-node ~]#
worker-2:172.16.11.161
[root@node-2 ~]# kubeadm join 172.16.11.106:6443 --token krj5pn.7ss2hpvqiz56er6d --discovery-token-ca-cert-hash sha256:46de29a95c709b4b9d08aa764f033d9291c55ddbd2c44ddfb3b0ffbf76f72da9
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[root@node-2 ~]#
####根据提示，禁用SWAP，继续
[root@node-2 ~]# swapoff -a
[root@node-2 ~]# kubeadm join 172.16.11.106:6443 --token krj5pn.7ss2hpvqiz56er6d --discovery-token-ca-cert-hash sha256:46de29a95c709b4b9d08aa764f033d9291c55ddbd2c44ddfb3b0ffbf76f72da9
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@node-2 ~]#
最后，master节点上执行kubectl get nodes看到：
[root@master-node ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,master 98m v1.22.3
node-1 Ready <none> 5m17s v1.22.3
node-2 Ready <none> 2m1s v1.22.3
[root@master-node ~]#
三小结和链接
kubernetes功能强大，知识体系比较多。但，不要紧，万事开头难，从成功的安装和配置开始迈出第一步。为接下来的配置Rancher做环境准备。
How to Install a Kubernetes Cluster on CentOS 7
四 补充
2021.11.4
该文档创建的kubernetes cluster还是有问题的，具体表现为，创建和发布service之后，不能正常访问；且，在所有节点上，并没有生成/run/flannel/subnet.env文件；
具体的解决思路是：推倒整个cluster(通过先在所有worker节点执行kubeadm reset命令，最后在master节点执行该命令)；重建cluster：master节点执行：kubeadm init –image-repository registry.aliyuncs.com/google_containers –pod-network-cidr=10.244.0.0/16
然后：worker节点，根据提示，join到cluster；
最后，master节点再次执行：kubectl apply -f flannel.yml；
确认，所有节点上自动生成了/run/flannel/subnet.env文件。且：kubectl get nodes能看到所有节点Ready方可。
具体解决错误步骤，见文章：在kubernetes cluster上部署NGINX服务流程及错误解决
3条评论
Pingback:
Pingback:
Pingback: