目录
6.我在B站学云原生之Kubernetes入门实践Ubuntu系统上安装部署高可用的K8S集群环境
全栈工程师修炼指南
编辑于 2022年04月23日 17:32
收录于文集
共39篇

帅哥（靓仔）、美女，点个关注后续不迷路！

本章目录

cut-off

  • 0x03 高可用K8s集群部署(v1.19.6)

    • Step 1.高可用组件安装所有Master节点通过apt安装HAProxy和KeepAlived

    • Step 2.所有Master节点配置HAProxy

    • Step 3.配置KeepAlived健康检查文件

    • Step 4.启动haproxy和keepalived以及测试VIP

    • Step 5.集群安装其它指定版本

    • Step 6.集群初始化的yaml文件

    • Step 7.Kubernetes 集群访问配置

    • Step 8.现在应该将pod网络部署到集群,此处选用calico网络插件而非使用flannel插件

    • Step 9.高可用Master初始化其他master加入集群

    • Step 10.工作节点或者负载加入到集群中

    • Step 11.集群状态查看 & 镜像查看

  • 0x04 高可用集群使用初体验

cut-off

作者: WeiyiGeek

原文地址：https://blog.weiyigeek.top/2020/4-27-470.html

cut-off

0x03 高可用K8s集群部署(v1.19.6)

Step 1.高可用组件安装所有Master节点通过apt安装HAProxy和KeepAlived

代码块
JavaScript
自动换行
复制代码
# PS: 此时只下载不安装(方便传到没有网络的master节点上)
sudo apt -d install keepalived haproxy
# ~/k8s-init$ scp -P 20211 /home/weiyigeek/k8s-init/High-Availability/* weiyigeek@192.168.1.108:~/k8s-init/
# ~/k8s-init$ scp -P 20211 /home/weiyigeek/k8s-init/High-Availability/* weiyigeek@192.168.1.109:~/k8s-init/

# 通过 dpkg 命令 安装 上一步下载的 deb包
/var/cache/apt/archives$ dpkg -i *.deb
# ~/k8s-init$ ssh -p20211 weiyigeek@192.168.1.108 "sudo -S dpkg -i ~/k8s-init/*.deb"
# ~/k8s-init$ ssh -p20211 weiyigeek@192.168.1.109 "sudo -S dpkg -i ~/k8s-init/*.deb"
复制成功

Step 2.所有Master节点配置HAProxy

PS : 详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同

代码块
JavaScript
自动换行
复制代码
# Ubuntu haproxy 配置文件目录
$ls /etc/haproxy/
errors/      haproxy.cfg

$sudo vim /etc/haproxy/haproxy.cfg
global
  log /dev/log    local0
  log /dev/log    local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
  log     global
  mode    http
  option  httplog
  option  dontlognull
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

# 注意: 16443 为VIP的Apiserver-控制平面端口
frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

# 注意: Master 节点的默认apiserver是6443端口
backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server weiyigeek-107 192.168.1.107:6443  check
  server weiyigeek-108 192.168.1.108:6443  check
  server weiyigeek-109 192.168.1.109:6443  check
复制成功

  • 所有Master节点配置KeepAlived，配置不一样，注意区分注意每个节点的IP和网卡（interface参数）

代码块
JavaScript
自动换行
复制代码
mkdir -vp /etc/keepalived
# weiyigeek-107 Master 1
sudo tee /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
script_user root
  enable_script_security
}
vrrp_script chk_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 5
  weight -5
  fall 2  
  rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens160
    mcast_src_ip 192.168.1.107
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.1.110
    }
#    track_script {
#       chk_apiserver
#    }
}
EOF

# weiyigeek-108 Master 2 => Backup
sudo tee /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
script_user root
  enable_script_security
}
vrrp_script chk_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 5
  weight -5
  fall 2  
  rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    mcast_src_ip 192.168.1.108
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.1.110
    }
#    track_script {
#       chk_apiserver
#    }
}
EOF

# weiyigeek-108 Master 3 => Backup
sudo tee /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
script_user root
  enable_script_security
}
vrrp_script chk_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 5
  weight -5
  fall 2  
  rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    mcast_src_ip 192.168.1.109
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.1.110
    }
#    track_script {
#       chk_apiserver
#    }
}
EOF
复制成功

PS : 注意上述的健康检查是关闭的，集群建立完成后再开启：

Step 3.配置KeepAlived健康检查文件

代码块
JavaScript
自动换行
复制代码
sudo tee /etc/keepalived/check_apiserver.sh <<'EOF'
#!/bin/bash
err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EOF
sudo chmod +x /etc/keepalived/check_apiserver.sh

# 备份高可用相关配置文件
mkdir ~/k8s-init/haproxy && cp /etc/haproxy/haproxy.cfg ~/k8s-init/haproxy 
mkdir ~/k8s-init/keepalived && cp /etc/keepalived/keepalived.conf /etc/keepalived/check_apiserver.sh ~/k8s-init/keepalived
复制成功

Step 4.启动haproxy和keepalived以及测试VIP

代码块
JavaScript
自动换行
复制代码
# 自启动并立即启动
sudo systemctl daemon-reload
sudo systemctl enable --now haproxy
sudo systemctl enable --now keepalived

# VIP 漂移测试
~$ ps aux | grep "haproxy"
root         892  0.0  0.1  15600  9236 ?        Ss   10:39   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy      893  0.0  0.0 531724  3480 ?        Sl   10:39   0:02 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

weiyigeek-107:$ ip addr
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:8a:e8:db brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.107/24 brd 192.168.1.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet 192.168.1.110/32 scope global ens160  # 可以看见该VIP在weiyigeek-107
       valid_lft forever preferred_lft forever

weiyigeek-107:~$ systemctl stop keepalived.service  # 现在 weiyigeek-107 停止 keepalived.service
weiyigeek-107:~$ ip addr | grep "192.168.1.110"    # 此时会发现 weiyigeek-107 已经不存在该VIP了
weiyigeek-107:~$ ssh -p20211 weiyigeek@192.168.1.109 "ip addr | grep '192.168.1.110'" # 会发现漂移到weiyigeek-109的master节点的机器上
  inet 192.168.1.110/32 scope global ens160
weiyigeek-107:~$ ping 192.168.1.110               # ping 通信正常
# PING 192.168.1.110 (192.168.1.110) 56(84) bytes of data.
# 64 bytes from 192.168.1.110: icmp_seq=1 ttl=64 time=0.218 ms
复制成功

PS : 至此基于Haproxy与keepalive实现的高可用已经OK了;

Step 5.集群安装其它指定版本

PS : 在基础环境的安装中我们采用了 考虑到 k8s 在 1.23.x 版本之后将会丢弃docker而使用其它的CRI-O, 此处为了集群环境的稳定性将原本docker以及kubernetes版本删除，分别安装1.19.x版本;

代码块
JavaScript
自动换行
复制代码
# (1) 停止服务
sudo systemctl stop docker.service kubelet.service docker.socket

# (2) docker 容器卸载 && K8s 卸载
sudo apt-get remove docker-ce docker-ce-cli containerd.io kubectl kubeadm kubelet

# (3) 查看 docker 以及 k8s 版本
sudo apt-cache madison docker-ce docker-ce-cli kubelet
# sudo apt-get -d install docker-ce=5:19.03.14~3-0~ubuntu-focal docker-ce-cli=5:19.03.14~3-0~ubuntu-focal containerd.io  # Download complete and in download only mode
# sudo dpkg -i /var/cache/apt/archives/*.deb
sudo apt-get install docker-ce=5:19.03.14~3-0~ubuntu-focal docker-ce-cli=5:19.03.14~3-0~ubuntu-focal containerd.io -y
sudo apt install kubelet=1.19.6-00 kubeadm=1.19.6-00 kubectl=1.19.6-00 -y
$ kubeadm version
# kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
$ kubelet --version
# Kubernetes v1.19.6
$ kubectl version
# Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

# (4) 自启并启动服务
sudo systemctl daemon-reload
sudo systemctl enable --now docker 
sudo systemctl enable --now kubelet
sudo systemctl restart docker kubelet
复制成功

Step 6.集群初始化的yaml文件

描述: kubeadm 可以打印出初始化以及节点加入的配置模板;

代码块
JavaScript
自动换行
复制代码
# (1) 初始化
$ kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ubuntu
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

# (2) 节点加入
$ kubeadm config print join-defaults
apiVersion: kubeadm.k8s.io/v1beta2
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: kube-apiserver:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ubuntu
  taints: null
复制成功

K8s 集群初始化配置文件()

代码块
Shell
自动换行
复制代码
# (1) 注意 : Token的格式以及Pod子网的网段 (Calico)和ipvs的支持
K8SVERSION=1.19.6
k8SIMAGEREP="registry.cn-hangzhou.aliyuncs.com/google_containers"
APISERVER_NAME=weiyigeek-lb-vip.k8s
APISERVER_IP=192.168.1.110
APISERVER_PORT=16443
LOCALAPIVERVER_NAME=weiyigeek-107
LOCALAPISERVER_IP=192.168.1.107
LOCALAPISERVER_PORT=6443
SERVICE_SUBNET=172.16.0.0/16

cat <<EOF > ~/k8s-init/kubeadm-init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 20w21w.httpweiyigeektop
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: ${LOCALAPISERVER_IP}
  bindPort: ${LOCALAPISERVER_PORT}
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ${LOCALAPIVERVER_NAME}
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - ${APISERVER_IP}
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: ${k8SIMAGEREP}
kind: ClusterConfiguration
kubernetesVersion: v${K8SVERSION}
controlPlaneEndpoint: ${APISERVER_NAME}:${APISERVER_PORT}
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: ${SERVICE_SUBNET}
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1    
kind: KubeProxyConfiguration    
featureGates:      
  SupportIPVSProxyMode: true    
mode: ipvs
EOF

# (2) 初始化Master节点控制平面以及Worker加入的参数生成(# 根据您服务器网速的情况，您需要等候 3 - 10 分钟建议采用下面备注中的操作)
sudo kubeadm init --config=/home/weiyigeek/k8s-init/kubeadm-init-config.yaml --upload-certs | tee kubeadm_init.log
# [certs] Using certificateDir folder "/etc/kubernetes/pki"
# [certs] Generating "ca" certificate and key
# [certs] Generating "apiserver" certificate and key
# [certs] apiserver serving cert is signed for DNS names [weiyigeek-107 weiyigeek-lb-vip.k8s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.107 192.168.1.110]
# [certs] Generating "apiserver-kubelet-client" certificate and key
# [certs] Generating "front-proxy-ca" certificate and key
# [certs] Generating "front-proxy-client" certificate and key
# [certs] Generating "etcd/ca" certificate and key
# [certs] Generating "etcd/server" certificate and key
# [certs] etcd/server serving cert is signed for DNS names [weiyigeek-107 localhost] and IPs [192.168.1.107 127.0.0.1 ::1]
# [certs] Generating "etcd/peer" certificate and key
# [certs] etcd/peer serving cert is signed for DNS names [weiyigeek-107 localhost] and IPs [192.168.1.107 127.0.0.1 ::1]
# [certs] Generating "etcd/healthcheck-client" certificate and key
# [certs] Generating "apiserver-etcd-client" certificate and key
# [certs] Generating "sa" key and public key
# [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "admin.conf" kubeconfig file
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "kubelet.conf" kubeconfig file
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "controller-manager.conf" kubeconfig file
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [kubeconfig] Writing "scheduler.conf" kubeconfig file
# [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
# [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
# [kubelet-start] Starting the kubelet
# [control-plane] Using manifest folder "/etc/kubernetes/manifests"
# [control-plane] Creating static Pod manifest for "kube-apiserver"
# [control-plane] Creating static Pod manifest for "kube-controller-manager"
# [control-plane] Creating static Pod manifest for "kube-scheduler"
# [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
# [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
# [apiclient] All control plane components are healthy after 21.750931 seconds
# [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
# [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
# [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
# [upload-certs] Using certificate key:
# bb5b2f0b287d35e179ef4efawwww9f61a38f62343a9b06fc143e3b
# [mark-control-plane] Marking the node weiyigeek-107 as control-plane by adding the label "node-role.kubernetes.io/master=''"
# [mark-control-plane] Marking the node weiyigeek-107 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
# [bootstrap-token] Using token: 2021wq.httpweiyigeektop
# [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
# [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
# [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
# [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
# [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
# [addons] Applied essential addon: CoreDNS
# [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
# [addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!
复制成功

PS : 将该yaml文件复制到其他master节点，之后所有Master节点提前下载镜像，可以节省初始化时间：

代码块
Shell
自动换行
复制代码
# 方式1.配置文件上传到其他master节点
scp -P 20211 /home/weiyigeek/k8s-init/kubeadm-init-config.yaml weiyigeek@192.168.1.108:~/k8s-init/kubeadm-init-config.yaml
kubeadm config images pull --config /home/weiyigeek/k8s-init/kubeadm-init-config.yaml
# W0111 11:24:15.905316    5481 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.6
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
# [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
复制成功

Step 7.Kubernetes 集群访问配置

代码块
Shell
自动换行
复制代码
# (1) 普通用户对集群访问配置文件设置
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# (2) 自动运行设置 KUBECONFIG 环境以及k8s命令自动补齐
grep "export KUBECONFIG" ~/.profile | echo "export KUBECONFIG=$HOME/.kube/config" >> ~/.profile
tee -a ~/.profile <<'EOF'
source <(kubectl completion bash)
source <(kubeadm completion bash)
# source <(kubelet completion bash)
# source <(helm completion bash)
EOF
source ~/.profile

# (3) 查看节点状态和kube-system命名空间内Pod状态
~$ kubectl get node
# NAME       STATUS     ROLES    AGE   VERSION
# weiyigeek-107   NotReady   master   45m   v1.19.6   # 此处NotReady 是由于我们的calico网络插件未配置好
~$ kubectl get pod -n kube-system
# NAME                               READY   STATUS    RESTARTS   AGE
# coredns-6c76c8bb89-6cl4d           0/1     Pending   0          45m
# coredns-6c76c8bb89-xzkms           0/1     Pending   0          45m
# etcd-weiyigeek-107                      1/1     Running   0          45m
# kube-apiserver-weiyigeek-107            1/1     Running   0          45m
# kube-controller-manager-weiyigeek-107   1/1     Running   0          45m
# kube-proxy-l7bp6                   1/1     Running   0          45m
# kube-scheduler-weiyigeek-107            1/1     Running   0          45m
复制成功

PS : 当没有安装第三方的  时候，通过kubectl get node 查看到节点状态是 NotReady ，当安装完成扁平化网络插件后, 如果没有其它意外则显示Ready

Step 8.现在应该将pod网络部署到集群,此处选用calico网络插件而非使用flannel插件

代码块
Shell
自动换行
复制代码
# Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
# https://kubernetes.io/docs/concepts/cluster-administration/addons/

# 安装 calico 网络插件(此处安装最新的v3.17.1版本不采用kuboard脚本)
# 参考文档 https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
# wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
# kubectl apply -f calico-3.13.1.yaml

# (1) Install Calico with Kubernetes API datastore, 50 nodes or less
# curl https://docs.projectcalico.org/manifests/calico.yaml -O

# (2) Install Calico with etcd datastore (使用etcd数据存储安装Calico) 
curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -O

# calico-etcd 网络与etc集群连接修改(此处指定pod子网地址)
ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
ETCD_KEY=`sudo cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
POD_SUBNET=`sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml

sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.12.107:2379,https://192.168.12.108:2379,https://192.168.12.109:2379"#g' calico-etcd.yaml
sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml

# (3) 部署到集群中
kubectl apply -f calico-etcd.yaml
# secret/calico-etcd-secrets created
# configmap/calico-config created
# clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
# clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
# clusterrole.rbac.authorization.k8s.io/calico-node created
# clusterrolebinding.rbac.authorization.k8s.io/calico-node created
# daemonset.apps/calico-node created
# serviceaccount/calico-node created
# deployment.apps/calico-kube-controllers created
# serviceaccount/calico-kube-controllers created
# poddisruptionbudget.policy/calico-kube-controllers created

# (4) 只在 master 节点执行
# 执行如下命令，等待 3-10 分钟，直到所有的容器组处于 Running 状态
watch kubectl get pod -n kube-system -o wide
echo -e "---等待容器组构建完成---" && sleep 180
kubectl get nodes -o wide       # 查看 master 节点初始化结果

# (5) 各节点网卡地址已经未设置的POD子网地址
~$ route
# Kernel IP routing table
# Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
# default         _gateway        0.0.0.0         UG    0      0        0 ens160
# default         _gateway        0.0.0.0         UG    0      0        0 ens160
# 172.16.0.192    0.0.0.0         255.255.255.192 U     0      0        0 *
# 172.16.0.193    0.0.0.0         255.255.255.255 UH    0      0        0 calib6c39f0a1d5
# 172.16.0.194    0.0.0.0         255.255.255.255 UH    0      0        0 calic77cbbaf4da
# 172.16.24.192   weiyigeek-223        255.255.255.192 UG    0      0        0 tunl0
# 172.16.100.64   weiyigeek-224        255.255.255.192 UG    0      0        0 tunl0
# 172.16.135.192  weiyigeek-108        255.255.255.192 UG    0      0        0 tunl0
# 172.16.182.192  weiyigeek-226        255.255.255.192 UG    0      0        0 tunl0
# 172.16.183.64   weiyigeek-225        255.255.255.192 UG    0      0        0 tunl0
# 172.16.243.64   weiyigeek-109        255.255.255.192 UG    0      0        0 tunl0
复制成功

Step 9.高可用Master初始化，将其他master节点加入集群控制平面

代码块
Shell
自动换行
复制代码
sudo kubeadm join weiyigeek-lb-vip.k8s:16443 --token 20w21w.httpweiyigeektop \
  --discovery-token-ca-cert-hash sha256:7ea900ef214c98aef6d7daf1380320d0a43f666f2d4b6b7469077bd51790118e \
  --control-plane --certificate-key 8327482265975b7a60f3549222f1093353ecaa148a3404cd10c605d4111566fc

PS : 作为安全措施，上传的证书将在两小时内被删除; 如果需要您可以使用下面的命令重新加载证书。
"kubeadm init phase upload-certs --upload-certs"

# 查看 Token 
~/k8s-init$ kubectl get secret
  # NAME                  TYPE                                  DATA   AGE
  # default-token-xzcbz   kubernetes.io/service-account-token   3      17m
复制成功

Step 10.工作节点或者负载加入到集群中

代码块
Shell
自动换行
复制代码
# (1) 为了便于工作节点快速接入到集群中，我们先把master中的镜像导入到工作节点
docker save -o v1.19.6.tar $(docker images | grep -v TAG | cut -d ' ' -f1)  # 导出
docker load -i v1.19.6.tar   # 加载

# (2) 然后你可以加入任意数量的worker节点，在每个worker节点上以root用户运行如下命令:
sudo kubeadm join weiyigeek-lb-vip.k8s:16443 --token 20w21w.httpweiyigeektop \
  --discovery-token-ca-cert-hash sha256:7ea900ef214c98aef6d7daf1380320d0a43f666f2d4b6b7469077bd51790118e
复制成功

Step 11.集群状态查看 & 镜像查看

代码块
Shell
自动换行
复制代码
# (1) Master 集群状态
weiyigeek-107:~$ kubectl get node -o wide
NAME       STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
weiyigeek-107   Ready    master   25m    v1.19.6   192.168.1.107   <none>        Ubuntu 20.04.1 LTS   5.4.0-60-generic   docker://19.3.14
weiyigeek-108   Ready    master   15m    v1.19.6   192.168.1.108   <none>        Ubuntu 20.04.1 LTS   5.4.0-60-generic   docker://19.3.14
weiyigeek-109   Ready    master   100s   v1.19.6   192.168.1.109   <none>        Ubuntu 20.04.1 LTS   5.4.0-60-generic   docker://19.3.14

# (2) K8s 集群 SVC 信息查看
~$ kubectl get svc -o wide
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   32m   <none>
~$ kubectl get svc -o wide -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   31m   k8s-app=kube-dns

# (3) 为了可以节省资源将其余两台Master节点关闭污点
kubectl taint node weiyigeek-108 node-role.kubernetes.io/master=:NoSchedule-
kubectl taint node weiyigeek-109 node-role.kubernetes.io/master=:NoSchedule-
复制成功

0x04 高可用集群使用初体验

描述: 简单对nginx应用从镜像打包到部署到k8s集群中使用流程进行演示

  • Step 1.编写dockerfile和相关脚本 $ cat dockerfile

代码块
Shell
自动换行
复制代码
FROM nginx
LABEL maintainer="Nginx Test Demo , Authorr: weiyigeek, Version: 2.2"
ENV PATH /usr/local/nginx/sbin:$PATH
ENV IMAGE_VERSION 2.2
COPY ./host.sh /
RUN chmod a+x /host.sh
ENTRYPOINT ["/host.sh"]
复制成功

$ cat host.sh

代码块
Shell
自动换行
复制代码
#!/bin/sh
echo "Hostname: $HOSTNAME" > /usr/share/nginx/html/host.html
echo "Image Version: $IMAGE_VERSION" >> /usr/share/nginx/html/host.html
echo "Nginx Version: $NGINX_VERSION" >> /usr/share/nginx/html/host.html
sh -c "nginx -g 'daemon off;'"
复制成功

  • Step 2.镜像构建 & 上传到私有得Harbor仓库:

代码块
Shell
自动换行
复制代码
$ docker build -t harbor.weiyigeek.top/test/nginx:v2.2 .
$ docker images | grep "v2.2"
  # harbor.weiyigeek.top/test/nginx                                  v2.2                b8a212b2bc88        10 hours ago        133MB
$ docker push harbor.weiyigeek.top/test/nginx:v2.2
  # The push refers to repository [harbor.weiyigeek.top/test/nginx]
  # v2.2: digest: sha256:4c49fc25d52e5331146699ff605561f59fb326505074c0474a3ce4898f0fcb02 size: 1776
复制成功

  • Step 3.部署镜像 & 查看:

方式1

代码块
Shell
自动换行
复制代码
# 方式1.不推荐(配置标签以及标签选择需要添加参数，比较麻烦)
$ kubectl run nginx-deployment --image=harbor.weiyigeek.top/test/nginx:v2.2 --port=80
  # pod/nginx-deployment created

$ kubectl get pod -o wide
  # NAME               READY   STATUS    RESTARTS   AGE    IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment   1/1     Running   0          108s   10.244.1.2   k8s-node-4   <none>           <none>

# 利用podSubnet进行查看
weiyigeek@ubuntu:~$ curl http://10.244.1.2/host.html
  # Hostname: nginx-deployment
  # Image Version: 2.2
  # Nginx Version: 1.19.4
复制成功

方式2

代码块
Shell
自动换行
复制代码
cat > nginx-deployment.yaml <<'EOF'
apiVersion: apps/v1	#与k8s集群版本有关，使用 kubectl api-versions 即可查看当前集群支持的版本
kind: Deployment   	#该配置的类型，我们使用的是 Deployment
metadata:	          #译名为元数据，即 Deployment 的一些基本属性和信息
  name: nginx-deployment	#Deployment 的名称
  namespace: default 
  labels:	      #标签可以灵活定位一个或多个资源，其中key和value均可自定义，可以定义多组，目前不需要理解
    app: nginx	#为该Deployment设置key为app，value为nginx的标签
spec:	          #这是关于该Deployment的描述，可以理解为你期待该Deployment在k8s中如何使用
  replicas: 1  	#使用该Deployment创建一个应用程序实例
  selector:	    #标签选择器，与上面的标签共同作用，目前不需要理解
    matchLabels: #选择包含标签app:nginx的资源
      app: nginx
  template:	     #这是选择或创建的Pod的模板
    metadata:	   #Pod的元数据
      labels:    #Pod的标签，上面的selector即选择包含标签app:nginx的Pod
        app: nginx
    spec:	        #期望Pod实现的功能(即在pod中部署)
      containers:	#生成container，与docker中的container是同一种
      - name: nginx	#container的名称
        image: harbor.weiyigeek.top/test/nginx:v2.2 	#使用镜像nginx最新版本创建container，该container默认80端口可访问
EOF

$ kubectl apply -f nginx-deployment.yaml
  # deployment.apps/nginx-deployment created

$ kubectl get deployment
  # NAME               READY   UP-TO-DATE   AVAILABLE   AGE
  # nginx-deployment   1/1     1            1           87s
  # weiyigeek@ubuntu:~/nginx$ kubectl get pod
  # NAME                                READY   STATUS    RESTARTS   AGE
  # nginx-deployment-7f5d9779c6-flmsf   1/1     Running   0          92s
  # weiyigeek@ubuntu:~/nginx$ kubectl get pod -o wide
  # NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment-7f5d9779c6-flmsf   1/1     Running   0          99s   10.244.1.4   k8s-node-4   <none>           <none>

~/nginx$ curl http://10.244.1.4/host.html
  # Hostname: nginx-deployment-7f5d9779c6-flmsf
  # Image Version: 2.2
  # Nginx Version: 1.19.4
复制成功

  • Step 4.Pod 副本 & 收缩 & 端口映射

代码块
Shell
自动换行
复制代码
$ kubectl delete pod nginx-deployment-7f5d9779c6-flmsf
  # pod "nginx-deployment-7f5d9779c6-flmsf" deleted
$ kubectl get pod -o wide
  # NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment-7f5d9779c6-hhl7k   1/1     Running   0          60s   10.244.1.5   k8s-node-4   <none>           <none>

$ kubectl scale --replicas=3 deployment/nginx-deployment
  # deployment.apps/nginx-deployment scaled
$ kubectl get pod -o wide
  # NAME                                READY   STATUS    RESTARTS   AGE    IP           NODE         NOMINATED NODE   READINESS GATES
  # nginx-deployment-7f5d9779c6-dr5h8   1/1     Running   0          4s     10.244.1.7   k8s-node-4   <none>           <none>
  # nginx-deployment-7f5d9779c6-hhl7k   1/1     Running   0          109s   10.244.1.5   k8s-node-4   <none>           <none>
  # nginx-deployment-7f5d9779c6-sk2f4   1/1     Running   0          4s     10.244.1.6   k8s-node-4   <none>           <none>

$ kubectl edit svc nginx-deployment  # 第一章中有介绍 SVC，不知道回去看一哈
$ kubectl expose -f nginx-controller.yaml --port=80 --target-port=8000 --protocol=TCP --type=NodePort --node-port=31855  # 后续验证
$ kubectl expose svc nginx-deployment --port=80 --target-port=8000 --protocol=TCP --type=NodePort
  # type: NodePort
$ kubectl get svc
  # NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
  # kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP           2d22h
  # nginx-deployment   NodePort    10.99.90.247   <none>        30000:31855/TCP   39h

$ sudo netstat -anpt | grep "31855"
  # tcp        0      0 0.0.0.0:31855           0.0.0.0:*               LISTEN      1931726/kube-proxy
复制成功

  • Step 5.IPVS 负载均衡和rr轮询机制查看

代码块
Shell
自动换行
复制代码
$ sudo ipvsadm -Ln
  # IP Virtual Server version 1.2.1 (size=4096)
  # Prot LocalAddress:Port Scheduler Flags
  #   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  # TCP  10.10.107.202:31855 rr
  #   -> 10.244.1.5:80                Masq    1      0          1
  #   -> 10.244.1.6:80                Masq    1      1          1
  #   -> 10.244.1.7:80                Masq    1      0          1
  # TCP  10.96.0.1:443 rr
  #   -> 10.10.107.202:6443           Masq    1      3          0
  # TCP  10.96.0.10:53 rr
  #   -> 10.244.0.4:53                Masq    1      0          0
  #   -> 10.244.0.5:53                Masq    1      0          0
  # TCP  10.96.0.10:9153 rr
  #   -> 10.244.0.4:9153              Masq    1      0          0
  #   -> 10.244.0.5:9153              Masq    1      0          0
  # TCP  10.99.90.247:30000 rr
  #   -> 10.244.1.5:80                Masq    1      0          0
  #   -> 10.244.1.6:80                Masq    1      0          0
  #   -> 10.244.1.7:80                Masq    1      0          0
  # TCP  127.0.0.1:31855 rr
  #   -> 10.244.1.5:80                Masq    1      0          0
  #   -> 10.244.1.6:80                Masq    1      0          0
  #   -> 10.244.1.7:80                Masq    1      0          0

# 基于RR轮训负载
Hostname: nginx-deployment-7f5d9779c6-dr5h8 Image Version: 2.2 Nginx Version: 1.19.4 
Hostname: nginx-deployment-7f5d9779c6-sk2f4 Image Version: 2.2 Nginx Version: 1.19.4 
Hostname: nginx-deployment-7f5d9779c6-hhl7k Image Version: 2.2 Nginx Version: 1.19.4
复制成功

cut-off

原文地址：https://blog.weiyigeek.top/2020/4-27-470.html

cut-off

本文至此完毕，更多技术文章，尽情期待下一章节！

cut-off

已发布的相关历史文章（点击即可进入）

1.我在B站学云原生之Kubernetes基础入门学习概述系统架构及组件浅析​

2.我在B站学云原生之Kubernetes基础概念名词浅析与简单实践指南​

3.我在B站学云原生之Kubernetes入门实践CentOS系统上安装部署K8S单控制平面环境​

4.我在B站学云原生之Kubernetes入门实践CentOS系统上手动安装部署K8S集群环境​

网页链接​5.我在B站学云原生之Kubernetes入门实践Ubuntu系统上安装部署K8S单控制平面环境​

cut-off

欢迎各位志同道合的朋友一起学习交流，如文章有误请在下方留下您宝贵的经验知识，个人邮箱地址【master#weiyigeek.top】或者 个人公众号【WeiyiGeek】联系我。

cut-off

更多文章来源于【WeiyiGeek Blog - 为了能到远方，脚下的每一步都不能少】 个人博客。

个人主页: 【 https://weiyigeek.top 】

博客地址: 【 https://blog.weiyigeek.top 】

https://weiyigeek.top - Always keep a beginner's mind, don&#​39;t forget the beginner's mind

专栏书写不易，如果您觉得这个专栏还不错的，请给这篇专栏【点个赞、投个币、收个藏、关个注，转个发、留个言】，这将对我的肯定，谢谢！。

  • echo  "【点个赞】，动动你那粗壮的拇指或者芊芊玉手，亲！"

  • printf("%s", "【投个币】，万水千山总是情，投个硬币行不行，亲！")

  • fmt.Printf("【收个藏】，阅后即焚不吃灰，亲！")

  • System.out.println("【关个注】，后续浏览查看不迷路哟，亲！")

  • console.info("【转个发】，让更多的志同道合的朋友一起学习交流，亲！")

  • cout << "【留个言】，文章写得好不好、有没有错误，一定要留言哟，亲! " << endl;

谢谢，各位帅哥、美女四连支持！！这就是我的动力！

 更多网络安全、系统运维、应用开发、全栈文章，尽在【个人博客 - https://blog.weiyigeek.top】站点，谢谢支持！

私有云 Ubuntu Cloud Kubernetes K8S 容器化 容器编排 容器自动化部署 WeiyiGeek K8S环境安装
本文为我原创，未经授权禁止转载
cv16278860