Kubernetes 1.20.4 HA集群搭建基于【Ubuntu】&Rancher HA部署教程

环境

Ubuntu: 2020.4

Docker: 19.3

Kubernetes: v1.20.4

Rancher: v2.5.8

版本选择

k8s版本选择

如果要使用rancher请先到官网查看不同的版本支持的k8s的版本

https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.5.8/

可以看到rancher v2.5.8 支持的k8s最新版是v1.20.4

Docker版本选择

https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG

找到k8s版本的changelog 查看k8s支持的docker版本

系统初始化

设置系统主机名以及 Host 文件的相互解析

hostnamectl  set-hostname  k8s-01

vim /etc/hosts


192.168.70.20 k8s-01
192.168.70.21 k8s-02
192.168.70.22 k8s-03

添加多个ip解析

关闭防火墙

systemctl  stop firewalld  &&  systemctl  disable firewalld

设置iptables

sudo apt-get install -y  ipvsadm  iptables  wget  vim  net-tools 

关闭缓存

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

关闭 SELINUX

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

调整内核参数，对于 K8S

cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0 # 开启 OOM    
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
cp kubernetes.conf  /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

调整系统时区

保证多个机器的时间一致

# 设置系统时区为 中国/上海
timedatectl set-timezone Asia/Shanghai
# 将当前的 UTC 时间写入硬件时钟
timedatectl set-local-rtc 0
# 重启依赖于系统时间的服务
systemctl restart rsyslog 
systemctl restart crond

设置 rsyslogd 和 systemd journald

mkdir -p /etc/systemd

cat > journald.conf <<EOF
[Journal]
# 持久化保存到磁盘
Storage=persistent

# 压缩历史日志
Compress=yes

SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000

# 最大占用空间 10G
SystemMaxUse=10G

# 单日志文件最大 200M
SystemMaxFileSize=200M

# 日志保存时间 2 周
MaxRetentionSec=2week

# 不将日志转发到 syslog
ForwardToSyslog=no
EOF

mv journald.conf /etc/systemd/

Docker安装

更新 apt 包索引

sudo apt-get update

安装 apt 依赖包，用于通过HTTPS来获取仓库:

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

添加 Docker 的官方 GPG 密钥：

没有这一步apt update的时候会报错

curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

使用以下指令设置稳定版仓库

sudo add-apt-repository \
   "deb [arch=amd64] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/ \
  $(lsb_release -cs) \
  stable"

再次更新 apt 包索引。

sudo apt-get update

最新版安装

sudo apt-get install docker-ce docker-ce-cli containerd.io

安装指定版本

apt-cache madison docker-ce

5:20.10.73-0ubuntu-focal 就是

sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

k8s集群搭建

高可用搭建

load balance采用官方推荐的HA proxy + keepalives策略

https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing

HAProxy安装

haproxy使用docker部署 ,多个master节点都需要部署

访问haproxy的dockerhub拉取合适的镜像

​ https://registry.hub.docker.com/\_/haproxy?tab=tags&page=1&ordering=last\_updated

创建haproxy.cfg配置文件

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#前页面配置 访问/dbs
listen  admin_stats
    bind  0.0.0.0:7050
    mode        http
    stats uri   /dbs
    stats realm     Global\ statistics
    stats auth  admin:123456
    
#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:${APISERVER_DEST_PORT}
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
# 注意 如果在集群刚部署时apiserver下面只留一个server 全部部署好后在加上
#---------------------------------------------------------------------
backend apiserver
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server  k8s-01 ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} 
        server  k8s-02 ${HOST2_ADDRESS}:${APISERVER_SRC_PORT} 
        server  k8s-03 ${HOST3_ADDRESS}:${APISERVER_SRC_PORT} 

编写启动脚本

该脚本默认haproxy.cfg在/opt/k8s/haproxy 目录下

vim start-haproxy.sh 


#!/bin/bash
HOST1_ADDRESS=192.168.70.20
HOST2_ADDRESS=192.168.70.21
HOST3_ADDRESS=192.168.70.22
APISERVER_SRC_PORT=6443
APISERVER_DEST_PORT=6444

docker run -d --restart=always --name HAProxy-K8S -p 6444:6444  -p 7050:7050 \
        -e HOST1_ADDRESS=$HOST1_ADDRESS \
        -e HOST2_ADDRESS=$HOST2_ADDRESS \
        -e HOST3_ADDRESS=$HOST3_ADDRESS \
        -e APISERVER_DEST_PORT=$APISERVER_DEST_PORT \
        -e APISERVER_SRC_PORT=$APISERVER_SRC_PORT \
        -v /opt/k8s/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg \
        haproxy

启动

chmod +x start-haproxy.sh

sudo ./start-haproxy.sh

验证

浏览器访问ip:7050/dbc 用户名 admin 密码 123456

有用的网站

配置: https://blog.csdn.net/get\_set/article/details/107998958

Keepalived安装

所有master节点都需要安装

安装

sudo apt install -y keepalived

创建配置文件

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   #可以填写你的hostname
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   #间隔
   interval 3
   #权重
    weight -2
   
    fall 3
          
}
vrrp_instance VI_1 {
   # 多个master 只有一个配置为MASTER其他为BACKUP
   state MASTER
   # ifconfig查看你的当前网卡 配置在这里
   interface eth0
   #当前ip地址
   mcast_src_ip 192.168.1.151
   #保证每个机器上的virtual_router_id不一样
   virtual_router_id 51
   priority 101
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass 123456
   }
   #配置为和你当前网段相同的ip
   virtual_ipaddress {
       192.168.1.160
   }
   #这部分在集群没有打起来之前需要先注释掉
   track_script {
      chk_apiserver
   }
}
EOF

启动

systemctl enable keepalived --now

验证

ip a

在你当前网卡下找到你配置ip,如果能够看到代表配置生效,否则就要排查问题.

有用的网站

配置: https://www.keepalived.org/manpage.html

原理: https://www.cnblogs.com/rexcheny/p/10778567.html

k8s集群搭建

添加阿里源

sudo apt-add-repository "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main"

curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg   | sudo apt-key add -

sudo apt-get update

安装kubeadm kubectl kubelet

最新版安装

sudo apt  -y install  kubeadm kubectl kubelet

systemctl enable kubelet.service

安装指定版本

apt-cache madison kubeadm

sudo apt -y install  kubeadm=<VERSION_STRING> kubectl=<VERSION_STRING> kubelet=<VERSION_STRING>

systemctl enable kubelet.service

查询k8s版本需要的镜像

kubeadm --kubernetes-version=v1.21.0 config images list 

下载镜像

因为k8s.gcr.io需要翻墙,所以从阿里云下载,所有master节点都需要下载

docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.21.0
docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.21.0
docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.21.0
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.21.0
docker pull registry.aliyuncs.com/google_containers/pause:3.2
docker pull registry.aliyuncs.com/google_containers/etcd:3.4.13-0
docker pull coredns/coredns:1.7.0

之所以从dockerhub上下载dns是因为aliyun下没有它的镜像,之后讲coredns的镜像重新打tag

docker images|grep  coredns

docker tag 镜像id docker pull registry.aliyuncs.com/google_containers/coredns:1.7.0

获取k8s的配置文件

kubeadm config print init-defaults > kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  #修改为当前ip地址
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  #修改为当前的hostname
  name: k8s-02
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
#默认配置下没有这个配置,如果是集群需要手动加上,否则不需要
#这个ip为keepalived配置的vip,端口为haproxy监听的端口
controlPlaneEndpoint: 192.168.70.100:6444
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.4
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  #该位置也需要手动加上是fannel的配置
  podSubnet: 10.244.0.0/16
scheduler: {}
---
#下面的配置是启动ipvs的配置,每个版本都不一样
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

ipvs配置查看

https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md

集群初始化

kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.log

启动成功会出下如下画面

这时一个master节点已经搭建成功,首先指定下列命令

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
 root用户指定
 export KUBECONFIG=/etc/kubernetes/admin.conf

其他机器加入到master,执行

 在指定该命令之前,请确保你的第一个主节点上的haproxy代理的一个server
 
 kubeadm join 192.168.70.100:6444 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:db5964c163eaf5a280616ab0736dae7056293855a6d1811d78496e070bc94202 \
    --control-plane --certificate-key 7f819f2d00cfdf75e8318a1dc40bc41dbf2540084742a5fa31c68d49316ff390

加入node节点执行

kubeadm join 192.168.70.100:6444 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:db5964c163eaf5a280616ab0736dae7056293855a6d1811d78496e070bc94202 

所有节点全部加入以后执行

kubectl get nodes

可以看到集群是NotReady的状态,需要部署flannel

Flannel部署

官网: https://github.com/flannel-io/flannel

部署

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

网络原因可以先将kube-flannel.yml下载下来

kubectl apply -f  kube-flannel.yml

部署完成后

kubectl get nodes

Rancher部署

helm安装

https://helm.sh/zh/docs/intro/install/

ingress-controller安装

官网: https://kubernetes.github.io/ingress-nginx/deploy/

github: https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/index.md

执行Bare-metal下的安装命令

Rancher安装

中文文档: http://docs.rancher.cn/rancher2.5/

安装: http://docs.rancher.cn/docs/rancher2.5/installation/install-rancher-on-k8s/\_index

• latest: 建议在尝试新功能时使用。
• stable: 建议在生产环境中使用。（推荐）
• alpha: 未来版本的实验性预览。

helm repo add rancher-<CHART_REPO> http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/<CHART_REPO>

为 Rancher 创建 Namespace

kubectl create namespace cattle-system

创建证书

创建自签名证书

http://docs.rancher.cn/docs/rancher2.5/installation/resources/advanced/self-signed-ssl/\_index/

#!/bin/bash -e

help ()
{
    echo  ' ================================================================ '
    echo  ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'
    echo  ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；'
    echo  ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'
    echo  ' --ssl-size: ssl加密位数，默认2048；'
    echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
    echo  ' 使用示例:'
    echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
    echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
    echo  ' ================================================================'
}

case "$1" in
    -h|--help) help; exit;;
esac

if [[ $1 == '' ]];then
    help;
    exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
    key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
    value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
    case "$key" in
        --ssl-domain) SSL_DOMAIN=$value ;;
        --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
        --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
        --ssl-size) SSL_SIZE=$value ;;
        --ssl-date) SSL_DATE=$value ;;
        --ca-date) CA_DATE=$value ;;
        --ssl-cn) CN=$value ;;
    esac
done

# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}

SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt

echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"

if [[ -e ./${CA_KEY} ]]; then
    echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"
    mv ${CA_KEY} "${CA_KEY}"-bak
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
    echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi

if [[ -e ./${CA_CERT} ]]; then
    echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"
    mv ${CA_CERT} "${CA_CERT}"-bak
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
    echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi

echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM

if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then
    cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
    IFS=","
    dns=(${SSL_TRUSTED_DOMAIN})
    dns+=(${SSL_DOMAIN})
    for i in "${!dns[@]}"; do
      echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
    done

    if [[ -n ${SSL_TRUSTED_IP} ]]; then
        ip=(${SSL_TRUSTED_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi

echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}

echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}

echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
    -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_DATE} -extensions v3_req \
    -extfile ${SSL_CONFIG}

echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt

生成证书

/start.sh --ssl-domain=rancher.test.cn --ssl-trusted-domain=rancher.test.cn --ssl-trusted-ip=192.168.20.101,192.168.21.30,192.168.21.31 --ssl-size=2048 --ssl-date=3650

生效

kubectl -n cattle-system create secret tls tls-rancher-ingress   --cert=tls.crt   --key=tls.key
kubectl -n cattle-system create secret generic tls-ca   --from-file=cacerts.pem=./cacerts.pem

使用helm安装rancher

这里的hostname需要和上面生成的证书域名保持一致

helm install rancher rancher-stable/rancher   --namespace cattle-system   --set hostname=rancher.test.cn   --set ingress.tls.source=secret   --set privateCA=true

验证

kubectl -n cattle-system rollout status deploy/rancher

若出现下面信息（successfully rolled out）则表示成功，否则继续等待：

deployment "rancher" successfully rolled out

检查 deployment 的状态：

kubectl -n cattle-system get deploy rancher

若出现下面信息则表示安装成功

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   3/3     3            3           2d1h

UI 访问

正常情况下在电脑下修改host 配置域名和ip 通过域名访问即可

https://域名

非正常情况

kubectl get svc -n cattle-system

从图上可以rancher的port 是 80 和443 端口,但是它的类型是ClusterIP.这代表这个ip和端口都是虚拟的我们是不能直接访问的,需要通过ingress暴漏一个NodePort的端口让我们访问

暴漏虚拟端口

vim service-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kubectl apply -f  service-nodeport.yaml

kubectl get svc -n ingress-nginx

如上图所示,该服务的类型是NodePort ,我们可以通过30001访问80端口

https://域名:30002访问