k8s17.3二进制部署

发表于 2020-05-04 更新于 2020-10-21 分类于 k8s 阅读次数：本文字数： 91k 阅读时长 ≈ 1:22

K8s简介

Kubernetes（简称k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能。，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。它支持一系列容器工具, 包括Docker等。

# kubernetes

注意本文所有操作均在所有节点执行

本文环境

名称	配置
Kubernetes版本	v1.17.3
Cluster Pod CIDR	10.244.0.0/16
Cluster Service CIDR	10.250.0.0/24
kubernetes service	10.250.0.1
dns service	10.250.0.10

本文档主要介绍在裸机环境下以二进制的方式安装部署kubernetes

本次文档主要是对于kubernetes 1.17.3版本

内容主要有环境准备安装配置docker 升级内核调整系统参数

环境准备

主机名	IP	组件	配置
k8s-master1	192.168.220.101	apiserver、controller-manager、schedule、docker-ce	2核 4G
k8s-master2	192.168.220.102	apiserver、controller-manager、schedule、docker-ce	2核 4G
k8s-node1	192.168.220.103	kube-proxy、kubelet、docker-ce,etcd	2核 4G
k8s-node1	192.168.220.104	kube-proxy、kubelet、docker-ce,etcd	2核 4G
k8s-node1	192.168.220.105	kube-proxy、kubelet、docker-ce,etcd	2核 4G

以下操作除非具体说明的步骤其他的均在所有节点执行

环境变量

此环境变量为主节点（控制节点）的主机名和IP地址

export VIP=192.168.220.100
export MASTER1_HOSTNAME=k8s-master1
export MASTER1_IP=192.168.220.101
export MASTER2_HOSTNAME=k8s-master2
export MASTER2_IP=192.168.220.102
export NODE1_HOSTNAME=k8s-node1
export NODE1_IP=192.168.220.103
export NODE1_HOSTNAME=k8s-node2
export NODE1_IP=192.168.220.104
export NODE1_HOSTNAME=k8s-node3
export NODE1_IP=192.168.220.105

SSH免密/时间同步/主机名修改

· SSH免密

· NTP时间同步

· 主机名修改

· 环境变量生成

· Host 解析

这里需要说一下，所有的密钥分发以及后期拷贝等都在master1上操作，因为master1做免密了

k8s集群所有的机器都需要进行host解析

cat >> /etc/hosts << EOF
192.168.220.101 k8s-master1
192.168.220.102 k8s-master2
192.168.220.103 k8s-node1
192.168.220.104 k8s-node2
192.168.220.105 k8s-node3
EOF

批量免密

# 做免密前请修改好主机名对应的host

yum -y install expect
ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa
for i in 192.168.220.101 192.168.220.102 192.168.220.103 192.168.220.104 192.168.220.105;do
expect -c "
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i
    expect {
        \"*yes/no*\" {send \"yes\r\"; exp_continue}
        \"*password*\" {send \"quhuixx\r\"; exp_continue}
        \"*Password*\" {send \"quhuixx\r\";}
    } "
done

切记所有机器需要自行设定ntp,否则不只HA下apiserver通信有问题,各种千奇百怪的问题。

yum -y install ntp
systemctl enable ntpd
systemctl start ntpd
ntpdate -u time1.aliyun.com
hwclock --systohc
timedatectl set-timezone Asia/Shanghai

测试通信

1	for i in k8s-master1 k8s-master2 k8s-node1 k8s-node2 k8s-node3; do ssh root@$i "hostname";done

升级内核

安装内核语言编译器（内核是用perl语言编写的）

1	yum -y install perl

下载密钥和yum源

导入密钥

1	rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

安装7版本的yum源文件

1	yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

安装 ml版本 5版本的内核名字叫ml

1	yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64

附：4.4版本内核安装方法

1	yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64

然后配置从新的内核启动

1	grub2-set-default 0

然后重新生成grub2.cfg 使虚拟机使用新内核启动

1	grub2-mkconfig -o /boot/grub2/grub.cfg

调整系统内核参数

cat >> /etc/sysctl.conf <<EOF
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

然后执行sysctl -p 使配置的内核响应参数生效

sysctl -p

关闭防火墙、selinux、swap、NetworkManager

systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
iptables -P FORWARD ACCEPT
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
systemctl stop NetworkManager
systemctl disable NetworkManager

修改资源限制

echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft memlock unlimited"  >> /etc/security/limits.conf
echo "* hard memlock unlimited"  >> /etc/security/limits.conf

常用软件安装

yum -y install bridge-utils chrony ipvsadm ipset sysstat conntrack libseccomp wget tcpdump screen vim nfs-utils bind-utils wget socat telnet sshpass net-tools sysstat lrzsz yum-utils device-mapper-persistent-data lvm2 tree nc lsof strace nmon iptraf iftop rpcbind mlocate ipvsadm

加载内核ipvs模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
lsmod | grep -e ip_vs -e nf_conntrack_ipv4

安装docker-ce：

添加yun源

tee /etc/yum.repos.d/docker-ce.repo <<-'EOF'
[aliyun-docker-ce]
name=aliyun-docker-ce
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/
enable=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF

安装docker-ce

1	yum -y install docker-ce

重启docker并设置为开机自启

1
2
3

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

然后重启系统验证内核是否升级成功

下载命令

下载cfssl以及cfssljson命令

此举目的是下载创建证书以及配置文件需要的命令

PKI基础概念

什么是PKI？

公开密钥基础建设（英语：Public Key Infrastructure，缩写：PKI），又称公开密钥基础架构、公钥基础建设、公钥基础设施、公开密码匙基础建设或公钥基础架构，是一组由硬件、软件、参与者、管理政策与流程组成的基础架构，其目的在于创造、管理、分配、使用、存储以及撤销数字证书。（节选维基百科）

PKI是借助CA（权威数字证书颁发/认证机构）将用户的个人身份跟公开密钥链接在一起，它能够确保每个用户身份的唯一性，这种链接关系是通过注册和发布过程实现，并且根据担保级别，链接关系可能由CA和各种软件或在人为监督下完成。PKI用来确定链接关系的这一角色称为RA（Registration Authority, 注册管理中心），RA能够确保公开密钥和个人身份链接，可以防抵赖，防篡改。在微软的公钥基础建设下，RA又被称为CA，目前大多数称为CA。

PKI组成要素

从上面可以得知PKI的几个主要组成要素，用户（使用PKI的人或机构），认证机构（CA，颁发证书的人或机构），仓库（保存证书的数据库）等。

非对称加密

本文提到的密钥均为非对称加密，有公钥和私钥之分，并且他们总是成对出现，它们的特点就是其中一个加密的数据，只能使用另一个解密，即使它本身也无法解密，也就是说公钥加密的，私钥可以解密，私钥加密的，公钥可以解密。

证书签名请求CSR

它是向CA机构申请数字证书时使用的请求文件，这里的CSR不是证书，而向权威证书颁发机构获得签名证书的申请，当CA机构颁发的证书过期时，你可以使用相同的CSR来申请新的证书，此时key不变。

数字签名

数字签名就是“非对称加密+摘要算法”，其目的不是为了加密，而是为了防抵赖或者他们篡改数据。其核心思想是：比如A要给B发送数据，A先用摘要算法得到数据的指纹，然后用A的私钥加密指纹，加密后的指纹就是A的签名，B收到数据和A的签名后，也用同样的摘要算法计算指纹，然后用A公开的公钥解密签名，比较两个指纹，如果相同，说明数据没有被篡改，确实是A发过来的数据。假设C想改A发给B的数据来欺骗B，因为篡改数据后指纹会变，要想跟A的签名里面的指纹一致，就得改签名，但由于没有A的私钥，所以改不了，如果C用自己的私钥生成一个新的签名，B收到数据后用A的公钥根本就解不开。(来源于网络)

数字证书格式

数字证书格式有很多，比如.pem，.cer或者.crt等。

PKI工作流程

下图来源于网络，上半部分最右边就是CA机构，可以颁发证书。证书订阅人，首先去申请一个证书，为了申请这个证书，需要去登记，告诉它，我是谁，我属于哪个组织，到了登记机构，再通过CSR，发送给CA中心，CA中心通过验证通过之后，会颁发一对公钥和私钥，并且公钥会在CA中心存一份；证书订阅人拿到证书以后，部署在服务器；

当用户访问我们的web服务器时，它会请求我们的证书，服务器会把公钥发送给我们的客户端，客户端会去验证我们证书的合法性，客户端是如何验证证书是否有效呢？CA中心会把过期证书放在CRL服务器上面，这个CRL服务会把所有过期的证书形成一条链条，所以他的性能非常的差，所以又推出了OCSP程序，OCSP可以就一个证书进行查询，它是否过期，浏览器可以直接去查询OCSP响应程序，但OCSP响应程序效率还不是很高，最终往往我们会把web服务器如nginx有一个ocsp开关，当我们打开这个开关以后，会有nginx服务器主动的去ocsp服务器去查询，这样大量的客户端直接从web服务器就可以直接获取到证书是否有效。

CFSSL介绍

CFSSL是什么？

cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有cfssl，是CFSSL的命令行工具，cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。

安装CFSSL

可能下载时间较长

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
cfssl version

版本应该是1.2.0及以上版本

显示信息

1
2
3

Version: 1.2.0
Revision: dev
Runtime: go1.6

生成证书

注意本文所有操作均在master1节点执行

主要内容为生成kubernetes集群所需要的各种证书

主要有两个部分生成etcd的证书和生成kubernetes组件的证书

生成etcd证书

1 创建生成证书和临时存放证书的目录

1	mkdir /root/ssl/{etcd,kubernetes} -p

进入etcd目录

1	cd /root/ssl/etcd/

2 创建用来生成CA文件的JSON配置文件

此CA文件只用与etcd的证书

cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "etcd": {
         "expiry": "876000h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

server auth 表示client可以对使用该ca对server提供的证书进行验证

client auth 表示server可以使用该ca对client提供的证书进行验证

3 创建用来生成CA证书签名请求（CSR）的JSON配置文件

cat << EOF | tee ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

4 生成CA证书和私钥

1	cfssl gencert -initca ca-csr.json \| cfssljson -bare ca

输出内容

2019/10/12 19:35:14 [INFO] generating a new CA key and certificate from CSR
2019/10/12 19:35:14 [INFO] generate received request
2019/10/12 19:35:14 [INFO] received CSR
2019/10/12 19:35:14 [INFO] generating key: rsa-2048
2019/10/12 19:35:14 [INFO] encoded CSR
2019/10/12 19:35:14 [INFO] signed certificate with serial number 76399392328271693420688681207409409662642174207

查看生成的CA证书和私钥

1	ls ca*.pem

输出内容

1	ca-key.pem ca.pem

5 创建etcd证书请求

cat << EOF | tee etcd-csr.json
{
    "CN": "etcd",
    "hosts": [
      "200.200.100.71",
      "200.200.100.72",
      "200.200.100.73"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

6 生成etcd证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=etcd etcd-csr.json | cfssljson -bare etcd

输出内容

2019/10/12 19:39:16 [INFO] generate received request
2019/10/12 19:39:16 [INFO] received CSR
2019/10/12 19:39:16 [INFO] generating key: rsa-2048
2019/10/12 19:39:17 [INFO] encoded CSR
2019/10/12 19:39:17 [INFO] signed certificate with serial number 276878925110307603699002043209122885766807800060
2019/10/12 19:39:17 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

7 查看生成的所有etcd证书

1	ls \| grep pem

输出的4个文件

ca-key.pem
ca.pem
etcd-key.pem
etcd.pem

生成kubernetes组件证书

切换到kubernetes组件证书申请和存放目录

1	cd /root/ssl/kubernetes/

新建CA配置文件

用于kubernetes集群的组件和admin角色

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "8760h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

新建CA凭证签发请求文件

cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "hosts": [
    "127.0.0.1",
    "200.200.100.71",
    "200.200.100.72",
    "200.200.100.73",
    "200.200.100.70"
 ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Beijing",
      "ST": "Beijing"
    }
  ]
}
EOF

生成CA凭证和私钥

1	cfssl gencert -initca ca-csr.json \| cfssljson -bare ca

输出信息

2019/10/12 19:56:26 [INFO] generating a new CA key and certificate from CSR
2019/10/12 19:56:26 [INFO] generate received request
2019/10/12 19:56:26 [INFO] received CSR
2019/10/12 19:56:26 [INFO] generating key: rsa-2048
2019/10/12 19:56:26 [INFO] encoded CSR
2019/10/12 19:56:26 [INFO] signed certificate with serial number 679742542757179200541008226092035525850208663173

查看创建的证书和私钥

1	ls ca*.pem

输出文件

1	ca-key.pem ca.pem

client与server凭证

创建admin client 凭证签发请求文件

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

创建admin client 凭证和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin

输出信息

2019/10/12 19:59:38 [INFO] generate received request
2019/10/12 19:59:38 [INFO] received CSR
2019/10/12 19:59:38 [INFO] generating key: rsa-2048
2019/10/12 19:59:38 [INFO] encoded CSR
2019/10/12 19:59:38 [INFO] signed certificate with serial number 514625224786356937263551808946632861542829130401
2019/10/12 19:59:38 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

查看生成的文件

1 2	ls admin*.pem admin-key.pem admin.pem

生成kubelet客户端凭证

kubernetes使用special-purpose authorization mode(被称作 Node Authorizer) 授权来自kubelet的API请求

为了通过Node Authorizer的授权，kubelet 必须使用一个署名为system:node:的凭证来证明它属于system:nodes用户组。

本节将会给每台节点（包括master节点）创建凭证和私钥

创建master1节点的凭证签发请求文件

cat << EOF | tee k8s-master1-csr.json 
{
  "CN": "system:node:${MASTER1_HOSTNAME}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:nodes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成master节点的证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${MASTER1_HOSTNAME},${MASTER1_IP} \
  -profile=kubernetes \
 k8s-master1-csr.json | cfssljson -bare k8s-master1

输出信息

2019/10/12 20:08:33 [INFO] generate received request
2019/10/12 20:08:33 [INFO] received CSR
2019/10/12 20:08:33 [INFO] generating key: rsa-2048

2019/10/12 20:08:33 [INFO] encoded CSR
2019/10/12 20:08:33 [INFO] signed certificate with serial number 340503546795644080420594727795505971193705840974

输出的文件

1 2	ls k8s-master*.pem k8s-master1-key.pem k8s-master1.pem

创建master2节点的凭证签发请求文件

cat << EOF | tee k8s-master2-csr.json
{
  "CN": "system:node:${MASTER2_HOSTNAME}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:nodes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成master2节点的证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${MASTER2_HOSTNAME},${MASTER2_IP} \
  -profile=kubernetes \
 k8s-master2-csr.json | cfssljson -bare k8s-master2

输出信息

2019/10/12 20:08:33 [INFO] generate received request
2019/10/12 20:08:33 [INFO] received CSR
2019/10/12 20:08:33 [INFO] generating key: rsa-2048

2019/10/12 20:08:33 [INFO] encoded CSR
2019/10/12 20:08:33 [INFO] signed certificate with serial number 340503546795644080420594727795505971193705840974

输出的文件

1 2	ls k8s-master*.pem k8s-master1-key.pem k8s-master1.pem k8s-master2-key.pem k8s-master2.pem

创建master3节点的凭证签发请求文件

cat << EOF | tee k8s-master3-csr.json
{
  "CN": "system:node:${MASTER3_HOSTNAME}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:nodes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成k8s-node节点的证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${MASTER3_HOSTNAME},${MASTER3_IP} \
  -profile=kubernetes \
  k8s-master3-csr.json | cfssljson -bare k8s-master3

输出信息

2019/10/12 20:11:22 [INFO] generate received request
2019/10/12 20:11:22 [INFO] received CSR
2019/10/12 20:11:22 [INFO] generating key: rsa-2048
2019/10/12 20:11:22 [INFO] encoded CSR
2019/10/12 20:11:22 [INFO] signed certificate with serial number 329201759031912279536498320815194792351902510021

输出的文件

1 2	ls k8s-master3*.pem k8s-master3-key.pem k8s-master3.pem

创建k8s-node1节点的凭证签发请求文件

cat << EOF | tee k8s-node1-csr.json
{
  "CN": "system:node:${NODE1_HOSTNAME}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:nodes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成k8s-node1节点的证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${NODE1_HOSTNAME},${NODE1_IP} \
  -profile=kubernetes \
  k8s-node1-csr.json | cfssljson -bare k8s-node1

输出信息

2019/10/12 20:16:27 [INFO] generate received request
2019/10/12 20:16:27 [INFO] received CSR
2019/10/12 20:16:27 [INFO] generating key: rsa-2048
2019/10/12 20:16:27 [INFO] encoded CSR
2019/10/12 20:16:27 [INFO] signed certificate with serial number 11529605845303364851563251013549393798169113866

输出文件

1 2	ls k8s-node1*.pem k8s-node1-key.pem k8s-node1.pem

8 创建master组件需要的证书

创建kube-controller-manager客户端凭证

cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

输出信息

2019/10/12 20:29:06 [INFO] generate received request
2019/10/12 20:29:06 [INFO] received CSR
2019/10/12 20:29:06 [INFO] generating key: rsa-2048
2019/10/12 20:29:06 [INFO] encoded CSR
2019/10/12 20:29:06 [INFO] signed certificate with serial number 173346030426505912970345315612511532042452194730
2019/10/12 20:29:06 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

输出文件

1 2	ls kube-con*.pem kube-controller-manager-key.pem kube-controller-manager.pem

创建kube-proxy客户端凭证

cat <<EOF |tee kube-proxy-csr.json 
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:node-proxier",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy

输出信息

2019/10/12 20:31:11 [INFO] generate received request
2019/10/12 20:31:11 [INFO] received CSR
2019/10/12 20:31:11 [INFO] generating key: rsa-2048
2019/10/12 20:31:11 [INFO] encoded CSR
2019/10/12 20:31:11 [INFO] signed certificate with serial number 3973180903081703880688638425637585151040946194
2019/10/12 20:31:11 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

输出文件

1 2	ls kube-proxy*.pem kube-proxy-key.pem kube-proxy.pem

创建kube-scheduler凭证签发

cat <<EOF | tee kube-scheduler-csr.json 
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成证书

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler

输出信息

2019/10/12 20:18:57 [INFO] generate received request
2019/10/12 20:18:57 [INFO] received CSR
2019/10/12 20:18:57 [INFO] generating key: rsa-2048
2019/10/12 20:18:57 [INFO] encoded CSR
2019/10/12 20:18:57 [INFO] signed certificate with serial number 56094122509645103760584094055826646549201635795
2019/10/12 20:18:57 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

输出文件

1 2	ls kube-sch*.pem kube-scheduler-key.pem kube-scheduler.pem

创建kubernetes 证书

为了保证客户端与kubernetes API的认证，kubernetes API Server 凭证中必须包含master的静态IP地址

此IP地址使用上面配置的环境变量

创建kubernetes API Server 凭证签发请求文件

cat <<EOF | tee kubernetes-csr.json 
{
  "CN": "kubernetes",
  "hosts": [
   "127.0.0.1",
   "200.200.100.71",
   "200.200.100.72",
   "200.200.100.73",
   "200.200.100.70",
   "10.250.0.1",
   "kubernetes",
   "kubernetes.default",
   "kubernetes.default.svc",
   "kubernetes.default.svc.cluster",
   "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成kubernetes API Server 凭证与私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes

输出信息

2019/10/12 20:23:03 [INFO] generate received request
2019/10/12 20:23:03 [INFO] received CSR
2019/10/12 20:23:03 [INFO] generating key: rsa-2048
2019/10/12 20:23:03 [INFO] encoded CSR
2019/10/12 20:23:03 [INFO] signed certificate with serial number 319608271292119912072742471756939391576493389087

输出文件

1 2	ls kubernetes*.pem kubernetes-key.pem kubernetes.pem

Service Account 证书

创建凭证签发问

cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "China",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes",
      "ST": "Beijing"
    }
  ]
}
EOF

生成证书和私钥

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

输出信息

2019/10/12 20:25:31 [INFO] generate received request
2019/10/12 20:25:31 [INFO] received CSR
2019/10/12 20:25:31 [INFO] generating key: rsa-2048
2019/10/12 20:25:31 [INFO] encoded CSR
2019/10/12 20:25:31 [INFO] signed certificate with serial number 538955391110960009078645942634491132767864895292
2019/10/12 20:25:31 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

输出文件

1 2	ls service*.pem service-account-key.pem service-account.pem

9 拷贝etcd证书到相应节点的相应目录

创建etcd目录

for host in k8s-master1 k8s-master2 k8s-master3;do
  echo "---$host---"
  ssh root@$host "mkdir /usr/local/etcd/{bin,ssl,data,json,src} -p"; 
done

拷贝etcd证书

cd ../etcd/
拷贝方法一；
scp etcd-key.pem etcd.pem ca.pem ca-key.pem k8s-master1:/usr/local/etcd/ssl/
scp etcd-key.pem etcd.pem ca.pem ca-key.pem k8s-master2:/usr/local/etcd/ssl/
scp etcd-key.pem etcd.pem ca.pem ca-key.pem k8s-master3:/usr/local/etcd/ssl/

 
拷贝方法二；
for host in k8s-master1 k8s-master2 k8s-master3; \
     do     
    echo "--- $host---"
    scp -r *.pem $host:/usr/local/etcd/ssl/
done

配置和生成kubernetes配置文件

本文档是kubernetes1.17.3二进制安装的第四篇

注意本文所有操作均在master节点执行

本文将主要介绍创建kubeconfig配置文件她们是kubernetes客户端与API Server 认证与鉴权的保证

kubectl是kubernetes命令行客户端，一般情况集群都开启了TLS认证，kubectl或其它客户端每次与集群kube-apiserver交互都少不了身份验证，目前有两种常用认证方式，使用证书和token，这两种方式也是最通用的方式，本节简单说下kubectl客户端如何使用证书的认证方式访问集群。

使用证书的方式，一般情况下我们需要创建一个kubeconfig配置文件，这个文件用来组织有关集群、用户、命名空间和身份认证机制的信息。kubectl使用kubeconfig配置文件来查找选择集群所需信息，并且集群kube-apiserver进行通信，kubectl默认查到${HOME}/.kube目录下面的config文件，当然也可以通过设置KUBECONFIG环境变量或者在命令行使用–kubeconfig参数指定kubeconfig配置文件。

配置详情

步骤	配置选项	选项说明
1. 设置集群信息	set-cluster	kubectl config 设置集群信息时使用
–certificate-authority	设置集群的根证书路径
–embed-certs	将–certificate-authority根证书写入到kubeconfig中
–server	指定访问集群的socket
–kubeconfig	kubeconfig配置文件路径
2. 设置客户端参数	set-credentials	kubectl config 设置客户端认证信息
–client-certificate	指定kubectl使用的证书路径
–client-key	指定kubectl使用的私钥路径
–embed-certs	将kubectl使用的证书和私钥写入到kubeconfig中
–kubeconfig	kubeconfig配置文件路径
3. 设置上下文信息	set-context	kubectl config 设置上下文参数
–cluster	配置使用哪个集群信息，set-cluster中设置的
–user	配置使用哪个客户端，set-credentials中设置的
–kubeconfig	kubeconfig配置文件路径
4. kubeconfig中使用哪个上下文	use-context	kubectl config 设置使用哪个上下文信息

客户端认证配置

本节将会创建用于 kube-proxy kube-controll-manager kube-scheduler 和kubelet 的kubeconfig文件

kubelet配置文件

为了确保Node Authorizer授权 kubelet配置文件中的客户端证书必须匹配Node名字

Node名字还使用生成证书那一节配置的环境变量

为所有节点创建kubeconfig配置（都在master节点操作）

生成配置文件所在的目录就在上一节生成kubernetes组件所在的目录

首先安装kubectl

这里一次性将所有需要的软件都安装上

wget --timestamping \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kube-apiserver" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kube-controller-manager" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kube-scheduler" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kubectl"
chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl



#或者一次性下载
https://dl.k8s.io/v1.17.3/kubernetes-server-linux-amd64.tar.gz

#解压缩
tar -zxvf /root/kubernetes-server-1.17.3-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#拷贝文件至另外两个master节点
for host in k8s-master2 k8s-master3; do
  echo "---$host---"
  scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $host:/usr/local/bin/
done


#拷贝文件至node节点
scp /usr/local/bin/kube{let,-proxy} k8s-node1:/usr/local/bin/
cd /root/ssl/kubernetes/

不用环境变量的方法

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=k8s-master1.kubeconfig

kubectl config set-credentials system:node:k8s-master1 \
  --client-certificate=k8s-master1.pem \
  --client-key=k8s-master1-key.pem \
  --embed-certs=true \
  --kubeconfig=k8s-master1.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:node:k8s-master1 \
  --kubeconfig=k8s-master1.kubeconfig

kubectl config use-context default --kubeconfig=k8s-master1.kubeconfig

使用环境变量的方法

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=${MASTER1_HOSTNAME}.kubeconfig

kubectl config set-credentials system:node:${MASTER1_HOSTNAME} \
  --client-certificate=${MASTER1_HOSTNAME}.pem \
  --client-key=${MASTER1_HOSTNAME}-key.pem \
  --embed-certs=true \
  --kubeconfig=${MASTER1_HOSTNAME}.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:node:${MASTER1_HOSTNAME} \
  --kubeconfig=${MASTER1_HOSTNAME}.kubeconfig

kubectl config use-context default --kubeconfig=${MASTER1_HOSTNAME}.kubeconfig

输出文件

ls k8s-master*config
k8s-master1.kubeconfig
kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=k8s-master2.kubeconfig

kubectl config set-credentials system:node:k8s-master2 \
  --client-certificate=k8s-master2.pem \
  --client-key=k8s-master2-key.pem \
  --embed-certs=true \
  --kubeconfig=k8s-master2.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:node:k8s-master2 \
  --kubeconfig=k8s-master2.kubeconfig

kubectl config use-context default --kubeconfig=k8s-master2.kubeconfig

输出文件

1
2
3

ls k8s-master*config

k8s-master1.kubeconfig k8s-master2.kubeconfig

k8s-master3节点的配置文件

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=k8s-master3.kubeconfig

kubectl config set-credentials system:node:k8s-master3 \
  --client-certificate=k8s-master3.pem \
  --client-key=k8s-master3-key.pem \
  --embed-certs=true \
  --kubeconfig=k8s-master3.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:node:k8s-master3 \
  --kubeconfig=k8s-master3.kubeconfig

kubectl config use-context default --kubeconfig=k8s-master3.kubeconfig

输出文件

1 2	ls k8s-master3*config k8s-master3.kubeconfig

k8s-node1节点配置文件

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=k8s-node1.kubeconfig

kubectl config set-credentials system:node:k8s-node1 \
  --client-certificate=k8s-node1.pem \
  --client-key=k8s-node1-key.pem \
  --embed-certs=true \
  --kubeconfig=k8s-node1.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:node:k8s-node1 \
  --kubeconfig=k8s-node1.kubeconfig

kubectl config use-context default --kubeconfig=k8s-node1.kubeconfig

输出文件

1 2	ls k8s-node1*config k8s-node1.kubeconfig

kube-proxy配置文件

为kube-proxy服务生成kubeconfig配置文件

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
  --client-certificate=kube-proxy.pem \
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

输出文件

1 2	ls kube-proxy*config kube-proxy.kubeconfig

kube-controller-manager配置文件

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

输出文件

1 2	ls kube-con*config kube-controller-manager.kubeconfig

kube-scheduler配置文件

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
  --client-certificate=kube-scheduler.pem \
  --client-key=kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

输出文件

1 2	ls kube-sch*config kube-scheduler.kubeconfig

Admin配置文件

kubectl config set-cluster kubernetes-training \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://${VIP}:8443 \
  --kubeconfig=admin.kubeconfig

kubectl config set-credentials admin \
  --client-certificate=admin.pem \
  --client-key=admin-key.pem \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes-training \
  --user=admin \
  --kubeconfig=admin.kubeconfig

kubectl config use-context default --kubeconfig=admin.kubeconfig

输出文件

1 2	ls ad*config admin.kubeconfig

配置和生成密钥

Kubernetes 存储了集群状态、应用配置和密钥等很多不同的数据。而 Kubernetes 也支持集群数据的加密存储。

本部分将会创建加密密钥以及一个用于加密 Kubernetes Secrets 的加密配置文件。

所有操作主节点执行

加密密钥
建立加密密钥:

1	ENCRYPTION_KEY=$(head -c 32 /dev/urandom \| base64)

加密配置文件
生成名为 encryption-config.yaml 的加密配置文件：

cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

分发证书文件

将 kubelet 与 kube-proxy kubeconfig 配置文件复制到每个 worker 节点上：

创建配置文件目录

for host in k8s-master1 k8s-master2 k8s-master3 k8s-node1 ; do ssh root@$host "mkdir -p \
  /opt/cni/bin \
  /var/lib/kubelet \
  /var/lib/kube-proxy \
  /var/lib/kubernetes \
  /var/run/kubernetes " ; done

将 admin、kube-controller-manager 与 kube-scheduler kubeconfig 配置文件复制到每个控制节点上：

分发文件方法一；
scp  ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  service-account-key.pem service-account.pem \
  encryption-config.yaml \
  kube-controller-manager.kubeconfig kube-scheduler.kubeconfig k8s-master1:/var/lib/kubernetes/
  scp admin.kubeconfig k8s-master1:~/
  

scp  ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  service-account-key.pem service-account.pem \
  encryption-config.yaml \
  kube-controller-manager.kubeconfig kube-scheduler.kubeconfig k8s-master2:/var/lib/kubernetes/
 
 
scp  ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  service-account-key.pem service-account.pem \
  encryption-config.yaml \
  kube-controller-manager.kubeconfig kube-scheduler.kubeconfig k8s-master3:/var/lib/kubernetes/
分发文件方法二；
  
for NODE in k8s-master1 k8s-master2 k8s-master3; do
  echo "-----$NODE------"
  scp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  service-account-key.pem service-account.pem \
  encryption-config.yaml \
  kube-controller-manager.kubeconfig kube-scheduler.kubeconfig $NODE:/var/lib/kubernetes/;
done
scp  k8s-master1-key.pem  k8s-master1.pem k8s-master1:/var/lib/kubelet/
scp  k8s-master1.kubeconfig k8s-master1:/var/lib/kubelet/kubeconfig
scp  kube-proxy.kubeconfig k8s-master1:/var/lib/kube-proxy/kubeconfig
scp  kube-proxy.pem k8s-master1:/var/lib/kube-proxy/
scp  kube-proxy-key.pem k8s-master1:/var/lib/kube-proxy/
scp kube-controller-manager-key.pem k8s-master1:/var/lib/kubernetes/kube-controller-manager-key.pem                                           
scp kube-controller-manager.pem k8s-master1:/var/lib/kubernetes/
scp kube-scheduler.pem k8s-master1:/var/lib/kubernetes/kube-scheduler.pem
scp kube-scheduler-key.pem k8s-master1:/var/lib/kubernetes/


scp  k8s-master2-key.pem  k8s-master2.pem k8s-master2:/var/lib/kubelet/
scp  k8s-master2.kubeconfig k8s-master2:/var/lib/kubelet/kubeconfig
scp  kube-proxy.kubeconfig k8s-master2:/var/lib/kube-proxy/kubeconfig
scp  kube-proxy-key.pem k8s-master2:/var/lib/kube-proxy/
scp  kube-proxy.pem k8s-master2:/var/lib/kube-proxy/
scp kube-controller-manager-key.pem k8s-master2:/var/lib/kubernetes/kube-controller-manager-key.pem                                           
scp kube-controller-manager.pem k8s-master2:/var/lib/kubernetes/
scp kube-scheduler.pem k8s-master2:/var/lib/kubernetes/kube-scheduler.pem
scp kube-scheduler-key.pem k8s-master2:/var/lib/kubernetes/

scp  k8s-master3-key.pem  k8s-master3.pem k8s-master3:/var/lib/kubelet/
scp  k8s-master3.kubeconfig k8s-master3:/var/lib/kubelet/kubeconfig
scp  kube-proxy.kubeconfig k8s-master3:/var/lib/kube-proxy/kubeconfig
scp  kube-proxy-key.pem k8s-master3:/var/lib/kube-proxy/
scp  kube-proxy.pem k8s-master3:/var/lib/kube-proxy/
scp kube-controller-manager-key.pem k8s-master3:/var/lib/kubernetes/kube-controller-manager-key.pem                                           
scp kube-controller-manager.pem k8s-master3:/var/lib/kubernetes/
scp kube-scheduler.pem k8s-master3:/var/lib/kubernetes/kube-scheduler.pem
scp kube-scheduler-key.pem k8s-master3:/var/lib/kubernetes/



scp  ca.pem k8s-node1:/var/lib/kubernetes/
scp  k8s-node1-key.pem  k8s-node1.pem k8s-node1:/var/lib/kubelet/
scp  k8s-node1.kubeconfig k8s-node1:/var/lib/kubelet/kubeconfig
scp  kube-proxy.kubeconfig k8s-node1:/var/lib/kube-proxy/kubeconfig
scp  kube-proxy-key.pem k8s-node1:/var/lib/kube-proxy/
scp  kube-proxy.pem k8s-node1:/var/lib/kube-proxy/

kubernetes的组件都是无状态的，所有集群的状态都存储在etcd集群中

部署etcd集群

etcd目录结构

软件包解压目录

/usr/local/etcd/src/

ssl证书申请文件

/usr/local/etcd/json

ssl证书文件

/usr/local/etcd/ssl/

可执行文件

/usr/local/etcd/bin/

工作目录

/usr/local/etcd/data/

etcd的目录在生成证书的最后已经全部创建并且证书文件也全部拷贝完成

下载etcd二进制包并解压

1
2
3

cd
wget https://github.com/etcd-io/etcd/releases/download/v3.4.1/etcd-v3.4.1-linux-amd64.tar.gz
tar -zxvf etcd-v3.4.1-linux-amd64.tar.gz -C /usr/local/etcd/src/

复制可执行文件

复制方法一；
scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcd k8s-master1:/usr/local/etcd/bin/
scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcdctl k8s-master1:/usr/local/etcd/bin/

scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcd k8s-master2:/usr/local/etcd/bin/
scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcdctl k8s-master2:/usr/local/etcd/bin/

scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcd k8s-master3:/usr/local/etcd/bin/
scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcdctl k8s-master2:/usr/local/etcd/bin/


复制方法二；
for host in  k8s-master1 k8s-master2 k8s-master3; do
  echo "----$host----"
  scp -p /usr/local/etcd/src/etcd-v3.4.1-linux-amd64/etcd* $host:/usr/local/etcd/bin/;
  ssh $host "ln -s /usr/local/etcd/bin/* /usr/bin/";
done
或者直接复制执行文件到 /usr/bin,这里方便管理复制到/usr/local/etcd/bin下

设置etcd系统服务文件

在/etc/systemd/system/目录下导入文件etcd.service

配置master节点的配置文件：

配置master1的etcd

cat << EOF | tee /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Restart=on-failure
LimitNOFILE=65536
ExecStart=/usr/local/etcd/bin/etcd  \\
  --name=etcd00   \\
  --data-dir=/usr/local/etcd/data/  \\
  --listen-peer-urls=https://$MASTER1_IP:2380 \\
  --listen-client-urls=https://$MASTER1_IP:2379,https://127.0.0.1:2379  \\
  --advertise-client-urls=https://$MASTER1_IP:2379  \\
  --initial-advertise-peer-urls=https://$MASTER1_IP:2380  \\
  --initial-cluster=etcd00=https://$MASTER1_IP:2380,etcd01=https://$MASTER2_IP:2380,etcd02=https://$MASTER3_IP:2380 \\
  --initial-cluster-token=etcd-cluster \\
  --initial-cluster-state=new \\
  --cert-file=/usr/local/etcd/ssl/etcd.pem \\
  --key-file=/usr/local/etcd/ssl/etcd-key.pem \\
  --peer-cert-file=/usr/local/etcd/ssl/etcd.pem \\
  --peer-key-file=/usr/local/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/usr/local/etcd/ssl/ca.pem \\
  --peer-trusted-ca-file=/usr/local/etcd/ssl/ca.pem


[Install]
WantedBy=multi-user.target
EOF


配置master2的etcd

cat << EOF | tee /etc/systemd/system/etcd2.service
[Unit]
Description=Etcd Server
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Restart=on-failure
LimitNOFILE=65536
ExecStart=/usr/local/etcd/bin/etcd  \\
  --name=etcd01   \\
  --data-dir=/usr/local/etcd/data/  \\
  --listen-peer-urls=https://$MASTER2_IP:2380 \\
  --listen-client-urls=https://$MASTER2_IP:2379,https://127.0.0.1:2379  \\
  --advertise-client-urls=https://$MASTER2_IP:2379  \\
  --initial-advertise-peer-urls=https://$MASTER2_IP:2380  \\
  --initial-cluster=etcd00=https://$MASTER1_IP:2380,etcd01=https://$MASTER2_IP:2380,etcd02=https://$MASTER3_IP:2380 \\
  --initial-cluster-token=etcd-cluster \\
  --initial-cluster-state=new \\
  --cert-file=/usr/local/etcd/ssl/etcd.pem \\
  --key-file=/usr/local/etcd/ssl/etcd-key.pem \\
  --peer-cert-file=/usr/local/etcd/ssl/etcd.pem \\
  --peer-key-file=/usr/local/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/usr/local/etcd/ssl/ca.pem \\
  --peer-trusted-ca-file=/usr/local/etcd/ssl/ca.pem

[Install]
WantedBy=multi-user.target
EOF




scp /etc/systemd/system/etcd2.service k8s-master2:/etc/systemd/system/etcd.service
rm -rf /etc/systemd/system/etcd2.service


配置master3的etcd

cat << EOF | tee /etc/systemd/system/etcd3.service
[Unit]
Description=Etcd Server
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Restart=on-failure
LimitNOFILE=65536
ExecStart=/usr/local/etcd/bin/etcd  \\
  --name=etcd02   \\
  --data-dir=/usr/local/etcd/data/  \\
  --listen-peer-urls=https://$MASTER3_IP:2380 \\
  --listen-client-urls=https://$MASTER3_IP:2379,https://127.0.0.1:2379  \\
  --advertise-client-urls=https://$MASTER3_IP:2379  \\
  --initial-advertise-peer-urls=https://$MASTER3_IP:2380  \\
  --initial-cluster=etcd00=https://$MASTER1_IP:2380,etcd01=https://$MASTER2_IP:2380,etcd02=https://$MASTER3_IP:2380 \\
  --initial-cluster-token=etcd-cluster \\
  --initial-cluster-state=new \\
  --cert-file=/usr/local/etcd/ssl/etcd.pem \\
  --key-file=/usr/local/etcd/ssl/etcd-key.pem \\
  --peer-cert-file=/usr/local/etcd/ssl/etcd.pem \\
  --peer-key-file=/usr/local/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/usr/local/etcd/ssl/ca.pem \\
  --peer-trusted-ca-file=/usr/local/etcd/ssl/ca.pem

[Install]
WantedBy=multi-user.target
EOF




scp /etc/systemd/system/etcd3.service k8s-master3:/etc/systemd/system/etcd.service
rm -rf /etc/systemd/system/etcd3.service

配置选项	选项说明
wal	存放预写式日志,文件中记录了整个数据变化的全部历程，数据的修改在提交前，都要先写入到WAL中
data-dir	指定节点的数据存储目录（包括：节点ID、集群ID、集群初始化配置、Snapshot文件等），如果未指定，会写在当前目录
wal-dir	存放预写式日志，文件中记录了整个数据变化的全部历程，数据的修改在提交前，都要先写入到WAL中，如果未指定，写在–data-dir目录下面
name	节点名称，如果–initial-cluster-state=new这个值为new，哪么–name的参数值必须位于–initial-cluster列表中
cert-file	客户端与服务器之间TLS证书文件的路径
key-file	客户端与服务器之间TLS密钥文件的路径
trusted-ca-file	签名client证书的CA证书，用于验证client证书
peer-cert-file	对等服务器TLS证书文件的路径
peer-key-file	对等服务器TLS密钥文件的路径
peer-client-cert-auth	启用对等客户端证书验证
client-cert-auth	启用客户端验证
listen-peer-urls	与集群其它成员之间的通信地址
initial-advertise-peer-urls	通告给集群其它节点，本地的对等URL地址
listen-client-urls	监听本地端口，对外提供服务的地址
advertise-client-urls	客户端URL，用于通告集群的其余部分
initial-cluster-token	集群的token，整个集群中保持一致
initial-cluster	集群中的所有信息节点
initial-cluster-state	初始化集群状态，默认为new
auto-compaction-mode	配置基于时间的三种模式
auto-compaction-retention	设置保留历史时间为1小时
max-request-bytes	服务器将接受的最大客户端请求大小
quota-backend-bytes	当后端大小超过给定的配额时，报警
heartbeat-interval	心跳间隔的时间，单位毫秒
election-timeout	选举超时时间，单位毫秒

所有节点启动etcd服务并设置为开机自启

启动服务方法一（每台单独执行）；

systemctl daemon-reload && systemctl enable etcd.service && systemctl restart etcd.service &

启动服务方法二（master1执行）；
for NODE in k8s-master1 k8s-master2 k8s-master3; do
    echo "--- $NODE ---"
    ssh $NODE "systemctl daemon-reload"
    ssh $NODE "systemctl enable --now etcd" &
done
wait

如果有某一台节点启动失败请手动启动

systemctl daemon-reload
systemctl restart etcd.service

for NODE in k8s-master1 k8s-master2 k8s-master3; do
    echo "--- $NODE ---"
    ssh $NODE "systemctl daemon-reload"
    ssh $NODE "systemctl start etcd" &
done
wait

检查etcd集群状态

三个节点的IP

etcdctl \
--cacert=/usr/local/etcd/ssl/ca.pem \
--cert=/usr/local/etcd/ssl/etcd.pem \
--key=/usr/local/etcd/ssl/etcd-key.pem \
--endpoints="https://200.200.100.71:2379,\
https://200.200.100.72:2379,https://200.200.100.73:2379" endpoint health

输出信息

1 2	https://200.200.100.71:2379 is healthy: successfully committed proposal: took = 14.100152ms https://200.200.100.72:2379 is healthy: successfully committed proposal: took = 29.074303ms

全部节点为healthy 表明etcd集群搭建成功

部署keepalived+HAProxy

1.信息可以按照自己的环境填写，或者和我相同

2.网卡名称都为eth0，如有不相同建议修改下面配置，或者直接修改centos7网卡为eth0

3.cluster dns或domain有改变的话,需要修改kubelet-conf.yml

HA(haproxy+keepalived) 单台master就不要用HA了

首先所有master安装haproxy+keeplived

for NODE in k8s-master1 k8s-master2 k8s-master3; do
  echo "--- $NODE---"
  ssh $NODE 'yum install haproxy keepalived -y' &
done

安装完记得检查 (是每台master进行检查)

for NODE in k8s-master1 k8s-master2 k8s-master3;do
  echo "--- $NODE ---"
  ssh $NODE "rpm -qa|grep haproxy"
  ssh $NODE "rpm -qa|grep keepalived"
done

在k8s-master1修改配置文件，并分发给其他master

· haproxy配置文件修改

cat << EOF | tee /etc/haproxy/haproxy.cfg
global
  log     127.0.0.1 local2
  chroot   /var/lib/haproxy
  pidfile   /var/run/haproxy.pid
  maxconn   4000
  user    haproxy
  group    haproxy
  daemon
 
defaults
  mode          tcp
  log           global
  retries         3
  timeout connect     10s
  timeout client     1m
  timeout server     1m
 
frontend kubernetes
  bind *:8443
  mode tcp
  option tcplog
  default_backend kubernetes-apiserver
 
backend kubernetes-apiserver
  mode tcp
  balance roundrobin
  server k8s-master1 200.200.100.71:6443 check maxconn 2000
  server k8s-master2 200.200.100.72:6443 check maxconn 2000
  server k8s-master3 200.200.100.73:6443 check maxconn 2000
EOF

#在最后一行修改或者添加我们的master节点，端口默认是8443、这里更改了默认端口

· keeplived配置文件修改

cat << EOF | tee /etc/keepalived/keepalived.conf
global_defs {
  router_id LVS_DEVEL
}

vrrp_script check_haproxy {
  script "/etc/keepalived/check_haproxy.sh" 
  interval 3
  fall 10
  timeout 9
  rise 2
}
vrrp_instance VI_1 {
  state MASTER     #备服务器上改为BACKUP
  interface ens18    #改为自己的接口
  virtual_router_id 51
  priority 100     #备服务器上改为小于100的数字，90,80
  advert_int 1
  mcast_src_ip 200.200.100.71  #本机IP
  nopreempt
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  unicast_peer {
    200.200.100.72    #除本机外其余两个master的IP节点
    200.200.100.73
  }
  virtual_ipaddress {
    200.200.100.70     #虚拟vip，自己设定
  }
  track_script {
    check_haproxy
  }
}
EOF

另外keepalived需要修改

1
2
3

router_id LVS_DEVEL_1 修改为不同
state BACKUP   备用
priority 101   优先级需要修改

#unicast_peer 为master节点IP

#virtual_ipaddress 为VIP地址，自行修改

#interface 物理网卡地址

添加keeplived健康检查脚本

cat > /etc/keepalived/check_haproxy.sh <<EOF
#!/bin/bash
A=\`ps -C haproxy --no-header | wc -l\`
if [ \$A -eq 0 ];then
systemctl stop keepalived
fi
EOF
 chmod +x /etc/keepalived/check_haproxy.sh

##注意修改VIP地址

分发keeplived及haproxy文件给所有master

# 分发文件

for NODE in k8s-master1 k8s-master2 k8s-master3; do
  echo "--- $NODE ---"
  scp -r /etc/haproxy/haproxy.cfg $NODE:/etc/haproxy/
  scp -r /etc/keepalived/keepalived.conf $NODE:/etc/keepalived/
  scp -r /etc/keepalived/check_haproxy.sh  $NODE:/etc/keepalived/
done

ping下vip看看能通否,先等待大概四五秒等keepalived和haproxy起来

ping 200.200.100.70

这里的70是我们漂移IP (VIP)

如果vip没起来就是keepalived没起来就每个节点上去restart下keepalived或者确认下配置文件/etc/keepalived/keepalived.conf里网卡名和ip是否注入成功

for NODE in k8s-master1 k8s-master2 k8s-master3; do
  echo "--- $NODE ---"
  ssh $NODE 'systemctl enable --now haproxy keepalived'
  ssh $NODE  'systemctl restart haproxy keepalived'
done
systemctl status keepalived
systemctl status haproxy

部署master节点

注意本文所有操作均在master节点执行

本文主要介绍kubernetes master组件的安装

附 v1.17.3版本软件包的下载地址

1	https://dl.k8s.io/v1.17.3/kubernetes-server-linux-amd64.tar.gz

本部分将会在控制节点上部署 Kubernetes 控制服务。每个控制节点上需要部署的服务包括：Kubernetes API Server、Scheduler 以及 Controller Manager 等。

下载并安装kubernetes组件可执行文件

kubectl 文件已经下载过了可以选择不下载

wget --timestamping \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kube-apiserver" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kube-controller-manager" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kube-scheduler" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.17.3/bin/linux/amd64/kubectl"
chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

解压缩
tar -zxvf /root/kubernetes-server-1.17.3-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}



for host in k8s-master1 k8s-master2 k8s-master3; do
  echo "---$host---"
  cd /usr/local/bin/
  scp kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $host:/usr/local/bin/
done


scp  /usr/local/bin/kube{let,-proxy} k8s-node1:/usr/local/bin/

使kubectl命令可以使用table键

source /usr/share/bash-completion/bash_completion 
source <(kubectl completion bash)
cat > /var/lib/kubernetes/audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch
  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get
  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update
  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get
  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'
  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events
  # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch
  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection
  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch
  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF

拷贝配置文件到其他master节点

for host in k8s-master2 k8s-master3; do
  echo "---$host---"
  scp /var/lib/kubernetes/audit-policy.yaml  $host:/var/lib/kubernetes/audit-policy.yaml
done

生成kube-apiserver.service配置启动文件

cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
  --advertise-address=$VIP \\
  --default-not-ready-toleration-seconds=360 \\
  --default-unreachable-toleration-seconds=360 \\
  --feature-gates=DynamicAuditing=true \\
  --max-mutating-requests-inflight=2000 \\
  --max-requests-inflight=4000 \\
  --default-watch-cache-size=200 \\
  --delete-collection-workers=2 \\
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml  \\
  --etcd-cafile=/usr/local/etcd/ssl/ca.pem \\
  --etcd-certfile=/usr/local/etcd/ssl/etcd.pem \\
  --etcd-keyfile=/usr/local/etcd/ssl/etcd-key.pem \\
  --etcd-servers=https://$MASTER1_IP:2379,https://$MASTER2_IP:2379,https://$MASTER3_IP:2379 \\
  --bind-address=0.0.0.0 \\
  --secure-port=6443 \\
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\
  --insecure-port=0 \\
  --audit-dynamic-configuration \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=/var/log/audit.log \\
  --audit-policy-file=/var/lib/kubernetes/audit-policy.yaml \\
  --profiling \\
  --anonymous-auth=false \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --enable-bootstrap-token-auth \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/var/lib/kubernetes/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all=true \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --event-ttl=168h \\
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
  --kubelet-https=true \\
  --kubelet-timeout=10s \\
  --proxy-client-cert-file=/var/lib/kube-proxy/kube-proxy.pem \\
  --proxy-client-key-file=/var/lib/kube-proxy/kube-proxy-key.pem \\
  --service-cluster-ip-range=10.250.0.0/16 \\
  --service-node-port-range=30000-32767 \\
  --logtostderr=true \\
  --v=2

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

拷贝配置文件到另外节点

1 2	scp /etc/systemd/system/kube-apiserver.service k8s-master2:/etc/systemd/system/ scp /etc/systemd/system/kube-apiserver.service k8s-master3:/etc/systemd/system/

配置详解如下

配置选项	选项说明
–advertise-address	向集群成员通知 apiserver 消息的 IP 地址，这个地址必须能够被集群中其他成员访问，如果 IP 地址为空，将会使用 –bind-address，如果未指定 –bind-address，将会使用主机的默认接口地址
–default-not-ready-toleration-seconds	表示 notReady状态的容忍度秒数：默认情况下，NoExecute 被添加到尚未具有此容忍度的每个 Pod 中
–default-unreachable-toleration-seconds	表示 unreachable状态的容忍度秒数：默认情况下，NoExecute 被添加到尚未具有此容忍度的每个 Pod 中
–feature-gates=DynamicAuditing=true	用于实验性质的特性开关组,每个key=value表示
–max-mutating-requests-inflight=2000	在给定时间内进行中可变请求的最大数量，当超过该值时，服务将拒绝所有请求，0 值表示没有限制（默认值 200）
–max-requests-inflight=4000	在给定时间内进行中不可变请求的最大数量，当超过该值时，服务将拒绝所有请求，0 值表示没有限制。（默认值 400）
–default-watch-cache-size=200	默认监视缓存大小，0 表示对于没有设置默认监视大小的资源，将禁用监视缓存
–delete-collection-workers=2	用于 DeleteCollection 调用的工作者数量，这被用于加速 namespace 的清理( 默认值 1)
–encryption-provider-config	将Secret数据加密存储到etcd中的配置文件
–etcd-cafile	用于etcd 通信的 SSL CA 文件
–etcd-certfile	用于 etcd 通信的的 SSL 证书文件
–etcd-keyfile	用于 etcd 通信的 SSL 密钥文件 .
–etcd-servers	连接的 etcd 服务器列表 , 形式为（scheme://ip:port)，使用逗号分隔
–bind-address	监听 –seure-port 的 IP 地址，被关联的接口必须能够被集群其它节点和 CLI/web 客户端访问，如果为空，则将使用所有接口（0.0.0.0）
–secure-port=6443	用于监听具有认证授权功能的 HTTPS 协议的端口，默认值是6443
–tls-cert-file	包含用于 HTTPS 的默认 x509 证书的文件，（如果有 CA 证书，则附加于 server 证书之后），如果启用了 HTTPS 服务，并且没有提供 –tls-cert-file 和 –tls-private-key-file，则将为公共地址生成一个自签名的证书和密钥并保存于 /var/run/kubernetes 目录中
–tls-private-key-file	包含匹配 –tls-cert-file 的 x509 证书私钥的文件
–insecure-port=0	监听不安全端口，默认值是8080，设置为0，表示禁用不安全端口
–audit-dynamic-configuration	动态审计配置
–audit-log-maxage=15	基于文件名中的时间戳，旧审计日志文件的最长保留天数
–audit-log-maxbackup=3	旧审计日志文件的最大保留个数
–audit-log-maxsize=100	审计日志被轮转前的最大兆字节数
–audit-log-truncate-enabled	是否启用事件和batch截断功能。
–audit-log-path	如果设置，表示所有到apiserver的请求都会记录到这个文件中，‘-’表示写入标准输出
–audit-policy-file	定义审计策略配置文件的路径，需要打开 ‘AdvancedAuditing’ 特性开关，AdvancedAuditing 需要一个配置来启用审计功能
–profiling	在 web 接口 host:port/debug/pprof/ 上启用 profiling（默认值 true）
–anonymous-auth	启用到 API server 的安全端口的匿名请求，未被其他认证方法拒绝的请求被当做匿名请求，匿名请求的用户名为 system:anonymous，用户组名为 system:unauthenticated（默认值 true）
–client-ca-file	如果设置此标志，对于任何请求，如果存包含 client-ca-file 中的 authorities 签名的客户端证书，将会使用客户端证书中的 CommonName 对应的身份进行认证
–enable-bootstrap-token-auth	启用此选项以允许 ‘kube-system’ 命名空间中的 ‘bootstrap.kubernetes.io/token’ 类型密钥可以被用于 TLS 的启动认证
–requestheader-allowed-names	使用 –requestheader-username-headers 指定的，允许在头部提供用户名的客户端证书通用名称列表。如果为空，任何通过 –requestheader-client-ca-file 中 authorities 验证的客户端证书都是被允许的
–requestheader-client-ca-file	在信任请求头中以 –requestheader-username-headers 指示的用户名之前，用于验证接入请求中客户端证书的根证书捆绑
–requestheader-extra-headers-prefix=“X-Remote-Extra-”	用于检查的请求头的前缀列表，建议使用 X-Remote-Extra-
–requestheader-group-headers=X-Remote-Group	用于检查群组的请求头列表，建议使用 X-Remote-Group
–requestheader-username-headers=X-Remote-User	用于检查用户名的请求头列表，建议使用 X-Remote-User
–service-account-key-file	包含 PEM 加密的 x509 RSA 或 ECDSA 私钥或公钥的文件，用于验证 ServiceAccount 令牌，如果设置该值，–tls-private-key-file 将会被使用，指定的文件可以包含多个密钥，并且这个标志可以和不同的文件一起多次使用
–authorization-mode=Node,RBAC	在安全端口上进行权限验证的插件的顺序列表，以逗号分隔的列表，包括：AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node.（默认值 “AlwaysAllow”）
–runtime-config=api/all=true	传递给 apiserver 用于描述运行时配置的键值对集合
–enable-admission-plugins=NodeRestriction	资源限制的相关配置
–allow-privileged=true	如果为 true, 将允许特权容器
–apiserver-count=3	集群中运行的 apiserver 数量，必须为正数（默认值 1
–event-ttl=168h	事件驻留时间（默认值 1h0m0s)
–kubelet-certificate-authority	证书 authority 的文件路径
–kubelet-client-certificate	用于 TLS 的客户端证书文件路径
–kubelet-client-key	用于 TLS 的客户端证书密钥文件路径
–kubelet-https=true	为 kubelet 启用 https（默认值 true）
–kubelet-timeout=10s	kubelet 操作超时时间（默认值5秒）
–proxy-client-cert-file	当必须调用外部程序时，用于证明 aggregator 或者 kube-apiserver 的身份的客户端证书，包括代理到用户 api-server 的请求和调用 webhook 准入控制插件的请求，它期望这个证书包含一个来自于 CA 中的 –requestheader-client-ca-file 标记的签名，该 CA 在 kube-system 命名空间的 ‘extension-apiserver-authentication’ configmap 中发布，从 Kube-aggregator 收到调用的组件应该使用该 CA 进行他们部分的双向 TLS 验证
–proxy-client-key-file	当必须调用外部程序时，用于证明 aggregator 或者 kube-apiserver 的身份的客户端证书密钥。包括代理到用户 api-server 的请求和调用 webhook 准入控制插件的请求
–service-cluster-ip-range	Service网络地址分配，CIDR 表示的 IP 范围，服务的 cluster ip 将从中分配，一定不要和分配给 nodes 和 pods 的 IP 范围产生重叠
–service-node-port-range	Service使用的端口范围
–logtostderr=true	输出日志到标准错误控制台，不输出到文件
–v=2	指定输出日志的级别

生成kube-controller-manager.service 配置启动文件

kube-controller-manager（k8s控制器管理器）是一个守护进程，它通过kube-apiserver监视集群的共享状态（kube-apiserver收集或监视到的一些集群资源状态，供kube-controller-manager或其它客户端watch）, 控制器管理器并尝试将当前的状态向所定义的状态迁移（移动、靠近），它本身是有状态的，会修改集群状态信息，如果多个控制器管理器同时生效，则会有一致性问题，所以kube-controller-manager的高可用，只能是主备模式，而kubernetes集群是采用租赁锁实现leader选举，需要在启动参数中加入–leader-elect=true。

cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
  --profiling \\
  --cluster-name=kubernetes \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --kube-api-qps=1000 \\
  --kube-api-burst=2000 \\
  --leader-elect \\
  --use-service-account-credentials\\
  --concurrent-service-syncs=2 \\
  --bind-address=0.0.0.0 \\
  --secure-port=10257 \\
  --tls-cert-file=/var/lib/kubernetes/kube-controller-manager.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kube-controller-manager-key.pem \\
  --port=10252 \\
  --authentication-kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --requestheader-client-ca-file=/var/lib/kubernetes/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --authorization-kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  --experimental-cluster-signing-duration=876000h \\
  --horizontal-pod-autoscaler-sync-period=10s \\
  --concurrent-deployment-syncs=10 \\
  --concurrent-gc-syncs=30 \\
  --node-cidr-mask-size=24 \\
  --service-cluster-ip-range=10.250.0.0/16 \\
  --pod-eviction-timeout=6m \\
  --terminated-pod-gc-threshold=10000 \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --logtostderr=true \\
  --v=2

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

原第22行出问题的配置项

1	--requestheader-allowed-names="" \\

拷贝配置文件到另外节点

1 2	scp /etc/systemd/system/kube-controller-manager.service k8s-master2:/etc/systemd/system/ scp /etc/systemd/system/kube-controller-manager.service k8s-master3:/etc/systemd/system/

配置详解

配置选项	选项说明
–profiling	通过web界面启动分析接口，host:port/debug/pprof/
–cluster-name=kubernetes	集群名称，默认是kubernetes
–controllers=*,bootstrapsigner,tokencleaner	*是启用默认启用所有控制器，但bootstrapsigner,tokencleaner 这两个控制器默认是禁用的，需要人为指定启用
–kube-api-qps=1000	与kube-apiserver通信时的QPS
–kube-api-burst=2000	与kube-apiserver通信时使用
–leader-elect	高可用时启用选举功能
–use-service-account-credentials	如果为true，为每个控制器使用单独的service account
–concurrent-service-syncs=2	允许同时同步Service数量，数字越大，服务管理响应越快，同时消耗更多的CPU和网络资源；
–bind-address=0.0.0.0	监控地址
–secure-port=10257	提供HTTPS服务，默认端口为10257，如果为0，不提供https服务
–tls-cert-file	指定x509证书文件，如果启用了HTTPS服务，但是 –tls-cert-file和–tls-private-key-file 未提供，则会为公共地址生成自签名证书和密钥，并将其保存到–cert-dir指定的目录中。
–tls-private-key-file	指定与–tls-cert-file对应的私钥
–port=10252	提供HTTP服务，不认证，如果设置0，不提供HTTP服务，默认值是10252
–authentication-kubeconfig	kube-controller-kube-controller-manager也是kube-apiserver的客户端，也可以使用kubeconfig方式访问kube-apiserver，
–client-ca-file	启用客户端证书认证
–requestheader-allowed-names=“aggregator”	允许通过的客户端证书Common Name列表，可以提供 –requestheader-username-headers 中的 Header 的用户名。如果为空，则所有通过–requestheader-client-ca-file校验的都允许通过
–requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem	针对收到的请求，在信任–requestheader-username-header指定的header里包含的用户名之前，验证客户端证书的根证书
–requestheader-extra-headers-prefix=“X-Remote-Extra-”	要检查的请求头前缀列表，建议使用 X-Remote-Extra-
–requestheader-group-headers=X-Remote-Group	请求头中需要检查的组名
–requestheader-username-headers=X-Remote-User	请求头中需要检查的用户名
–authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig	指定kubeconfig配置文件路径
–cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem	指定用于集群签发的所有集群范围内证书文件（根证书文件）
–cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem	指定集群签发证书的key
–experimental-cluster-signing-duration=876000h	证书签发时间
–horizontal-pod-autoscaler-sync-period=10s	HPA控制器检查周期
–concurrent-deployment-syncs=10	允许并发同步的Deployment对象的数量，更大的数量等于更快的部署响应
–concurrent-gc-syncs=30	允许并发同步的垃圾收集器数量。默认值20
–node-cidr-mask-size=24	node节点的CIDR掩码，默认是24
–service-cluster-ip-range=10.254.0.0/16	集群Services 的CIDR范围
–pod-eviction-timeout=6m	在失败节点删除Pod的宽限期，默认是300秒
–terminated-pod-gc-threshold=10000	在Pod垃圾收集器开始删除终止Pod前，允许存在终止的Pod数量，默认是12500
–root-ca-file=/etc/kubernetes/cert/ca.pem	如果设置，该根证书权限将包含service acount的toker secret，这必须是一个有效的PEM编码CA 包
–service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem	包含用于签署service account token的PEM编码RSA或者ECDSA私钥的文件名
–kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig	指定kubeconfig配置文件
–logtostderr=true	错误日志到标准输出，而不是文件
–v=2	日志级别

1 控制器管理器管理控制器，各个控制器负责监视（watch）apiserver暴露的集群状态，并不断地尝试把当前状态向所期待的状态迁移；

配置使用kubeconfig访问kube-apiserver安全端口；
默认非安全端口10252，安全端口10257；
kube-controller-manager 3节点高可用，去竞争锁，成为leader；

生成kube-scheduler.service配置启动文件

kube-scheduler作为kubemaster核心组件运行在master节点上面，主要是watch kube-apiserver中未被调度的Pod，如果有，通过调度算法找到最适合的节点Node，然后通过kube-apiserver以对象（pod名称、Node节点名称等）的形式写入到etcd中来完成调度，kube-scheduler的高可用与kube-controller-manager一样，需要使用选举的方式产生。

for host in k8s-master1 k8s-master2 k8s-master3; do
  echo "---$host---"
  ssh $host "mkdir /etc/kubernetes/config/ -p"
done
cat <<EOF | sudo tee /etc/kubernetes/config/kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
bindTimeoutSeconds: 600
clientConnection:
  burst: 200
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
  qps: 100
enableContentionProfiling: false
enableProfiling: true
hardPodAffinitySymmetricWeight: 1
healthzBindAddress: 127.0.0.1:10251
leaderElection:
  leaderElect: true 
metricsBindAddress: $MASTER1_IP:10251
EOF
cat <<EOF | tee /etc/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
  --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  --bind-address=$MASTER1_IP \\
  --secure-port=10259 \\
  --port=10251 \\
  --tls-cert-file=/var/lib/kubernetes/kube-scheduler.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kube-scheduler-key.pem \\
  --authentication-kubeconfig=/var/lib/kubernetes/kube-scheduler.kubeconfig \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/var/lib/kubernetes/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --authorization-kubeconfig=/var/lib/kubernetes/kube-scheduler.kubeconfig \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

拷贝配置文件到另外节点并修改其配置！！！

scp /etc/kubernetes/config/kube-scheduler.yaml k8s-master2:/etc/kubernetes/config/
scp /etc/systemd/system/kube-scheduler.service k8s-master2:/etc/systemd/system/


scp /etc/kubernetes/config/kube-scheduler.yaml k8s-master3:/etc/kubernetes/config/
scp /etc/systemd/system/kube-scheduler.service k8s-master3:/etc/systemd/system/

启动参数详解

配置选项	选项说明
–config=/etc/kubernetes/kube-scheduler.yaml	配置文件的路径
–bind-address=	监控地址
–secure-port=10259	监听的安全端口，设置为0，不提供安全端口
–port=10251	监听非安全端口，设置为0，不提供非安全端口
–tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem	包含默认的 HTTPS x509 证书的文件，（CA证书（如果有）在服务器证书之后并置），如果启用了 HTTPS 服务，并且未提供 –tls-cert-file 和 –tls-private-key-file，则会为公共地址生成一个自签名证书和密钥，并将其保存到 –cert-dir 指定的目录中
–tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem	包含与 –tls-cert-file 匹配的默认 x509 私钥的文件
–authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig	指定kube-scheduler做为kube-apiserver客户端时使用kubeconfig文件
–client-ca-file=/etc/kubernetes/cert/ca.pem	如果已设置，由 client-ca-file 中的授权机构签名的客户端证书的任何请求都将使用与客户端证书的 CommonName 对应的身份进行身份验证
–requestheader-allowed-names=“aggregator”	客户端证书通用名称列表允许在 –requestheader-username-headers 指定的头部中提供用户名。如果为空，则允许任何由权威机构 –requestheader-client-ca-file 验证的客户端证书。
–requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem	在信任 –requestheader-username-headers 指定的头部中的用户名之前用于验证传入请求上的客户端证书的根证书包。警告：通常不依赖于传入请求已经完成的授权。
–requestheader-extra-headers-prefix=“X-Remote-Extra-”	要检查请求头部前缀列表。建议使用 X-Remote-Extra-
–requestheader-group-headers=X-Remote-Group	用于检查组的请求头部列表。建议使用 X-Remote-Group
–requestheader-username-headers=X-Remote-User	用于检查用户名的请求头部列表。X-Remote-User 很常见。
–authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig	指向具有足够权限以创建 subjectaccessreviews.authorization.k8s.io 的 ‘core’ kubernetes 服务器的 kubeconfig 文件，这是可选的，如果为空，则禁止所有未经授权跳过的请求
–logtostderr=true	日志记录到标准错误而不是文件
–v=2	日志级别详细程度的数字

kube-scheduler提供非安全端口10251，安全端口10259；
kube-scheduler 部署3节点高可用，通过选举产生leader；
它监视kube-apiserver提供的watch接口，它根据预选和优选策略两个环节找一个最佳适配，然后调度到此节点；

启动各组件

for host in k8s-master1 k8s-master2 k8s-master3; do
  echo "---$host---"
  ssh $host "systemctl daemon-reload"
  ssh $host "systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler"
done
for host in k8s-master1 k8s-master2 k8s-master3; do
  echo "---$host---"
  ssh $host "systemctl daemon-reload"
  ssh $host "systemctl restart kube-apiserver kube-controller-manager kube-scheduler"
done



for host in k8s-master1 k8s-master2 k8s-master3; do
  echo "---$host---"
  ssh $host "systemctl daemon-reload"
  ssh $host "systemctl stop kube-apiserver kube-controller-manager kube-scheduler"
done

请等待10秒以便 kubernetes api server初始化

1
2
3

systemctl status kube-apiserver
systemctl status kube-controller-manager
systemctl status kube-scheduler

拷贝admin 的kubeconfig 为kubectl默认读取的.kube/config

cd
for host in k8s-master1 k8s-master2 k8s-master3; do
  echo "---$host---"
  ssh $host "mkdir /root/.kube -p"
  scp /root/ssl/kubernetes/admin.kubeconfig $host:/root/.kube/config
done

查看集群信息

1	kubectl cluster-info

输出信息

1
2
3

Kubernetes master is running at https://127.0.0.1:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

查看 cs信息

1	kubectl get cs

输出信息

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}

Kubelet RBAC 授权

注意本文所有操作均在master节点执行

本节将会配置 API Server 访问 Kubelet API 的 RBAC 授权。访问 Kubelet API 是获取 metrics、日志以及执行容器命令所必需的。

所有操作均在主节点操作

这里设置 Kubeket –authorization-mode 为 Webhook 模式。Webhook 模式使用 SubjectAccessReview API 来决定授权。
创建 system:kube-apiserver-to-kubelet ClusterRole 以允许请求 Kubelet API 和执行大部分来管理 Pods 的任务:

cd /root/ssl/kubernetes
cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
EOF

输出信息

1	clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created

Kubernetes API Server 使用客户端凭证授权 Kubelet 为 kubernetes 用户，此凭证用 –kubelet-client-certificate flag 来定义。

绑定 system:kube-apiserver-to-kubelet ClusterRole 到 kubernetes 用户:

cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

输出信息

1	clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

部署node节点

注意本文所有操作所有节点执行

本部分将会部署 Kubernetes 工作节点。每个节点上将会安装以下服务：
container networking plugins
kubelet
kube-proxy

安装依赖
安装 OS 依赖组件：

for host in k8s-master1 k8s-master2 k8s-master3 k8s-node1;do
  echo "---$host---"
  ssh $host "yum install -y socat conntrack ipset";
done

socat 命令用于支持 kubectl port-forward 命令。

下载worker 二进制文件

1	wget https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz

解压cni插件

tar -zxvf cni-plugins-linux-amd64-v0.8.2.tgz -C /opt/cni/bin/

for host in k8s-master2 k8s-master3 k8s-node1; do
  echo "---$host---"
  ssh $host "mkdir /opt/cni/bin/ -p"
  scp /opt/cni/bin/* $host:/opt/cni/bin/
done

生成kubelet.service systemd服务文件

cat << EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
   --config=/var/lib/kubelet/kubelet-config.yaml \\
  --image-pull-progress-deadline=2m \\
  --kubeconfig=/var/lib/kubelet/kubeconfig \\
  --pod-infra-container-image=cargo.caicloud.io/caicloud/pause-amd64:3.1 \\
  --network-plugin=cni \\
  --register-node=true \\
  --v=2 \\
  --container-runtime=docker \\
  --container-runtime-endpoint=unix:///var/run/dockershim.sock \\
  --image-pull-progress-deadline=15m

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

拷贝启动文件到其他节点

for host in k8s-master2 k8s-master3 k8s-node1; do
  echo "---$host---"
  scp /etc/systemd/system/kubelet.service $host:/etc/systemd/system/
done

配置参数详解

配置选项	选项说明
–bootstrap-kubeconfig	指定令牌认证文件
–cert-dir	设置kube-controller-manager生成证书和私钥的目录
–cni-conf-dir=	指定cni配置文件目录
–container-runtime=docker	指定容器运行时引擎
–container-runtime-endpoint=	监听的unix socket位置（Windows上面为 tcp 端口）。
–root-dir=	kubelet 保存数据的目录，默认：/var/lib/kubelet
–kubeconfig=	kubelet作为客户端使用的kubeconfig认证文件，此文件是由kube-controller-mananger生成的
–config=	指定kubelet配置文件
–hostname-override=	用来配置该节点在集群中显示的主机名，kubelet设置了-–hostname-override参数后，kube-proxy也需要设置，否则会出现找不到Node的情况
–pod-infra-container-image=	每个pod中的network/ipc 名称空间容器将使用的镜像
–image-pull-progress-deadline=	镜像拉取进度最大时间，如果在这段时间拉取镜像没有任何进展，将取消拉取，默认：1m0s
–volume-plugin-dir=	第三方卷插件的完整搜索路径，默认：”/usr/libexec/kubernetes/kubelet-plugins/volume/exec/“
–logtostderr=true	日志记录到标准错误而不是文件
–v=2	日志级别详细程度的数字

master1节点

cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.250.0.10"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/k8s-master1.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/k8s-master1-key.pem"
address: "$MASTER1_IP"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "$MASTER1_IP"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "10.244.0.0/16"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF

master2节点

cat << EOF | sudo tee /var/lib/kubelet/k8s-master2.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.250.0.10"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/k8s-master2.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/k8s-master2-key.pem"
address: "$MASTER2_IP"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "$MASTER2_IP"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "10.244.0.0/16"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF


scp /var/lib/kubelet/k8s-master2.yaml k8s-master2:/var/lib/kubelet/kubelet-config.yaml
rm -rf /var/lib/kubelet/k8s-master2.yaml

k8s-master3 节点的配置文件在master主机上写入文件

cat << EOF | sudo tee /var/lib/kubelet/k8s-master3.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.250.0.10"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/k8s-master3.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/k8s-master3-key.pem"
address: "$MASTER3_IP"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "$MASTER3_IP"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "10.244.0.0/16"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF
scp /var/lib/kubelet/k8s-master3.yaml k8s-master3:/var/lib/kubelet/kubelet-config.yaml
rm -rf /var/lib/kubelet/k8s-master3.yaml

k8s-node1节点

cat << EOF | sudo tee /var/lib/kubelet/k8s-node1.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.250.0.10"
runtimeRequestTimeout: "15m"
address: "$NODE1_IP"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "$NODE1_IP"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "10.244.0.0/16"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
tlsCertFile: "/var/lib/kubelet/k8s-node1.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/k8s-node1-key.pem"
EOF
scp /var/lib/kubelet/k8s-node1.yaml k8s-node1:/var/lib/kubelet/kubelet-config.yaml
rm -rf /var/lib/kubelet/k8s-node1.yaml

配置详解

配置选项	选项说明
address	kubelet 服务监听的地址
staticPodPath: “”	kubelet 会定期的扫描这个文件夹下的YAML/JSON 文件来创建/删除静态Pod，使用kubeadm安装时非常有用
syncFrequency: 1m	同步运行容器和配置之间的最大时间间隔，默认为1m
fileCheckFrequency: 20s	检查新数据配置文件的周期，默认 20s
httpCheckFrequency: 20s	通过 http 检查新数据的周期，默认 20s
staticPodURL: “”
port: 10250	kubelet 服务的端口，默认 10250
readOnlyPort: 0	没有认证/授权的只读 kubelet 服务端口，设置为 0 表示禁用，默认 10255
rotateCertificates	远程认证，默认为false
serverTLSBootstrap: true	kubelet安全引导认证，重启 Kubelet，会发现出现了新的 CSR
authentication:	认证方式有以下几种
anonymous:	匿名
enabled: false	值为false
webhook:	webhook的方式
enabled: true	值为true
x509:	x509认证
clientCAFile: “/etc/kubernetes/cert/ca.pem”	集群ca证书
authorization:	授权
mode: Webhook	授权webhook
registryPullQPS	限制每秒拉取镜像个数，设置为 0 表示不限制，默认 5
registryBurst	仅当 –registry-qps 大于 0 时使用，设置拉取镜像的最大并发数，允许同时拉取的镜像数，不能超过 registry-qps ，默认 10
eventRecordQPS	限制每秒创建的事件数目，设置为0则不限制，默认为5
eventBurst	当–event-qps大于0时，临时允许该事件记录值超过设定值，但不能超过 event-qps 的值，默认10
enableDebuggingHandlers	启用用于日志收集和本地运行容器和命令的服务端端点，默认值：true
enableContentionProfiling	如果启用了 profiling，则启用性能分析锁
healthzPort	本地 healthz 端点的端口，设置为 0 将禁用，默认值：10248
healthzBindAddress	监听healthz 端口的地址
clusterDomain: “cluster.local”	集群域名, kubelet 将配置所有容器除了主机搜索域还将搜索当前域
clusterDNS:	DNS 服务器的IP地址列表
- “10.254.0.2”	指定一个DNS服务器地址
nodeStatusUpdateFrequency: 10s	node节点状态更新上报频率，默认：10s
nodeStatusReportFrequency: 1m	node节点上报自身状态频率，默认：1m
imageMinimumGCAge: 2m	设置镜像最少多久没有被使用才会被清理
imageGCHighThresholdPercent: 85	设置镜像占用磁盘比率最大值，超过此值将执行镜像垃圾回收，默认 85
imageGCLowThresholdPercent: 80	设置镜像占用磁盘比率最小值，低于此值将停止镜像垃圾回收，默认 80
volumeStatsAggPeriod: 1m	指定kubelet计算和缓存所有容器组及卷的磁盘使用量时间间隔，设置为0禁用卷计算，默认：1m
kubeletCgroups: “”	可选的 cgroups 的绝对名称来创建和运行 kubelet
systemCgroups: “”	可选的 cgroups 的绝对名称，用于将未包含在 cgroup 内的所有非内核进程放置在根目录 / 中，修改这个参数需要重启
cgroupRoot: “”	Pods 可选的root cgroup, 这是由container runtime，在最佳工作的基础上处理的，默认值: ‘’，意思是使用container runtime的默认处理
cgroupsPerQOS: true	支持创建QoS cgroup的层级结构，如果是true，最高层级的
cgroupDriver: systemd	Kubelet用来操作主机cgroups的驱动
runtimeRequestTimeout: 10m	除了长时间运行的请求(pull,logs,exec，attach)以外，所有runtime请求的超时时间，当触发超时时，kubelet会取消请求，抛出一个错误并稍后重试，默认值:2m0s
hairpinMode: promiscuous-bridge	Kubelet该怎么设置hairpin promiscuous-bridge.这个参数允许Service的端点试图访问自己的Service。如果网络没有正确配置为 “hairpin” 流量，通常当 kube-proxy 以 iptables 模式运行，并且 Pod 与桥接网络连接时，就会发生这种情况。Kubelet 公开了一个 hairpin-mode 标志，如果 pod 试图访问他们自己的 Service VIP，就可以让 Service 的 endpoints 重新负载到他们自己身上。hairpin-mode 标志必须设置为 hairpin-veth 或者 promiscuous-bridge。
maxPods	当前 kubelet 可以运行的容器组数目，默认：110
podCIDR	pod使用的CIDR网段
podPidsLimit: -1	pod中设置PID限制
resolvConf	用作容器DNS解析配置的解析器配置文件，默认： “/etc/resolv.conf”
maxOpenFiles: 1000000	kubelet 进程可以打开的文件数目，默认：1000000
kubeAPIQPS	与kube-apiserver会话时的QPS，默认：15
kubeAPIBurst	与kube-apiserver会话时的并发数，默认：10
serializeImagePulls: false	禁止一次只拉取一个镜像
evictionHard:	一个清理阈值的集合，达到该阈值将触发一次容器清理
memory.available: “100Mi”	小gf 100Mi时清理
nodefs.available: “10%”
nodefs.inodesFree: “5%”
imagefs.available: “15%”
evictionSoft: {}	清理阈值的集合，如果达到一个清理周期将触发一次容器清理
enableControllerAttachDetach: true	允许附加/分离控制器来管理调度到当前节点的附加/分离卷，并禁用kubelet执行任何附加/分离操作，默认：true
failSwapOn: true	如果在节点上启用了swap，kubelet将启动失败
containerLogMaxSize: 20Mi	容器日志容量最大值
containerLogMaxFiles: 10	容器日志文件数最大值
systemReserved: {}	https://kubernetes.io/zh/docs/tasks/administer-cluster/reserve-compute-resources/，系统守护进程争取资源预留，这里均设置为默认值
kubeReserved: {}	kubernetes 系统守护进程争取资源预留，这里均设置为默认值
systemReservedCgroup: “”	要选择性的在系统守护进程上执行 kube-reserved，需要把 kubelet 的 –kube-reserved-cgroup 标志的值设置为 kube 守护进程的父控制组要想在系统守护进程上可选地执行 system-reserved，请指定 –system-reserved-cgroup kubelet 标志的值为 OS 系统守护进程的父级控制组，这里均设置为默认值
kubeReservedCgroup: “”	要选择性的在系统守护进程上执行 kube-reserved，需要把 kubelet 的 –kube-reserved-cgroup 标志的值设置为 kube 守护进程的父控制组这里均设置为默认值
enforceNodeAllocatable: [“pods”]	无论何时，如果所有 pod 的总用量超过了 Allocatable，驱逐 pod 的措施将被执行

kubelet组件采用主动的查询机制，定期向kube-apiserver获取当前节点应该处理的任务，如果有任务分配到了自己身上（如创建Pod），从而他去处理这些任务；

kubelet暴露了两个端口10248，http形式的healthz服务，另一个是10250，https服务，其实还有一个只读的10255端口，这里是禁用的。

生成kube-proxy 服务启动文件

本文是二进制安装kubernetes v1.17.0 之kube-proxy，kube-proxy是什么，这里就不得不说下service，service是一组Pod的抽象集合，它相当于一组Pod的负载均衡器，负责将请求分发到对应的pod，kube-proxy就是负责service的实现的，当请求到达service时，它通过label关联到后端并转发到某个Pod；kube-proxy提供了三种负载均衡模式：用户空间、iptables、ipvs，网上有很多关于这三种模式的区别，这里先不详述，本文采用ipvs。

kube-proxy需要运行在所有节点上（因为我们master节点也有Pod，如果没有的话，可以只部署在非master节点上），kube-proxy它主动的去监听kube-apiserver中service和endpoint的变化情况，然后根据定义的模式，创建路由规则，并提供服务service IP（headless类型的service无IP）和负载均衡功能。注意：在所有节点安装ipvsadm和ipset命令，加载ip_vs内核模块，准备章节已经执行过。

master节点上操作

cat <<EOF | sudo tee /var/lib/kube-proxy/kube-proxy-config.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  burst: 200
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
  qps: 100
bindAddress: $VIP
healthzBindAddress: $VIP:10256
metricsBindAddress: $VIP:10249
enableProfiling: true
clusterCIDR: 10.244.0.0/16
mode: "ipvs"
portRange: ""
kubeProxyIPTablesConfiguration:
  masqueradeAll: false
kubeProxyIPVSConfiguration:
  scheduler: rr
  excludeCIDRs: []
EOF


for host in k8s-master2 k8s-master3;do
  echo "---$host---"
  scp /var/lib/kube-proxy/kube-proxy-config.yaml $host:/var/lib/kube-proxy/
done


node配置

cat << EOF | sudo tee k8s-node1.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  burst: 200
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
  qps: 100
bindAddress: $NODE1_IP
healthzBindAddress: $NODE1_IP:10256
metricsBindAddress: $NODE1_IP:10249
enableProfiling: true
clusterCIDR: 10.244.0.0/16
mode: "ipvs"
portRange: ""
kubeProxyIPTablesConfiguration:
  masqueradeAll: false
kubeProxyIPVSConfiguration:
  scheduler: rr
  excludeCIDRs: []
EOF


scp k8s-node1.yaml k8s-node1:/var/lib/kube-proxy/kube-proxy-config.yaml
rm -rf k8s-node1.yaml

配置详解

配置选项	选项说明
clientConnection	与kube-apiserver交互时的参数设置
burst: 200	临时允许该事件记录值超过qps设定值
kubeconfig	kube-proxy 客户端连接 kube-apiserver 的 kubeconfig 文件路径设置
qps: 100	与kube-apiserver交互时的QPS，默认值5
bindAddress	kube-proxy监听地址
healthzBindAddress	用于检查服务的IP地址和端口，默认：0.0.0.0:10256
metricsBindAddress	metrics服务的ip地址和端口，默认：127.0.0.1:10249
enableProfiling	如果设为true，则通过/debug/pprof处理程序上的web界面进行概要分析
clusterCIDR	kube-proxy 根据 –cluster-cidr 判断集群内部和外部流量，指定 –cluster-cidr 或 –masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT
hostnameOverride	参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
mode	使用 ipvs 模式
portRange	主机端口的范围(beginPort- endport，单端口或beginPort+偏移量)，可用于代理服务流量。如果(未指定，0，或0-0)端口将被随机选择
kubeProxyIPTablesConfiguration:
masqueradeAll: false	如果使用纯iptables代理，SNAT所有通过服务集群ip发送的通信
kubeProxyIPVSConfiguration:
scheduler: rr	当proxy为ipvs模式时，ipvs调度类型
excludeCIDRs: []

cat <<EOF | sudo tee /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

拷贝启动配置文件到其他节点

for host in k8s-master2 k8s-master3 k8s-node1; do
  echo "---$host---"
  scp /etc/systemd/system/kube-proxy.service $host:/etc/systemd/system/
done

启动worker服务

for host in k8s-master1 k8s-master2 k8s-master3 k8s-node1; do
  echo "---$host---"
  ssh $host "systemctl daemon-reload"
  ssh $host "systemctl enable --now kubelet kube-proxy"
done
systemctl status kubelet
systemctl status kube-proxy

此时所有节点的kubelet启动之后会自动加入集群

查看集群节点

1	kubectl get nodes

输出信息

NAME     STATUS     ROLES    AGE   VERSION
k8s-master1   NotReady   <none>   9s    v1.17.3
k8s-master2   NotReady   <none>   9s    v1.17.3
k8s-master3   NotReady   <none>   39m   v1.17.3
k8s-node1     NotReady   <none>   39m   v1.17.3

可以看到所有的节点都已经成功加入集群

状态显示 NotReady 是因为没有配置网络插件配置好网络插件就 Ready了

部署网络配置

注意本文所有操作均在master节点执行

本文主要介绍部署网络 dns 插件

本文的网络插件选用的是calico 以容器的方式运行
本文的dns’插件选用的是coredns以容器的方式运行

安装calico网络

下载calico网络插件的yaml文件

1.16版本的kubernetes calico网络插件应选用3.9及以上的版本

本文选用3.10

1	curl https://docs.projectcalico.org/v3.10/manifests/calico.yaml -O

修改网络插件yaml文件内容

1	sed -i 's\|192.168.0.0\|10.244.0.0\|' calico.yaml

应用yaml文件

1	kubectl apply -f calico.yaml

查看pod状态

1	kubectl get pods -n kube-system

输出信息

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6d85fdfbd8-8v2hs   1/1     Running   0          7m40s
calico-node-984fd                          1/1     Running   0          7m40s
calico-node-n5kn8                          1/1     Running   0          7m40s
calico-node-q4p7c                          1/1     Running   0          7m40s

网络插件安装成功

安装coredns插件

先下载jq命令(coredns提供的部署脚本需要使用jq命令对生成的yaml文件进行整理)

1 2	wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 -O /usr/bin/jq chmod a+x /usr/bin/jq

克隆GitHub仓库

1 2	yum -y install git git clone https://github.com/coredns/deployment.git

执行

1 2	cd deployment/kubernetes/ ./deploy.sh -i 10.250.0.10 \| kubectl apply -f -

ip为开头写的集群dns的IP

输出信息

serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created

查看pod

1 2	kubectl scale -n kube-system deployment coredns --replicas=3 kubectl get pods -n kube-system

输出信息

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6d85fdfbd8-8v2hs   1/1     Running   0          91m
calico-node-984fd                          1/1     Running   0          91m
calico-node-n5kn8                          1/1     Running   0          91m
calico-node-q4p7c                          1/1     Running   0          91m
coredns-68567cdb47-6j7bb                   1/1     Running   0          5m31s
coredns-68567cdb47-7nwcg                   1/1     Running   0          5m31s

可以看到calico网络和coredns插件都好了

验证

进入pod内部 pingbaidu.com

如果能ping同则插件部署完毕

安装Ingress-nginx插件

yaml文件

点击上方超链接将yaml文件的所有内容复制下来保存到主节点的nginx-ingress.yaml文件内

然后

1	kubectl apply -f nginx-ds.yaml

输出信息

namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
daemonset.apps/nginx-ingress-controller created

然后查看Pod

1	kubectl get pods -n ingress-nginx -o wide

输出信息全部为Running即可

NAME                             READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
nginx-ingress-controller-8xzrd   1/1     Running   0          2m13s   200.200.100.73   k8s-node    <none>           <none>
nginx-ingress-controller-c84dv   1/1     Running   0          2m13s   200.200.100.74   k8s-node1    <none>           <none>
nginx-ingress-controller-qlpn5   1/1     Running   0          2m13s   200.200.100.71   master   <none>           <none>

Ingress-nginx插件安装完毕

注意 本文所有操作均在所有节点执行

环境准备

环境变量

SSH免密/时间同步/主机名修改

升级内核

调整系统内核参数

关闭防火墙、selinux、swap、NetworkManager

修改资源限制

常用软件安装

加载内核ipvs模块

安装docker-ce：

然后重启系统 验证内核是否升级成功

下载命令

下载cfssl以及cfssljson命令

安装CFSSL

生成证书

注意 本文所有操作均在master1节点执行

生成etcd证书

生成kubernetes组件证书

新建CA配置文件

client与server凭证

生成kubelet客户端凭证

创建kube-controller-manager客户端凭证

创建kube-proxy客户端凭证

创建kube-scheduler凭证签发

创建kubernetes 证书

配置和生成kubernetes配置文件

注意 本文所有操作均在master节点执行

kubelet配置文件

kube-proxy配置文件

kube-controller-manager配置文件

kube-scheduler配置文件

Admin配置文件

配置和生成密钥

分发证书文件

部署etcd集群

下载etcd二进制包并解压

设置etcd系统服务文件

所有节点启动etcd服务并设置为开机自启

检查etcd集群状态

部署keepalived+HAProxy

部署master节点

注意 本文所有操作均在master节点执行

下载并安装kubernetes组件可执行文件

生成kube-apiserver.service配置启动文件

生成kube-controller-manager.service 配置启动文件

生成kube-scheduler.service配置启动文件

启动各组件

拷贝admin 的kubeconfig 为kubectl默认读取的.kube/config

Kubelet RBAC 授权

注意 本文所有操作均在master节点执行

部署node节点

注意 本文所有操作所有节点执行

生成kube-proxy 服务启动文件

部署网络配置

注意 本文所有操作均在master节点执行

安装calico网络

安装coredns插件

验证

安装Ingress-nginx插件

集群部署完毕

注意本文所有操作均在所有节点执行

然后重启系统验证内核是否升级成功

注意本文所有操作均在master1节点执行

注意本文所有操作均在master节点执行

注意本文所有操作均在master节点执行

注意本文所有操作均在master节点执行

注意本文所有操作所有节点执行

注意本文所有操作均在master节点执行