Kubernetes集群搭建 | Shawn's Blog

使用kubeadm工具快速安装Kubernetes集群

最简单的方法是yum install kubernetes，但需要修改大量各组件的启动参数，过程还是太复杂。

这里介绍使用kubeadm工具快速安装Kubernetes集群

0x01：安装kubeadm和相关工具

1	$ vim /etc/yum.repos.d/kubernetes.repo

添加内容：

[kubernetes]
name=Kubernetes Repository
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0

然后安装kubeadm和相关工具

1	$ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

启动Docker和kubelet服务，并设置开机启动（如果之前Docker已安装，则无需再次启动）

1 2	$ systemctl enable docker && systemctl start docker $ systemctl enable kubelet && systemctl start kubelet

0x02：kubeadm config

1	$ vim init-config.yaml

添加内容：

apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
networking:
  dnsDomain: cluster.local
  podSubnet: 10.0.0.0/16
  serviceSubnet: 192.168.100.0/24
scheduler: {}

其中networking项改成你自己的，注意不要和宿主机已使用的网段重合，包括子网在内，不然可能起冲突

0x03：下载K8S相关镜像

1	$ kubeadm config images pull --config=init-config.yaml

0x04：运行kubeadm init命令安装Master

1	$ kubeadm init --config=init-config.yaml

成功的话会像我这样显示，有这句话就对了Your Kubernetes control-plane has initialized successfully!：

[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.3. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [localhost kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [192.168.96.1 192.168.96.3]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost localhost] and IPs [192.168.96.3 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost localhost] and IPs [192.168.96.3 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.504678 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node localhost as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.96.3:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:8c8f82e6ab26d08dffa9bf450db44227758d1511fcc044f875bf4af5179e53e8

然后按照最后的提示执行下面命令：

1
2
3

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

还有最后的这一行要记下，包含了加入节点的指令和所需的Token

1 2	kubeadm join 192.168.96.3:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:8c8f82e6ab26d08dffa9bf450db44227758d1511fcc044f875bf4af5179e53e8

0x05：安装Node，加入集群

这里有两种形式的Node，一种是用另一台机子join进集群。还有就是所谓的单机部署，一台既是Master又是Node，自己试验的话这种也可以。

先说第一种集群部署：

（1）安装kubeadm相关环境，和之前安装Master的一样，可以先配置YUM源这里就不写了

1
2
3

$ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet

（2）创建配置文件

1	$ vim join-config.yaml

添加内容：

apiVersion: kubeadm.k8s.io/v1beta2
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: 192.168.96.3:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration

token和tlsBootstrapToken替换成之前init命令成功后输出的token，apiServerEndpoint填Master的地址

（3）加入集群

1	$ kubeadm join --config=join-config.yaml

再说下单机部署的：

1	$ kubectl taint nodes --all node-role.kubernetes.io/master-

这样就让Master成为了Node

0x06：安装网络插件

现在执行kubectl get nodes，会发现都是NotReady，因为没安装CNI网络插件

1 2	NAME STATUS ROLES AGE VERSION localhost NotReady master 20m v1.16.3

1	$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version \| base64 \| tr -d '\n')"

执行完后等个10几秒视情况而定，然后再次执行kubectl get nodes

1 2	NAME STATUS ROLES AGE VERSION localhost Ready master 32m v1.16.3

0x07：验证K8S集群是否安装完成

执行以下命令，看是否都是正常Running

1	$ kubectl get pods --all-namespaces

如果遇到类似下面的错误导致coredns容器一直起不来的话，可以刷新iptables试试（出处）

1	Failed to list *v1.Endpoints: Get https://192.168.100.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 192.168.100.1:443: connect: no route to host

systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

The route problem can be solved by flush iptables.

NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system   coredns-58cc8c89f4-4cdqf            1/1     Running   8          51m
kube-system   coredns-58cc8c89f4-8vzpz            1/1     Running   8          51m
kube-system   etcd-localhost                      1/1     Running   1          50m
kube-system   kube-apiserver-localhost            1/1     Running   1          50m
kube-system   kube-controller-manager-localhost   1/1     Running   1          50m
kube-system   kube-proxy-x9s65                    1/1     Running   1          51m
kube-system   kube-scheduler-localhost            1/1     Running   1          50m
kube-system   weave-net-wjlvv                     2/2     Running   3          19m

如果coredns还是一直重启的话，到这里自查一下网络问题（ https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ ），怀疑是coredns容器BUG？因为自己当时也会CrashLoopBackOff，看log也只是显示与k8s的api无法通信，但是DNS配置、网络配置都没问题，也没法解决，但是一段时间后又自己起来了，太玄学了。

所以实在查不出问题的，但是dns就是起不来的，不妨试试 docker rm -f $(sudo docker ps -a -q) ，然后等k8s自己重建一次所有pod，不行再试，说出来不信，我就是这样弄好的，重建了2次终于起来了。

补充：时隔一天，居然在机缘巧合下找到当时的症结所在了，我是本地用VM虚拟机起的k8s服务，但是每次用完挂起虚拟机，等下次再开启虚拟机的时候，coredns就会起不来，用上面的docker rm -f $(sudo docker ps -a -q)重建一下才ok，到底什么原因呢？Linux有个NetworkManager服务，会莫名其妙的删除静态路由表项，估计k8s自建的一些静态路由被它给删了，导致内部集群不能通信了，所以dns服务自然也就起不来了，况且network和NetworkManager本身就会有冲突，所以我们直接把他关闭！再重启网络，之后就再也没发生过这种情况了。

1
2
3

$ systemctl stop NetworkManager
$ systemctl disable NetworkManager
$ systemctl restart network