[kubernetes] mac m1 kubernetes 구축

1. vm 생성 (UTM 사용)

• m1의 경우 vmware, virturebox등을 m1에서 사용하기 힘들기 때문에 UTM이라는 가상 머신을 사용하였다.
• ubuntu-server만 깔았을 경우 왜인진 모르겠지만 clipboard copy가 되지 않아 ubuntu-desktop도 같이 깔아주니 clipboard copy가 되었다!
• master 3대와 worker1대로 구성하였다. (이유는 없고 master를 여러대 띄워보고 싶어서)

2. hostname 설정

각 노드마다 hostname을 다르게 설정해주어야 한다.

# vi /etc/hosts

192.168.100.10          master1
192.168.100.11          master2
192.168.100.12          master3
192.168.100.13          worker1

# hostname 변경

# master1
hostnamectl set-hostname master1

# master2
hostnamectl set-hostname master2

# master3
hostnamectl set-hostname master3

# worker1
hostnamectl set-hostname worker1

3. 방화벽 해제

• NTP 등의 서버 때문에도 그렇고 혹시 모를 경우를 대비해 방화벽을 해제하고 진행하였다.

$ sudo ufw disable

4. NTP 서버 동기화

• 클러스터를 구축하게되면 각 노드들이 네트워크 통신을 하게 되는데, 이때 각각의 시간이 맞지 않아 발생할 수 있는 위험을 없애기 위해 NTP 서버 동기화를 진행한다.
• 실제로 NTP서버를 동기화 시키지 않아서 많은 오류를 겪었다.

$ sudo apt install ntp

$ vim /etc/ntp.conf

pool 로 시작하는 라인 주석처리 후, 공식 ntp 사이트에 있는 한국 pool을 넣어준다.

https://www.ntppool.org/zone/kr

pool.ntp.org: NTP Servers in Korea, kr.pool.ntp.org

 

5. 스왑 메모리 해제

• 스왑 메모리란? 시스템에 메모리가 부족할 경우에 하드디스크 일부 공간을 활용하여 계속 작업을 도와주는 영역 

$ sudo swapoff -a

$ vim /etc/fstab 에서 swap에 대한 설정 주석

6. 컨테이너 런타임 설치

• 런타임으로는 containerd를 설치하였다.
• docker 런타임의 경우 kubernetes 특정 버전 이후부터 지원하지 않는다고 한다.
• 런타임 설치는 아래 공식문서들을 보며 설치하였다.

https://kubernetes.io/ko/docs/setup/production-environment/container-runtimes/#containerd

컨테이너 런타임

https://docs.docker.com/engine/install/ubuntu/

Install Docker Engine on Ubuntu

https://github.com/containerd/containerd/blob/main/docs/getting-started.md

GitHub - containerd/containerd: An open and reliable container runtime

7. 각 노드에 kubeadm, kubelet, kubectl 설치

https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#kubeadm-kubelet-%EB%B0%8F-kubectl-%EC%84%A4%EC%B9%98

kubeadm 설치하기

 

8. HAproxy 로드밸런서 설치 (주의 : 이 작업은 Master 1 에서만 작업할 것!! )

$ sudo apt install haproxy

[LB 설정]

cat <<EOF >> /etc/haproxy/haproxy.cfg
frontend kubernetes-master-lb
bind 0.0.0.0:26443
option tcplog
mode tcp
default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
option tcplog
server master1 192.168.100.10:6443 check
server master2 192.168.100.11:6443 check
server master3 192.168.100.12:6443 check
EOF

systemctl restart haproxy && systemctl enable haproxy

systemctl status haproxy

 

[Check]

netstat -nltp | grep 26443
tcp        0      0 0.0.0.0:26443           0.0.0.0:*               LISTEN      2897/haproxy

 

9. 클러스터링

[Cluster 생성 - Master 1에서 작업]

kubeadm init --control-plane-endpoint "192.168.100.10:26443" \
--upload-certs \
--pod-network-cidr "10.244.0.0/16"

..
..
..

## 아래 출력물은 특정한 곳에 복붙 해놓을 것

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.100.10:26443 --token t69wl1.6iq2xfgp8thiz9bb \
    --discovery-token-ca-cert-hash sha256:b94dc94f3a249c237f39808a6ff2d1f68cbe058b9d4aef6d54173b5523976238 \
    --control-plane --certificate-key 91295ae74b07d78d46b4bdc29e62615d50b5644a97ff6f4453c39a574c1f9bb6

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.10:26443 --token t69wl1.6iq2xfgp8thiz9bb \
    --discovery-token-ca-cert-hash sha256:b94dc94f3a249c237f39808a6ff2d1f68cbe058b9d4aef6d54173b5523976238

• 클러스터 생성 시 --upload-certs, --control-plane-endpoint Flag를 추가해야 Certificate가 자동 배포되고 Master Node에 Join 명령어가 출력된다.

 

[kubectl 명령어 사용]

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

[Cluster 연결 - Master 2,3에서 작업]

kubeadm join 192.168.100.10:26443 --token t69wl1.6iq2xfgp8thiz9bb \
--discovery-token-ca-cert-hash sha256:b94dc94f3a249c237f39808a6ff2d1f68cbe058b9d4aef6d54173b5523976238 \
--control-plane --certificate-key 91295ae74b07d78d46b4bdc29e62615d50b5644a97ff6f4453c39a574c1f9bb6

# 중간에 아래와 같은 경고문이 나온다. 6443과 10250 Port가 Open 되어있어야 한다는 뜻
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly

 

[kubectl 명령어 사용]

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

[Cluster 연결 - Worker 에서 작업]

kubeadm join 192.168.100.10:26443 --token t69wl1.6iq2xfgp8thiz9bb \
    --discovery-token-ca-cert-hash sha256:b94dc94f3a249c237f39808a6ff2d1f68cbe058b9d4aef6d54173b5523976238
    
[preflight] Running pre-flight checks
	[WARNING SystemVerification]: missing optional cgroups: blkio
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

 

[Test]

[root@master1 ~]# kubectl get nodes
NAME      STATUS     ROLES    AGE     VERSION
master1   NotReady   master   8m27s   v1.19.2
master2   NotReady   master   60s     v1.19.2
master3   NotReady   master   55s     v1.19.2
worker1   NotReady   master   55s     v1.19.2

[root@master2 ~]# kubectl get nodes
NAME      STATUS     ROLES    AGE     VERSION
master1   NotReady   master   8m27s   v1.19.2
master2   NotReady   master   60s     v1.19.2
master3   NotReady   master   55s     v1.19.2
worker1   NotReady   master   55s     v1.19.2

[root@master3 ~]# kubectl get nodes
NAME      STATUS     ROLES    AGE     VERSION
master1   NotReady   master   8m27s   v1.19.2
master2   NotReady   master   60s     v1.19.2
master3   NotReady   master   55s     v1.19.2
worker1   NotReady   master   55s     v1.19.2

⇒ NotReady는 CNI가 설치되어 있지 않아서이다. CNI 설치를 해주자.

10. CNI (가상 Network) 적용

• Calico, Canal, Clilum, Flannel, Weave등 다양항 CNI가 있다.
• weave를 적용시킬 예정

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

• CNI 적용 확인
• kube-system 네임스페이스에서 Pod를 확인하면 생성중임을 확인할 수 있다.

[root@master1 ~]# kubectl get pods -n kube-system
NAME                              READY   STATUS              RESTARTS   AGE
coredns-f9fd979d6-2rh9s           0/1     Pending             0          10m
coredns-f9fd979d6-96wjl           0/1     Pending             0          10m
etcd-master1                      1/1     Running             0          10m
etcd-master2                      1/1     Running             0          3m23s
etcd-master3                      1/1     Running             0          3m8s
kube-apiserver-master1            1/1     Running             0          10m
kube-apiserver-master2            1/1     Running             0          3m25s
kube-apiserver-master3            1/1     Running             0          116s
kube-controller-manager-master1   1/1     Running             1          10m
kube-controller-manager-master2   1/1     Running             0          3m25s
kube-controller-manager-master3   1/1     Running             0          109s
kube-proxy-d27sv                  1/1     Running             0          2m42s
kube-proxy-n8hp7                  1/1     Running             0          10m
kube-proxy-slbjt                  1/1     Running             0          3m28s
kube-scheduler-master1            1/1     Running             1          10m
kube-scheduler-master2            1/1     Running             0          3m27s
kube-scheduler-master3            1/1     Running             0          115s
weave-net-5w7cw                   1/2     Running             0          34s
weave-net-6l92n                   0/2     ContainerCreating   0          34s
weave-net-psnfk                   1/2     Running             0          34s

• kubectl get nodes

설치 완료

[root@master1 ~]# kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
master1   Ready    master   11m     v1.19.2
master2   Ready    master   4m10s   v1.19.2
master3   Ready    master   4m5s    v1.19.2

!! 참고 블로그 !!

https://velog.io/@pingping95/Kubernetes-kubeadm-Master-%EA%B3%A0%EA%B0%80%EC%9A%A9%EC%84%B1

[Kubernetes] kubeadm Master 3개 설정 (Haproxy)

https://velog.io/@miintto/k8s-installation

[쿠버네티스] 무작정 설치해보기

https://seulcode.tistory.com/570

리눅스 서버 스왑 메모리 끄기 (turn off swap memory)