kubernetes에서 metrics-server와 prometheus는 kubernetes의 어딘가에서 발생하는 메트릭을 수집해서 보여준다.
pod의 CPU 사용량, Memory 사용량 등과 같은 메트릭이 어딘가로 부터 제공해주고, 이를 수집하는 것이다.
이에 대한 정확한 메커니즘을 확인한다.
kubelet의 cAdvisor 기능
cAdvisor는 여러가지 리소스를 위해서 Google에서 개발된 모듈이다.
https://github.com/google/cadvisor
해당 모듈이 kubelet에 포함되어 동작한다.
https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/cadvisor
일반적으로 node에는 kubelet에 설치되며, 내부에 cAdvisor가 내부 리소스를 보니터링 해준다.
kubelet cAdvisor metrics에 접근
kubespary로 설치한 경우, 아래와 같이 kubelet은 10250 port로 접근이 가능하게 구성된다.
kubespray로 kubernetes 설치하기 - https://blog.encicle.com/kubespraylo-kubernetes-seolchihagi
# netstat -lnp|grep kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 5883/kubelet
tcp6 0 0 :::10250 :::* LISTEN 5883/kubelet
unix 2 [ ACC ] STREAM LISTENING 52352 5883/kubelet /var/lib/kubelet/device-plugins/kubelet.sock
unix 2 [ ACC ] STREAM LISTENING 52323 5883/kubelet /var/lib/kubelet/pod-resources/1079813466
해당 포트로 접근하기 위해서는 kubernetes 인증서가 필요하거나, ServiceAccount Token이 필요하다.
kubernetes 인증서를 통한 접근
control plane 서버에 접근해서 아래와 같이 확인해보면 apiserver-kubelet-client.crt, apiserver-kubelet-client.key 2개의 파일이 존재한다.
(node에는 인증서가 존재하지 않는다, 필요하면 복사해서 사용한다.)
# ls /etc/kubernetes/pki/
apiserver-kubelet-client.crt apiserver.crt ca.crt front-proxy-ca.crt front-proxy-client.crt sa.key
apiserver-kubelet-client.key apiserver.key ca.key front-proxy-ca.key front-proxy-client.key sa.pub
kubelet에 접근하여 metrics 확인하기
# curl https://localhost:10250/metrics -k \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key
....
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-08"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-06"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-05"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.001"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.01"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.1"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="10"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="+Inf"} 1
workqueue_work_duration_seconds_sum{name="DynamicCABundle-client-ca-bundle"} 4.5311e-05
ServiceAccount Token을 통한 접근
Kubernetes Dashboard를 설치했다면 ServiceAccount Token이 생성되어 있는 상태이다.
kubernetes dashboard 설치 - https://blog.encicle.com/kubernetes-dashboard-seolchi
생성된 토큰을 통해 아래와 같이 호출 가능하다.
# export TOKEN="eyJhbGciOiJSUzI1NiIsImtpZCI6IjRyQTBGSWQ1dUdSY2N5NWJWQXJHbXdDaV9iMkdsSHlXQlF1Ujk3alE3SEUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiYWRtaW4tdXNlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImU4ZDgwOTM4LWQyYTUtNDlhOC05YTNmLTQ1MzY4MWEzNzQxZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmFkbWluLXVzZXIifQ.uvSTHphjzShKgL-vL4JXNMXUOH6yF91QxzyNqQkD6qWhmLAmk6R3Cnkvyd5iVDN6ZJtJ_-D6q3QkLtBjcKRpuRKdMVw-CuUSwSHOVyT9eZBEdLavmx98BKVQBUNTGLBQZNRr14-OdwY3Z59Tpu2_cKWzBoc896IQw16gz4t3MoVwSxYLzNHhxtKSbfKj9XiENVzErIpD9pLcopy02q0ejfNqQAauhko8J6p6bKAlyfatC3dQsfETazDZhgxpRFq_Syj1mdxj-iTU_2RZUnZfPe2h4E4jYO60W0egx7ko_z6_xo6r1bNSGs3xPmm5rQxa740b6C-9E1FVXC-Rzmv_vA"
# curl -k -H "Authorization: Bearer $TOKEN" https://localhost:10250/metrics
...
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-08"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-06"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-05"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.001"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.01"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.1"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="10"} 1
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="+Inf"} 1
workqueue_work_duration_seconds_sum{name="DynamicCABundle-client-ca-bundle"} 3.2802e-05
workqueue_work_duration_seconds_count{name="DynamicCABundle-client-ca-bundle"} 1
kubelet cAdvisor metrics에 API를 통한 접근
아래와 같이 kubernetes svc를 통해서 접근이 가능하다.
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 5d1h
...
kubernetes pod 내부에서 ServiceAccount Token으로 아래와 같은 URL 형식으로 접근이 가능하다.
node 정보
# kubectl get node
NAME STATUS ROLES AGE VERSION
control-plane Ready control-plane 5d1h v1.29.10
node1 Ready <none> 5d1h v1.29.10
node2 Ready <none> 5d1h v1.29.10
node3 Ready <none> 5d1h v1.29.10
node4 Ready <none> 5d1h v1.29.10node1의 메트릭 정보 접근
https://kubernetes.default.svc/api/v1/nodes/node1/proxy/metrics/cadvisor
echo-server pod를 설치해봤다면, 아래와 같이 접근해서 curl을 설치하고 kubernetes api를 통해서 node 메트릭 정보를 확인한다.
# kubectl get pod |grep echo-server
echo-server-dpm-7cd9c8dc7-c95x8 1/1 Running 0 2d23h
echo-server-dpm-7cd9c8dc7-cfrnz 1/1 Running 0 2d23hecho-server 연결 및 curl 설치
~# kubectl exec -ti echo-server-dpm-7cd9c8dc7-cfrnz -- /bin/sh
/app # apk update && apk add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.4-141-g23d1aafc30d [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.4-143-g2bc5d095003 [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 23040 distinct packages available
(1/8) Installing ca-certificates (20240226-r0)
(2/8) Installing brotli-libs (1.1.0-r1)
(3/8) Installing c-ares (1.27.0-r0)
(4/8) Installing libunistring (1.1-r2)
(5/8) Installing libidn2 (2.3.4-r4)
(6/8) Installing nghttp2-libs (1.58.0-r0)
(7/8) Installing libcurl (8.9.1-r1)
(8/8) Installing curl (8.9.1-r1)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20240226-r0.trigger
OK: 15 MiB in 25 packagesServiceAccount Token을 통한 api 요청
# export TOKEN="eyJhbGciOiJSUzI1NiIsImtpZCI6IjRyQTBGSWQ1dUdSY2N5NWJWQXJHbXdDaV9iMkdsSHlXQlF1Ujk3alE3SEUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiYWRtaW4tdXNlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImU4ZDgwOTM4LWQyYTUtNDlhOC05YTNmLTQ1MzY4MWEzNzQxZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmFkbWluLXVzZXIifQ.uvSTHphjzShKgL-vL4JXNMXUOH6yF91QxzyNqQkD6qWhmLAmk6R3Cnkvyd5iVDN6ZJtJ_-D6q3QkLtBjcKRpuRKdMVw-CuUSwSHOVyT9eZBEdLavmx98BKVQBUNTGLBQZNRr14-OdwY3Z59Tpu2_cKWzBoc896IQw16gz4t3MoVwSxYLzNHhxtKSbfKj9XiENVzErIpD9pLcopy02q0ejfNqQAauhko8J6p6bKAlyfatC3dQsfETazDZhgxpRFq_Syj1mdxj-iTU_2RZUnZfPe2h4E4jYO60W0egx7ko_z6_xo6r1bNSGs3xPmm5rQxa740b6C-9E1FVXC-Rzmv_vA"
# curl -k -H "Authorization: Bearer $TOKEN" https://kubernetes.default.svc/api/v1/nodes/node1/proxy/metrics/cadvisor
...
# HELP machine_scrape_error 1 if there was an error while getting machine metrics, 0 otherwise.
# TYPE machine_scrape_error gauge
machine_scrape_error 0
# HELP machine_swap_bytes Amount of swap memory available on the machine.
# TYPE machine_swap_bytes gauge
machine_swap_bytes{boot_id="1a2887f7-5a8e-47d1-9a77-d73e4043ccf9",machine_id="cac1e9451931438495adeced5c4e2a42",system_uuid="17a8aaa7-a2be-436c-a1c5-4dc1536b468a"} 0