helm으로 template kube-prometheus-stack(구 명칭 prometheus-operator)를 구축하고 kubernetes를 모니터링하는 시스템을 구축한다.

kube-prometheus-stack template 생성

https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

kube-prometheus-stack template 생성

# helm template prometheus kube-prometheus-stack \
  --repo https://prometheus-community.github.io/helm-charts \
  --dependency-update --include-crds --skip-tests > ../prometheus-deploy.yaml

만약, prometheus, alertmanager, grafana를 Loadbalancer (80 port)로 서비스를 구성하려면 아래와 같이 template을 생성하면 된다.

# helm template prometheus kube-prometheus-stack \
  --repo https://prometheus-community.github.io/helm-charts \
  --dependency-update --include-crds --skip-tests \
  --set alertmanager.service.type=LoadBalancer \
  --set alertmanager.service.port=80 \
  --set prometheus.service.type=LoadBalancer \
  --set prometheus.service.port=80 \
  --set grafana.service.type=LoadBalancer  > prometheus-deploy.yaml

kubernetes에 배포

# kubectl apply -f prometheus-deploy.yaml --server-side

"--server-side" 옵션을 붙여서 배포한다. "--client-side"로 배포 시, 오류가 발생된다.

최초 배포 시, 아래와 같이 오류가 발생될 수 있다. "kubectl apply" 명령어로 다시 한번 배포하면 문제 없다.

# kubectl apply -f prometheus-deploy.yaml --server-side
...
...

resource mapping not found for name: "prometheus-kube-prometheus-kube-proxy" namespace: "default" from "prometheus-deploy.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
resource mapping not found for name: "prometheus-kube-prometheus-kube-scheduler" namespace: "default" from "prometheus-deploy.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
resource mapping not found for name: "prometheus-kube-prometheus-kubelet" namespace: "default" from "prometheus-deploy.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
resource mapping not found for name: "prometheus-kube-prometheus-operator" namespace: "default" from "prometheus-deploy.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
resource mapping not found for name: "prometheus-kube-prometheus-prometheus" namespace: "default" from "prometheus-deploy.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first

구성된 서비스 확인 (Loadbalancer 방식으로 구성)

# kubectl get svc
NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                       AGE
alertmanager-operated                     ClusterIP      None            <none>           9093/TCP,9094/TCP,9094/UDP    6s
echo-server-svc                           ClusterIP      10.233.28.255   <none>           80/TCP                        111m
kubernetes                                ClusterIP      10.233.0.1      <none>           443/TCP                       24h
metrics-server                            ClusterIP      10.233.20.91    <none>           443/TCP                       23h
prometheus-grafana                        LoadBalancer   10.233.22.192   192.168.122.12   80:31015/TCP                  8s
prometheus-kube-prometheus-alertmanager   LoadBalancer   10.233.2.85     192.168.122.13   80:32398/TCP,8080:32131/TCP   7s
prometheus-kube-prometheus-operator       ClusterIP      10.233.46.82    <none>           443/TCP                       7s
prometheus-kube-prometheus-prometheus     LoadBalancer   10.233.58.125   192.168.122.14   80:32451/TCP,8080:31951/TCP   7s
prometheus-kube-state-metrics             ClusterIP      10.233.38.126   <none>           8080/TCP                      8s
prometheus-operated                       ClusterIP      None            <none>           9090/TCP                      6s
prometheus-prometheus-node-exporter       ClusterIP      10.233.14.99    <none>           9100/TCP                      7s

kube-prometheus-stack 구성 확인

Prometheus WEBUI 확인 (http://192.168.122.14)

prometheus-kube-prometheus-prometheus     LoadBalancer   10.233.58.125   192.168.122.14   80:32451/TCP,8080:31951/TCP   7s

Alertmanager WEBUI 확인 (http://192.168.122.13)

prometheus-kube-prometheus-alertmanager   LoadBalancer   10.233.2.85     192.168.122.13   80:32398/TCP,8080:32131/TCP   7s

grafana WEBUI 확인 (http://192.168.122.12)

prometheus-grafana                        LoadBalancer   10.233.22.192   192.168.122.12   80:31015/TCP                  8s

패스워드 : admin / prom-operator

아래와 같이 생성된 template로 확인이 가능하다

# cat prometheus-deploy.yaml |grep admin-password
  admin-password: "cHJvbS1vcGVyYXRvcg=="
...
...

# echo "cHJvbS1vcGVyYXRvcg==" |base64 -d;echo
prom-operator