kubernetes-k8s-prometheus
普罗米修斯也可监控k8s相关功能
相关仓库:https://github.com/coreos/kube-prometheus/
-
下载配置文件
查看k8s版本
kubectl version,选择适配的版本kube-prometheus stack Kubernetes 1.29 Kubernetes 1.30 Kubernetes 1.31 Kubernetes 1.32 Kubernetes 1.33 Kubernetes 1.34 release-0.14✔ ✔ ✔ x x x release-0.15x x ✔ ✔ ✔ x release-0.16x x ✔ ✔ ✔ ✔ mainx x x ✔ ✔ ✔ git clone https://github.com/prometheus-operator/kube-prometheus.git -b release-0.16其他文件可删除保留 manifests 文件夹即可
-
下载相关镜像,导入到私有仓库内(以下以0.16版本为例)
离线包以存入:k8s-images | OpenList
quay.io/prometheus/alertmanager:v0.28.1 quay.io/prometheus/blackbox-exporter:v0.27.0 ghcr.io/jimmidyson/configmap-reload:v0.15.0 quay.io/brancz/kube-rbac-proxy:v0.19.1 grafana/grafana:12.1.0 registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.16.0 quay.io/prometheus/node-exporter:v1.9.1 registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0 quay.io/prometheus-operator/prometheus-operator:v0.85.0 quay.io/prometheus/prometheus:v3.5.0 quay.io/prometheus-operator/prometheus-config-reloader:v0.85.0 -
新建一个文件夹,将manifests 文件夹移动到里面
-
更改资源文件内的镜像地址,资源太多,推荐用下列脚本修改
#!/bin/bash PRIVATE_REPO="hub.godone.top/pub/" #这里修改为私有仓库地址 # 第一步:通用 image 替换(保留镜像名) grep -rl "image: " *.yaml | while read FILENAME; do echo "Processing (generic image) $FILENAME" sed -i "s|image: .*/\([^/]*\):\([^[:space:]]*\)|image: ${PRIVATE_REPO}\1:\2|" "$FILENAME" done # 第二步:专门替换 prometheus-config-reloader 的完整路径(无论出现在哪) # 注意:这里直接在整个文件中搜索并替换,不限于 image 行 for FILENAME in *.yaml; do #if grep -q "quay.io/prometheus-operator/prometheus-config-reloader" "$FILENAME"; then if grep -q "quay.io/prometheus-operator/prometheus-config-reloader" "$FILENAME"; then echo "Processing (specific reloader) $FILENAME" sed -i "s|quay.io/prometheus-operator/prometheus-config-reloader|${PRIVATE_REPO}prometheus-config-reloader|g" "$FILENAME" fi done echo "镜像地址替换完成" -
部署资源文件
部署资源:
kubectl apply --server-side -f manifests/setup kubectl wait \ --for condition=Established \ --all CustomResourceDefinition \ --namespace=monitoring kubectl apply -f manifests/删除资源:
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup -
部署成功后会创建一个名为 monitoring 的命名空间,kube-prometheus相关资源都部署在这
[root@k8s-master manifests]# kubectl -n monitoring get pod NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 116m alertmanager-main-1 2/2 Running 0 116m alertmanager-main-2 2/2 Running 0 116m blackbox-exporter-79c7f8686b-xt7cm 3/3 Running 0 116m grafana-86c5d99cd4-r2dg7 1/1 Running 0 56m kube-state-metrics-5c5987cc47-pr2h8 3/3 Running 0 116m node-exporter-6vh47 2/2 Running 0 116m node-exporter-8fvqn 2/2 Running 0 116m node-exporter-wjfgs 2/2 Running 0 116m prometheus-adapter-6f5f89cc6f-5df87 1/1 Running 0 116m prometheus-adapter-6f5f89cc6f-kmpqr 1/1 Running 0 116m prometheus-k8s-0 2/2 Running 0 116m prometheus-k8s-1 2/2 Running 0 116m prometheus-operator-77847d49f7-lzglb 2/2 Running 0 116m -
kube-prometheus默认只会创建ClusterIp,如果要在集群外访问的话,可以自己创建一个NodePode资源
--- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 12.1.0 name: grafana-NodePort namespace: monitoring spec: type: nodePort ports: - name: http port: 3000 targetPort: 3000 nodePort: 30000 selector: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus
补充
有时候 grafana 可能会报以下错误
Warning Unhealthy 13m (x5 over 14m) kubelet Readiness probe failed: Get "http://10.244.1.222:3000/api/health": dial tcp 10.244.1.222:3000: connect: connection refused
Warning Unhealthy 10m (x3 over 13m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning Unhealthy 4m9s (x40 over 13m) kubelet Readiness probe failed: Get "http://10.244.1.222:3000/api/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
这是因为grafana的健康检查所导致,可以修改manifests目录下的 grafana-deployment.yaml 配置文件,在以下位置加入新内容
...
readinessProbe:
httpGet:
path: /api/health
port: http
initialDelaySeconds: 60 #容器启动后,等待 60 秒才开始第一次健康检查。
periodSeconds: 30 #每隔 30 秒执行一次健康检查。
timeoutSeconds: 10 #每次健康检查的超时时间为 10 秒。
failureThreshold: 5 #连续 5 次健康检查失败后,才将 Pod 标记为“未就绪”。
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
...