【容器监控系列教程】Prometheus普罗米修斯监控Kubernetes

本教程涉及到的所有文件下载地址：https://pan.baidu.com/s/1IOk6PMEleZY5O_hNg9agxw?pwd=hng7
关于文件中提供的tar包，是教程用到的image镜像文件，我已经打包好了，可以使用docker load把镜像快速上传到node节点上。
更多关于Docker镜像操作教程：https://www.wsjj.top/archives/134

一、Prometheus普罗米修斯的介绍

• prometheus是由谷歌研发的一款开源的监控软件，它通过安装在远程机器上的exporter，通过HTTP协议从远程的机器收集数据并存储在本地的时序数据库上
• 同时Prometheus后端用golang语言开发，前端是Grafana
• Prometheus为了支持各种中间件以及第三方的监控提供了exporter，大家可以把它理解成监控适配器，将不同指标类型和格式的数据统一转化为Prometheus能够识别的指标类型。
• 例如Node exporter主要通过读取Linux的/proc以及/sys目录下的系统文件获取操作系统运行状态，reids exporter通过Reids命令行获取指标，mysql exporter通过读取数据库监控表获取MySQL的性能数据。他们将这些异构的数据转化为标准的Prometheus格式，并提供HTTP查询接口。

二、部署Prometheus服务端

1.使用ConfigMap存储Prometheus配置文件

[root@k8s-master prometheus]# vim prometheus-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: prometheus
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']

[root@k8s-master prometheus]# kubectl create -f prometheus-config.yaml

2.部署Prometheus服务端

[root@k8s-master prometheus]# vim prometheus-deploy.yaml

apiVersion: v1
kind: "Service"
metadata:
  name: prometheus
  namespace: prometheus
  labels:
    name: prometheus
spec:
  ports:
  - name: prometheus
    protocol: TCP
    port: 9090
    targetPort: 9090
  selector:
    app: prometheus
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: prometheus
  labels:
    name: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:v2.37.8
        imagePullPolicy: IfNotPresent
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - mountPath: "/etc/prometheus"
          name: prometheus-config
      volumes:
      - name: prometheus-config
        configMap:
          name: prometheus-config

[root@k8s-master prometheus]# kubectl create -f prometheus-deploy.yaml

查看服务和端口

[root@k8s-master prometheus]# kubectl get svc -n prometheus
NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
prometheus   NodePort   172.16.40.197   <none>        9090:30678/TCP   3h16m

[root@k8s-master prometheus]# kubectl get pod -n prometheus -o wide
NAME                          READY   STATUS    RESTARTS   AGE    IP                NODE                   NOMINATED NODE   READINESS GATES
prometheus-5cdccb54f6-xnpf2   1/1     Running   2          143m   192.168.201.213   k8s-node01.linux.com   <none>           <none>

浏览器访问测试

三、访问授权

为了让prometheus能够获取到kubernetes集群中的状态数据，需要为其创建用户并进行授权。kubernets基于RBAC机制实现认证授权，因此需要创建用户、角色，并将用户与角色进行关联。

1.创建用户和角色

[root@k8s-master prometheus]# vim prometheus-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: prometheus

• ClusterRole是属于全局的，不需要指定命名空间
• ServiceAccount必须指定命名空间

[root@k8s-master prometheus]# kubectl create -f prometheus-rbac.yaml

2.配置Prometheus关联用户

更新prometheus-deploy.yaml文件

[root@k8s-master prometheus]# vim prometheus-deploy.yaml
#配置文件不完整，仅展示添加部分
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus	#关联刚刚创建的用户
      containers:
      - name: prometheus
        image: prom/prometheus:v2.37.8
        imagePullPolicy: IfNotPresent

更新配置文件

[root@k8s-master prometheus]# kubectl apply -f prometheus-deploy.yaml

[root@k8s-master prometheus]# kubectl get pod -n prometheus
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-5cdccb54f6-xnpf2   1/1     Running   2          154m

重启Pod

删除Pod等待自动创建
教程后面将默认不再赘述，遇到重启或者重建Pod步骤时，请自行删除Pod

[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-xnpf2 -n prometheus

检查容器内部证书

配置完用户后，会默认在/var/run/secrets/kubernetes.io/serviceaccount/目录下生成和K8s集群通信用的证书

[root@k8s-master prometheus]# kubectl exec -it prometheus-5cdccb54f6-xnpf2 -n prometheus -- ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt     namespace  token

四、配置服务发现

1.介绍

prometheus与kubernetes API集成主要有5种方式，分别为Node, Service, Pod, Endpoints, Ingress

2.为了让promethues能够获取到集群所有节点的信息，在prometheus-config.yaml配置文件中，添加如下job配置

指定kubernetes_sd_config的模式为node，Prometheus会自动从kubernetes中发现所有node节点并且作为当前Job监控的Target

[root@k8s-master prometheus]# vim prometheus-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: prometheus
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']
      - job_name: 'kubernetes-nodes'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node

      - job_name: 'kubernetes-service'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: service

      - job_name: 'kubernetes-endpoints'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: endpoints

      - job_name: 'kubernetes-ingress'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: ingress

      - job_name: 'kubernetes-pods'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: pod

[root@k8s-master prometheus]# kubectl apply -f prometheus-config.yaml

重建Prometheus的Pod

删除Pod等待自动创建

[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-xnpf2 -n prometheus

浏览器访问查看

刷新访问prometheus页面，可以查看到发现的K8S集群中的资源

五、配置监控Kubelet

1.修改prometheus-config.yaml文件，添加监控项

[root@k8s-master prometheus]# vim prometheus-config.yaml
#在配置文件末尾添加以下内容

      - job_name: 'kubernetes-kubelet'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics

更新配置并重启容器

[root@k8s-master prometheus]# kubectl apply -f prometheus-config.yaml
configmap/prometheus-config configured

[root@k8s-master prometheus]# kubectl get pods -n prometheus
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-5cdccb54f6-xnpf2   1/1     Running   2          3h1m
[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-xnpf2 -n prometheus
pod "prometheus-5cdccb54f6-xnpf2" deleted

访问web端查看是否更新

六、通过cAdvisor监控节点容器资源使用情况

1.修改prometheus-config.yaml文件，添加监控项

[root@k8s-master prometheus]# vim prometheus-config.yaml
#在配置文件末尾添加以下内容

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)

更新配置并重启容器

[root@k8s-master prometheus]# kubectl apply -f prometheus-config.yaml
configmap/prometheus-config configured

[root@k8s-master prometheus]# kubectl get pods -n prometheus
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-5cdccb54f6-rj4fc   1/1     Running   0          5m19s
[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-rj4fc -n prometheus
pod "prometheus-5cdccb54f6-rj4fc" deleted

访问web端查看是否更新

七、基于node-exporter监控集群资源使用情况

1.使用DaemonSet类型部署node-exporter

[root@k8s-master prometheus]# vim node-exporter.yml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: prometheus
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9100'
        prometheus.io/path: 'metrics'
      labels:
        app: node-exporter
      name: node-exporter
    spec:
      containers:
      - image: prom/node-exporter
        imagePullPolicy: IfNotPresent
        name: node-exporter
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: scrape
      hostNetwork: true
      hostPID: true	#node exporter要运行在物理机上，因此指定了hostNetwork和hostPID参数

[root@k8s-master prometheus]# kubectl create -f node-exporter.yml
daemonset.apps/node-exporter created

[root@k8s-master prometheus]# kubectl get pod -n prometheus -o wide
NAME                          READY   STATUS    RESTARTS   AGE     IP                NODE                   NOMINATED NODE   READINESS GATES
node-exporter-hvkdb           1/1     Running   0          2m22s   192.168.140.12    k8s-node02.linux.com   <none>           <none>
node-exporter-ktfm8           1/1     Running   0          2m22s   192.168.140.11    k8s-node01.linux.com   <none>           <none>
prometheus-5cdccb54f6-wprjh   1/1     Running   0          23m     192.168.242.158   k8s-node02.linux.com   <none>           <none>

2.添加采集任务

[root@k8s-master prometheus]# vim prometheus-config.yaml
#在配置文件末尾添加内容

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)

      - job_name: 'kubernetes-node-exporter'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

3.更新配置文件

[root@k8s-master prometheus]# kubectl apply -f prometheus-config.yaml 
configmap/prometheus-config configured

[root@k8s-master prometheus]# kubectl get pods -n prometheus
NAME                          READY   STATUS    RESTARTS   AGE
node-exporter-hvkdb           1/1     Running   0          6m1s
node-exporter-ktfm8           1/1     Running   0          6m1s
prometheus-5cdccb54f6-wprjh   1/1     Running   0          27m

[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-wprjh -n prometheus
pod "prometheus-5cdccb54f6-wprjh" deleted

4.访问web端查看

八、配置从api-server获取监控指标

1.修改Prometheus-config添加监控项

[root@k8s-master prometheus]# vim prometheus-config.yaml
#在文件末尾添加内容

      - job_name: 'kubernetes-apiservers'
        kubernetes_sd_configs:
        - role: endpoints
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https
        - target_label: __address__
          replacement: kubernetes.default.svc:443

2.更新配置文件，重启Pod

[root@k8s-master prometheus]# kubectl apply -f prometheus-config.yaml
configmap/prometheus-config configured

[root@k8s-master prometheus]# kubectl get pods -n prometheus
NAME                          READY   STATUS    RESTARTS   AGE
node-exporter-hvkdb           1/1     Running   0          16m
node-exporter-ktfm8           1/1     Running   0          16m
prometheus-5cdccb54f6-vsw6m   1/1     Running   0          9m47s

[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-vsw6m -n prometheus
pod "prometheus-5cdccb54f6-vsw6m" deleted

3.访问web端查看

九、配置grafana图形化

1.部署grafana

[root@k8s-master prometheus]# vim grafana.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-core
  namespace: prometheus
  labels:
    app: grafana
    component: core
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: grafana/grafana:9.5.3
        name: grafana-core
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
          - name: GF_AUTH_BASIC_ENABLED
            value: "true"
          - name: GF_AUTH_ANONYMOUS_ENABLED
            value: "false"
        readinessProbe:
          httpGet:
            path: /login
            port: 3000
        volumeMounts:
        - name: grafana-persistent-storage
          mountPath: /var/lib/grafana
      volumes:
      - name: grafana-persistent-storage
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: prometheus
  labels:
    app: grafana
spec:
  type: NodePort
  ports:
    - port: 3000
  selector:
    app: grafana

[root@k8s-master prometheus]# kubectl create -f grafana.yaml

2.查看Pod

[root@k8s-master prometheus]# kubectl get pods -n prometheus -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP                NODE                   NOMINATED NODE   READINESS GATES
grafana-core-6497854f7b-z26nw   1/1     Running   0          46s   192.168.201.215   k8s-node01.linux.com   <none>           <none>
node-exporter-hvkdb             1/1     Running   0          66m   192.168.140.12    k8s-node02.linux.com   <none>           <none>
node-exporter-ktfm8             1/1     Running   0          66m   192.168.140.11    k8s-node01.linux.com   <none>           <none>
prometheus-5cdccb54f6-xgkk8     1/1     Running   0          49m   192.168.242.159   k8s-node02.linux.com   <none>           <none>

3.查看svc

[root@k8s-master prometheus]# kubectl get svc -n prometheus -o wide
NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE     SELECTOR
grafana      NodePort   172.16.34.121   <none>        3000:30916/TCP   63s     app=grafana
prometheus   NodePort   172.16.40.197   <none>        9090:30678/TCP   5h28m   app=prometheus

4.登录web界面

5.添加数据源

6.导入模板

模板ID：162

7.显示内存、CPU、磁盘的使用情况

内存

(sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes) ) / sum(node_memory_MemTotal_bytes) * 100

把上面的内容复制进去

另外2个图形也是同样的操作，替换里面的内容

CPU

sum(sum by (name)( rate(container_cpu_usage_seconds_total{image!=""}[1m] ) )) / count(node_cpu_seconds_total{mode="system"}) * 100

磁盘

(sum(node_filesystem_size_bytes{device="/dev/mapper/centos-root"}) - sum(node_filesystem_free_bytes{device="/dev/mapper/centos-root"}) ) / sum(node_filesystem_size_bytes{device="/dev/mapper/centos-root"}) * 100

效果图展示

8.保存图形

9.主页面快速查看创建的图形

十、监控Mysql

1.创建Mysql的Pod

我这里使用上期教程部署WordPress用到的Mysql
关于Kubernetes部署WordPress博客教程：https://www.wsjj.top/archives/147

[root@k8s-master prometheus]# vim /root/wordpress/wp-master-mysql.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: wp-master-mysql
    namespace: wordpress
spec:
    replicas: 1
    selector:
        matchLabels:
            app: wp-master-mysql
    serviceName: "wp-master-mysql"
    template:
        metadata:
            labels:
                app: wp-master-mysql
        spec:
            containers:
            - name: wp-master-mysql
              image: mysql:5.7
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 3306
              env:
              - name: MYSQL_ROOT_PASSWORD
                value: "WWW.1.com"
              - name: MYSQL_DATABASE
                value: "wordpress"
              - name: MYSQL_USER
                value: "wpuser"
              - name: MYSQL_PASSWORD
                value: "redhat"
              resources:
                requests:
                  memory: "1G"
                  cpu: "1"
                limits:
                  memory: "2G"
                  cpu: "2"
              volumeMounts:
                  - name: mysql-data-volume
                    mountPath: /var/lib/mysql
                  - name: mysql-config-volume
                    mountPath: /etc/mysql/conf.d
                    readOnly: true
            volumes:
                - name: mysql-data-volume
                  persistentVolumeClaim:
                    claimName: master-db-pvc
                - name: mysql-config-volume
                  projected:
                    sources:
                    - configMap:
                        name: master-db-config
---
apiVersion: v1
kind: Service
metadata:
    name: wp-master-mysql
    namespace: wordpress
spec:
    ports:
    - port: 3306
    selector:
      app: wp-master-mysql

[root@k8s-master prometheus]# kubectl create -f /root/wordpress/wp-master-mysql.yml

2.登录Mysql创建监控需要的用户

[root@k8s-master prometheus]# kubectl exec -it wp-master-mysql-0 -n wordpress -- bash
bash-4.2# mysql -uroot -pWWW.1.com

mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'prometheus'@'%' IDENTIFIED BY 'WWW.1.com';

Query OK, 0 rows affected, 1 warning (0.05 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)

3.部署Mysql-exporter

[root@k8s-master prometheus]# vim mysql-exporter.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-exporter
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-exporter
  template:
    metadata:
      labels:
        app: mysql-exporter
    spec:
      containers:
      - name: mysql-exporter
        image: prom/mysqld-exporter:v0.14.0
        env:
        - name: DATA_SOURCE_NAME
          value: "prometheus:WWW.1.com@(wp-master-mysql.wordpress.svc.cluster.local:3306)/" 	#配置全名连接地址，和刚才创建的用户
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9104
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-exporter
  namespace: prometheus
spec:
  type: ClusterIP
  selector:
    app: mysql-exporter
  ports:
  - name: api
    port: 9104
    protocol: TCP

[root@k8s-master prometheus]# kubectl create -f mysql-exporter.yaml
deployment.apps/mysql-exporter created
service/mysql-exporter created

查看Mysql-exporter

[root@k8s-master prometheus]# kubectl get pods -n prometheus
NAME                              READY   STATUS    RESTARTS   AGE
grafana-core-6497854f7b-z26nw     1/1     Running   0          85m
mysql-exporter-856d6b9fd9-gh7kg   1/1     Running   0          12s
node-exporter-hvkdb               1/1     Running   0          151m
node-exporter-ktfm8               1/1     Running   0          151m
prometheus-5cdccb54f6-xgkk8       1/1     Running   0          133m

[root@k8s-master prometheus]# kubectl get svc -n prometheus
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
grafana          NodePort    172.16.34.121    <none>        3000:30916/TCP   87m
mysql-exporter   ClusterIP   172.16.228.238   <none>        9104/TCP         101s
prometheus       NodePort    172.16.40.197    <none>        9090:30678/TCP   6h54m

更新prometheus-config.yaml文件

[root@k8s-master prometheus]# vim prometheus-config.yaml
#在文件末尾添加以下内容

      - job_name: 'mysql_monitor'
        static_configs:
        - targets: ['mysql-exporter:9104']	#服务连接地址

[root@k8s-master prometheus]# kubectl apply -f prometheus-config.yaml 
configmap/prometheus-config configured

[root@k8s-master prometheus]# kubectl get pod -n prometheus
NAME                              READY   STATUS    RESTARTS   AGE
grafana-core-6497854f7b-z26nw     1/1     Running   0          91m
mysql-exporter-856d6b9fd9-gh7kg   1/1     Running   0          5m44s
node-exporter-hvkdb               1/1     Running   0          156m
node-exporter-ktfm8               1/1     Running   0          156m
prometheus-5cdccb54f6-xgkk8       1/1     Running   0          139m

[root@k8s-master prometheus]# kubectl delete pod prometheus-5cdccb54f6-xgkk8 -n prometheus
pod "prometheus-5cdccb54f6-xgkk8" deleted

访问Prometheus的web界面

4.在Grafana添加数据源

模板ID：7362