Helm Chart部署Kafka添加helm chart仓库helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update 自定义配置安装之前可以下载chart包，解压后查看values.yaml中的配置项，根据实际情况进行修改。helm pull bitnami/kafka --version 31.5.0 tar zxvf kafka-31.5.0.tgz 配置私有镜像仓库global: imageRegistry: "docker.ascs.tech" security: allowInsecureImages: true 配置外部访问启用外部访问（默认不启用）。服务类型配置为NodePort并指定nodePorts，nodePorts类型是一个列表，列表的长度要和kafka的副本数相对应，bitnami kafka chart默认有三个副本，所以这里配置三个nodePort端口。同理，externalIPs也要跟副本数相对应。externalAccess: enabled: true controller: service: type: NodePort nodePorts: - 30092 - 30093 - 30094 externalIPs: - 192.168.100.149 - 192.168.100.149 - 192.168.100.149 配置访问协议配置所有的listeners访问协议为PLAINTEXT​。在默认情况下，访问协议为SASL_PLAINTEXT​，当启用SASL​后就需要使用用户名密码访问Kafka集群，可以通过修改sasl.client.users​ 和 sasl.client.passwords​来指定用户名和密码。如果密码未指定，会生成随机密码，可以通过命令 kubectl get secret {{ .Release.Name }}-user-passwords -o jsonpath='{.data.client-passwords}' | base64 -d​ 进行查看。listeners: client: protocol: PLAINTEXT controller: protocol: PLAINTEXT interbroker: protocol: PLAINTEXT external: protocol: PLAINTEXT 其它配置项资源请求限制如果不需要对Pod资源做限制，可以将resourcesPreset的值设置为none​。如果需要对其进行合理的设置，可以参考_resources.tpl1controller.resourcesPreset broker.resourcesPreset volumePermissions.resourcesPreset metrics.jmx.resourcesPreset 存储卷默认的PVC大小为8Gi，可以根据实际需要进行调整controller.persistence.size 完整配置文件# 指定私有仓库,加速镜像下载 global: imageRegistry: "docker.ascs.tech" security: allowInsecureImages: true # 配置允许外部访问. 默认不开启此配置 externalAccess: enabled: true controller: service: type: NodePort nodePorts: - 30092 - 30093 - 30094 externalIPs: - 192.168.100.149 - 192.168.100.149 - 192.168.100.149 # 配置访问协议为PLAINTEXT(明文). 默认配置为SASL_PLAINTEXT listeners: client: protocol: PLAINTEXT controller: protocol: PLAINTEXT interbroker: protocol: PLAINTEXT external: protocol: PLAINTEXT 部署helm upgrade --install kafka-dev \ bitnami/kafka \ -f values.yaml \ --namespace middleware \ --create-namespace \ --version 31.5.0 验证打开IDEA，安装Kafka插件。安装成功后，点击左侧工具窗口，选择Kafka，点击添加新的连接。在引导服务器中输入 192.168.100.149:30092,192.168.100.149:30093,192.168.100.149:30094​ ，点击测试连接进行验证。另外，可以下载Kafka二进制包，使用包中的bat脚本（Windows）或者sh脚本（Linux）进行验证。_resources.tpl {{/* Copyright Broadcom, Inc. All Rights Reserved. SPDX-License-Identifier: APACHE-2.0 */}} {{/* vim: set filetype=mustache: */}} {{/* Return a resource request/limit object based on a given preset. These presets are for basic testing and not meant to be used in production {{ include "common.resources.preset" (dict "type" "nano") -}} */}} {{- define "common.resources.preset" -}} {{/* The limits are the requests increased by 50% (except ephemeral-storage and xlarge/2xlarge sizes)*/}} {{- $presets := dict "nano" (dict "requests" (dict "cpu" "100m" "memory" "128Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "150m" "memory" "192Mi" "ephemeral-storage" "2Gi") ) "micro" (dict "requests" (dict "cpu" "250m" "memory" "256Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "375m" "memory" "384Mi" "ephemeral-storage" "2Gi") ) "small" (dict "requests" (dict "cpu" "500m" "memory" "512Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "750m" "memory" "768Mi" "ephemeral-storage" "2Gi") ) "medium" (dict "requests" (dict "cpu" "500m" "memory" "1024Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "750m" "memory" "1536Mi" "ephemeral-storage" "2Gi") ) "large" (dict "requests" (dict "cpu" "1.0" "memory" "2048Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "1.5" "memory" "3072Mi" "ephemeral-storage" "2Gi") ) "xlarge" (dict "requests" (dict "cpu" "1.0" "memory" "3072Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "3.0" "memory" "6144Mi" "ephemeral-storage" "2Gi") ) "2xlarge" (dict "requests" (dict "cpu" "1.0" "memory" "3072Mi" "ephemeral-storage" "50Mi") "limits" (dict "cpu" "6.0" "memory" "12288Mi" "ephemeral-storage" "2Gi") ) }} {{- if hasKey $presets .type -}} {{- index $presets .type | toYaml -}} {{- else -}} {{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}} {{- end -}} {{- end -}} ↩
环境kubernets version：v1.27.15Docker version：27.0.3Helm version：v3.15.2GitLab version：v17.4.1快速启动添加helm gitlab chart仓库helm repo add gitlab https://charts.gitlab.io helm repo update执行下面命令快速进行部署helm upgrade --install gitlab gitlab/gitlab \ --version 8.4.1 \ --timeout 6000s \ --set global.hosts.domain=example.com \ --set certmanager-issuer.email=mail@example.com \ --namespace=gitlab \ --create-namespace这里我们快速启动一套GitLab服务，这里选择自定义了一些配置项--version 8.4.1 ：Chart版本为8.4.1，对应的GitLab版本为v17.4.1--timeout 6000s：部署超时时间为6000sglobal.hosts.domain=example.com：配置域名，后续实际生成的域名会根据所配置的域名进行扩展，比如配置了example.com，后续GitLab服务的域名就是gitlab.example.com，Minio服务的域名为minio.example.com。当然，也可以针对每个组件服务自定义域名，详情查看values.yaml或者查阅官方文档certmanager-issuer.email：当使用certmanager提供的证书，必须提供一个邮箱除此，其余配置都为默认配置。等待片刻，Job执行完成，Pod启动成功，就可以使用域名在浏览器访问GitLab了。默认的用户名为：root，密码可通过下面命令获取：kubectl get secret -n gitlab gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo自定义部署在很多情况下，我们不需要安装chart中的所有组件，比如集群环境中已经部署certmanager或者计划使用已有的证书，那就不需要再安装certmanager，我们就可以certmanager.install设置为false，同时设置global.ingress.tls.secretName来自定义证书；还有，如果不需要部署runner服务，那我们就可以将gitlab-runner.install设置为false避免安装runner。配置私有镜像仓库配置私有镜像仓库，加快镜像拉取速度global: enterpriseImages: # Default repositories used to pull Gitlab Enterprise Edition images. # See the image.repository and workhorse.repository template helpers. migrations: repository: docker.example.com/gitlab-org/build/cng/gitlab-toolbox-ee sidekiq: repository: docker.example.com/gitlab-org/build/cng/gitlab-sidekiq-ee toolbox: repository: docker.example.com/gitlab-org/build/cng/gitlab-toolbox-ee webservice: repository: docker.example.com/gitlab-org/build/cng/gitlab-webservice-ee workhorse: repository: docker.example.com/gitlab-org/build/cng/gitlab-workhorse-ee geo-logcursor: repository: docker.example.com/gitlab-org/build/cng/gitlab-geo-logcursor certificates: image: repository: docker.example.com/gitlab-org/build/cng/certificates kubectl: image: repository: docker.example.com/gitlab-org/build/cng/kubectl gitlabBase: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-base gitlab: gitlab-shell: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-shell service: name: gitlab-shell type: NodePort nodePort: 32022 kas: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-kas gitlab-exporter: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-exporter gitaly: image: repository: docker.example.com/gitlab-org/build/cng/gitaly cgroups: initContainer: image: repository: docker.example.com/gitlab-org/build/cng/gitaly-init-cgroups praefect: image: repository: docker.example.com/gitlab-org/build/cng/gitaly registry: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-container-registry自定义IngressClassGitLab默认安装nginx-ingress，在集群中已经安装有nginx-ingress的前提下，就需要禁止安装：nginx-ingress.enabled=false：禁止安装GitLab自带的nginx-ingressglobal.ingress.class=traefik-ingress：设置ingressClassName为已有的traefik-ingress修改values.yaml，禁用内置Ingress Class，手动指定已有的Ingress Classglobal: ingress: class: traefik-ingress nginx-ingress: &nginx-ingress enabled: false自定义SSL证书自定义SSL证书的时候，就可以不用安装certmanager，需要修改以下配置项：--set certmanager.install=false：禁止部署certmanager--set global.ingress.tls.secretName=example-com-tls：自定义证书创建自定义证书kubectl create secret tls -n gitlab example-com-tls --key privkey.pem --cert fullchain.pem修改values.yaml，手动指定已经创建好的证书global: ingress: tls: secretName: example-com-tls配置LDAP创建secret gitlab-ldap-main-password 用来存放LDAP密码kubectl create secret generic -n gitlab gitlab-ldap-main-password --from-literal=password=changeme修改values.yaml，更新LDAP相关配置 appConfig: ldap: preventSignin: false servers: main: base: ou=Gitlab用户组,dc=example,dc=com bind_dn: uid=admin,ou=system encryption: plain host: 192.168.1.100 label: LDAP password: secret: gitlab-ldap-main-password port: 30389 uid: uid user_filter: (objectclass=*)配置SMTP创建secret gitlab-outgoing-mail 用来存放邮箱密码kubectl create secret generic-n gitlab gitlab-outgoing-mail --from-literal=password=changeme修改values.yaml，更新SMTP相关配置 smtp: enabled: true address: smtp.ym.163.com port: 994 user_name: no-reply@example.com password: secret: gitlab-outgoing-mail authentication: "plain" tls: true email: from: "no-reply@example.com"更改gitlab-shell service类型为了保证可以通过ssh拉取代码，这里需要修改gitlab-shell的服务类型为NodePort，nodePort为32022global: shell: port: 32022 gitlab: gitlab-shell: service: name: gitlab-shell type: NodePort nodePort: 32022完整values.yaml针对以上的一些自定义选项，整理一份较完整的清单global: enterpriseImages: # Default repositories used to pull Gitlab Enterprise Edition images. # See the image.repository and workhorse.repository template helpers. migrations: repository: docker.example.com/gitlab-org/build/cng/gitlab-toolbox-ee sidekiq: repository: docker.example.com/gitlab-org/build/cng/gitlab-sidekiq-ee toolbox: repository: docker.example.com/gitlab-org/build/cng/gitlab-toolbox-ee webservice: repository: docker.example.com/gitlab-org/build/cng/gitlab-webservice-ee workhorse: repository: docker.example.com/gitlab-org/build/cng/gitlab-workhorse-ee geo-logcursor: repository: docker.example.com/gitlab-org/build/cng/gitlab-geo-logcursor certificates: image: repository: docker.example.com/gitlab-org/build/cng/certificates kubectl: image: repository: docker.example.com/gitlab-org/build/cng/kubectl gitlabBase: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-base ## https://docs.gitlab.com/charts/charts/globals#configure-host-settings hosts: domain: example.com ## https://docs.gitlab.com/charts/charts/globals#configure-ingress-settings ingress: class: traefik-ingress tls: secretName: example-com-tls appConfig: ## https://docs.gitlab.com/charts/charts/globals#ldap ldap: preventSignin: false servers: main: base: ou=Gitlab用户组,dc=example,dc=com bind_dn: uid=admin,ou=system encryption: plain host: 192.168.200.12 label: LDAP password: secret: gitlab-ldap-main-password port: 30389 uid: uid user_filter: (objectclass=*) ## https://docs.gitlab.com/charts/charts/globals#outgoing-email ## Outgoing email server settings smtp: enabled: true address: smtp.ym.163.com port: 994 user_name: no-reply@example.com ## https://docs.gitlab.com/charts/installation/secrets#smtp-password password: secret: gitlab-outgoing-mail # domain: authentication: "plain" tls: true ## https://docs.gitlab.com/charts/charts/globals#outgoing-email ## Email persona used in email sent by GitLab email: from: "no-reply@example.com" ## Timezone for containers. time_zone: Asia/Shanghai ## https://docs.gitlab.com/charts/charts/globals#configure-gitlab-shell shell: port: 32022 ## Settings for individual sub-charts under GitLab ## Note: Many of these settings are configurable via globals gitlab: gitlab-shell: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-shell service: name: gitlab-shell type: NodePort nodePort: 32022 kas: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-kas gitlab-exporter: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-exporter gitaly: image: repository: docker.example.com/gitlab-org/build/cng/gitaly cgroups: initContainer: image: repository: docker.example.com/gitlab-org/build/cng/gitaly-init-cgroups praefect: image: repository: docker.example.com/gitlab-org/build/cng/gitaly registry: image: repository: docker.example.com/gitlab-org/build/cng/gitlab-container-registry ## Settings to for the Let's Encrypt ACME Issuer certmanager-issuer: # The email address to register certificates requested from Let's Encrypt. # Required if using Let's Encrypt. email: mail@example.com ## Installation & configuration of jetstack/cert-manager ## See requirements.yaml for current version certmanager: installCRDs: false # Install cert-manager chart. Set to false if you already have cert-manager # installed or if you are not using cert-manager. install: false nginx-ingress: &nginx-ingress enabled: false ## Installation & configuration of gitlab/gitlab-runner ## See requirements.yaml for current version gitlab-runner: install: false minio: persistence: size: 100Gi部署helm upgrade --install gitlab gitlab/gitlab \ --version 8.4.1 \ --timeout 6000s \ -f values.yaml \ --namespace=gitlab \ --create-namespace部署完成后，获取超级管理员root用户的密码kubectl get secret -n gitlab gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
Kubeadm设置clusterName使用Kubeadm可以快速的创建一个k8s集群，但是默认情况下Kubeadm创建的集群名称都是kubernetes(集群名称可以使用命令kubectl config get-clusters查询)。在实际使用中，如果管理多个k8s集群，就需要对集群名称做区分，方便管理维护。通过网上查找资料，现在可以通过修改kubeadm初始化配置文件，自定义clusterName，操作步骤如下：[scode type="yellow"]配置文件的功能仍然处于 alpha 状态并且在将来的版本中可能会改变，但是一些更加高级的功能只能够通过配置文件设置，比如clusterName[/scode]使用命令kubeadm config print init-defaults > kubeadm-init.yaml获取kubeadm init 的默认 init 配置对象，并将其写入到文件kubeadm-init.yaml中，文件内容如下：apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock imagePullPolicy: IfNotPresent name: node taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: 1.23.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {}对生成的默认配置文件进行修改，这里按照实际需求，对下面的几个配置项进行修改：集群名称clusterName、镜像仓库地址imageRepository、Service网段networking.serviceSubnet、Pod网段networking.podSubnet，最终修改后的文件内容如下：apiVersion: kubeadm.k8s.io/v1beta3 clusterName: k8s-dev imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration networking: serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16使用修改后的文件初始化集群kubeadm init --config kubeadm-init.yaml踩坑flannel CrashLoopBackOff问题和原因集群初始化操作完成之后，部署flannel，过一会就会看到flannel Pod CrashLoopBackOff，查看flannel日志有这么一行Error registering network: failed to acquire lease: node "ubuntu-vm" pod cidr not assigned，很明确告知了flannel CrashLoopBackOff 的原因是未分配Pod网段。I0503 10:17:21.246851 1 main.go:205] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true} W0503 10:17:21.246945 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0503 10:17:21.342762 1 kube.go:378] Starting kube subnet manager I0503 10:17:21.342913 1 kube.go:120] Waiting 10m0s for node controller to sync I0503 10:17:22.344507 1 kube.go:127] Node controller sync successful I0503 10:17:22.344528 1 main.go:225] Created subnet manager: Kubernetes Subnet Manager - ubuntu-vm I0503 10:17:22.344531 1 main.go:228] Installing signal handlers I0503 10:17:22.344745 1 main.go:454] Found network config - Backend type: vxlan I0503 10:17:22.344771 1 match.go:189] Determining IP address of default interface I0503 10:17:22.344988 1 match.go:242] Using interface with name enp0s5 and address 10.33.33.7 I0503 10:17:22.345003 1 match.go:264] Defaulting external address to interface address (10.33.33.7) I0503 10:17:22.345054 1 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false E0503 10:17:22.345208 1 main.go:317] Error registering network: failed to acquire lease: node "ubuntu-vm" pod cidr not assigned I0503 10:17:22.345281 1 main.go:434] Stopping shutdownHandler... W0503 10:17:22.345306 1 reflector.go:436] github.com/flannel-io/flannel/subnet/kube/kube.go:379: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding解决方法集群初始化之前，在配置文件中定义networking.podSubnet: 10.244.0.0/16，再进行初始化集群的操作如果集群已经启动通过打补丁的方式配置解决，kubectl patch node $(hostname) -p '{"spec":{"podCIDR":"10.244.0.0/24"}}'（如果集群只是初始化尚未使用，建议kubeadm reset删除集群，重新初始化）参考：stackoverflow： Kubernetes cluster name changeGitHub：Ability to configure user and cluster name in AdminKubeConfigFileGithub：https://github.com/kubernetes/kubernetes/pull/60852Github：Error registering network: failed to acquire lease: node "nodeName" pod cidr not assigned
kube-promethesu解决KubeProxyDown等默认告警在kube-promethesu部署完成之后，会出现KubeControllerManagerDown、KubeProxyDown、KubeSchedulerDown、Watchdog四个告警Watchdog查看Watchdog的告警描述：This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always firing, therefore it should always be firing in Alertmanager and always fire against a receiver. There are integrations with various notification mechanisms that send a notification when this alert is not firing. For example the "DeadMansSnitch" integration in PagerDuty.，简单来说，这个告警是为了保证告警功能可以正常使用，如果这个告警消失，那就表示告警系统出了问题。如果不需要这个功能，可以修改prometheus目录中的相关文件kubePrometheus-prometheusRule.yaml，注释或删除相关的内容。apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-prometheus app.kubernetes.io/part-of: kube-prometheus prometheus: k8s role: alert-rules name: kube-prometheus-rules namespace: monitoring spec: groups: - name: general.rules ... # - alert: Watchdog # annotations: # description: | # This is an alert meant to ensure that the entire alerting pipeline is functional. # This alert is always firing, therefore it should always be firing in Alertmanager # and always fire against a receiver. There are integrations with various notification # mechanisms that send a notification when this alert is not firing. For example the # "DeadMansSnitch" integration in PagerDuty. # runbook_url: https://runbooks.prometheus-operator.dev/runbooks/general/watchdog # summary: An alert that should always be firing to certify that Alertmanager # is working properly. # expr: vector(1) # labels: # severity: none - name: node-network ... - name: kube-prometheus-node-recording.rules ... - name: kube-prometheus-general.rules ...修改完成后，使用kubectl apply -f prometheus/kubePrometheus-prometheusRule.yaml更新配置，稍等片刻，就可以看到告警Watchdog已从告警列表消失。KubeControllerManagerDownKubeControllerManagerDown告警描述：KubeControllerManager has disappeared from Prometheus target discovery.，翻译：KubeControllerManager已经从Prometheus目标发现中消失。原因是这个文件kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml定义了一个ServiceMonitor资源对象，最后一段从名称空间kube-system中匹配包含labelapp.kubernetes.io/name: kube-controller-manager的svcapiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/name: kube-controller-manager app.kubernetes.io/part-of: kube-prometheus name: kube-controller-manager namespace: monitoring spec: ... jobLabel: app.kubernetes.io/name namespaceSelector: matchNames: - kube-system selector: matchLabels: app.kubernetes.io/name: kube-controller-manager创建一个新目录other，在这个目录中新建一个yaml文件kube-system-Service.yaml，文件内容如下：apiVersion: v1 kind: Service metadata: name: kube-controller-manager namespace: kube-system labels: # 这里定义的标签要保证跟ServiceMonitor资源对象中selector的标签一致 app.kubernetes.io/name: kube-controller-manager spec: selector: # 这里选择的标签要保证是kube-controller-manager Pod中定义的标签之一 component: kube-controller-manager ports: - port: 10257 targetPort: 10257 name: https-metrics修改完成后，使用kubectl apply -f other/创建kube-controller-manager Service资源，稍等片刻，就可以看到Service已经创建成功 ➜ ~ kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-controller-manager ClusterIP 10.98.64.108 <none> 10257/TCP 110s kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 11d kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 10d然而，告警并没有消失，但是告警内容发生了改变：PrometheusTargetMissing，这时在prometheus Web查看targets，跟kube-controller-manager相关的三个全都不在线，原因是kube-controller-manager默认监听地址是127.0.0.1，导致无法外部访问。在k8s集群每个master节点修改/etc/kubernetes/manifests/kube-controller-manager.yaml，将- --bind-address=127.0.0.1改为- --bind-address=0.0.0.0，保存退出，对应节点的pod会自动重启（如果没有pod没有自动重启，重启kubelet服务或手动重启）。等待pod重启完成，刷新Web，可以看到target都已在线，告警KubeControllerManagerDown已从告警列表消失。KubeSchedulerDownKubeSchedulerDown同理，在kube-system-Service.yaml后面追加kube-scheduler Service。apiVersion: v1 kind: Service metadata: name: kube-controller-manager namespace: kube-system labels: app.kubernetes.io/name: kube-controller-manager spec: selector: component: kube-controller-manager ports: - port: 10257 targetPort: 10257 name: https-metrics --- apiVersion: v1 kind: Service metadata: name: kube-scheduler namespace: kube-system labels: app.kubernetes.io/name: kube-scheduler spec: selector: component: kube-scheduler ports: - port: 10259 targetPort: 10259 name: https-metrics使用kubectl apply -f other/创建kube-scheduler Service资源。在k8s集群每个master节点修改/etc/kubernetes/manifests/kube-scheduler.yaml，将- --bind-address=127.0.0.1改为- --bind-address=0.0.0.0，保存退出，对应节点的pod会自动重启（如果没有pod没有自动重启，重启kubelet服务或手动重启）。重启完成，刷新Web，可以看到target已在线，告警KubeSchedulerDown已从告警列表消失。KubeProxyDownKubeProxyDown跟KubeControllerManagerDown和KubeSchedulerDown类似，不同的是kube-prometheus并没有为kube-proxy创建一个serviceMonitor对象。首先参考其它serviceMonitor资源对象的yaml文件在kubernetesControlPlane目录下新建一个yaml文件kubernetesControlPlane-serviceMonitorKubeProxy.yaml，定义一个名为kube-proxy的ServiceMonitor对象，文件内容如下：apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/name: kube-proxy app.kubernetes.io/part-of: kube-prometheus name: kube-proxy namespace: monitoring spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: http-metrics scheme: http tlsConfig: insecureSkipVerify: true jobLabel: app.kubernetes.io/name namespaceSelector: matchNames: - kube-system selector: matchLabels: app.kubernetes.io/name: kube-proxy使用kubectl apply -f kubernetesControlPlane/kubernetesControlPlane-serviceMonitorKubeProxy.yaml创建资源。继续在kube-system-Service.yaml后面追加kube-proxy Service。apiVersion: v1 kind: Service metadata: name: kube-controller-manager namespace: kube-system labels: app.kubernetes.io/name: kube-controller-manager spec: selector: component: kube-controller-manager ports: - port: 10257 targetPort: 10257 name: https-metrics --- apiVersion: v1 kind: Service metadata: name: kube-scheduler namespace: kube-system labels: app.kubernetes.io/name: kube-scheduler spec: selector: component: kube-scheduler ports: - port: 10259 targetPort: 10259 name: https-metrics --- apiVersion: v1 kind: Service metadata: name: kube-proxy namespace: kube-system labels: app.kubernetes.io/name: kube-proxy spec: selector: k8s-app: kube-proxy ports: - port: 10249 targetPort: 10249 name: http-metrics使用kubectl apply -f other/创建kube-proxy Service资源。kube-proxy也需要修改监听地址，在任意一台master节点修改kube-proxy ConfigMapkubectl edit cm/kube-proxy -n kube-system将metricsBindAddress: ""改为metricsBindAddress: "0.0.0.0:10249"，重启Pod。重启完成，稍等片刻，就可以在Web看到kube-proxy的target都已经在线，告警KubeProxyDown已从告警列表消失。
自定义抓取配置在prometheus目录下创建一个新目录AdditionalScrapeConfigs用来存放集群外部一些抓取的配置文件 ➜ ~ mkdir -p prometheus/AdditionalScrapeConfigs在目录AdditionalScrapeConfigs中新建两个文件：additional-scrape-configs-Secret.yaml和additional-scrape-configs-ConfigMap.yaml，示例如下：additional-scrape-configs-Secret.yaml（新建）apiVersion: v1 kind: Secret metadata: name: prometheus-additional-scrape-configs-secret namespace: monitoring labels: app.kubernetes.io/component: prometheus-targets app.kubernetes.io/instance: main app.kubernetes.io/name: prometheus-additional-scrape-configs app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.23.0 stringData: prometheus-additional-scrape.yaml: | # 以下内容自定义 - job_name: node-exporter file_sd_configs: - files: - /additional-scrape-configs/node-exporter.yaml refresh_interval: 30s relabel_configs: - source_labels: - __address__ regex: '(.*):9100' target_label: instance action: replace replacement: $1 - job_name: container-exporter file_sd_configs: - files: - /additional-scrape-configs/container-exporter.yaml refresh_interval: 30s type: Opaqueadditional-scrape-configs-ConfigMap.yaml（新建）apiVersion: v1 kind: ConfigMap metadata: name: prometheus-additional-scrape-configs-configmap namespace: monitoring labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: main app.kubernetes.io/name: prometheus-additional-scrape-configs app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.23.0 data: node-exporter.yaml: | # 以下内容自定义 - targets: - "192.168.33.254:9100" container-exporter.yaml: | # 以下内容自定义 - targets: - "192.168.33.254:9080"prometheus-prometheus.yaml（更新）apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.32.1 name: k8s namespace: monitoring spec: ... version: 2.32.1 ## 新增以下配置 # 挂载configMap volumeMounts: - name: prometheus-additional-scrape-configs-configmap mountPath: /additional-scrape-configs readOnly: true # 定义volumes volumes: - name: prometheus-additional-scrape-configs-configmap configMap: name: prometheus-additional-scrape-configs-configmap # 添加集群外的targets additionalScrapeConfigs: name: prometheus-additional-scrape-configs-secret key: prometheus-additional-scrape.yaml使用kubectl apply创建和更新相关资源# 创建定义的Secret和ConfigMap资源 ➜ ~ kubectl apply -f prometheus/AdditionalScrapeConfigs configmap/prometheus-additional-scrape-configs-configmap created secret/prometheus-additional-scrape-configs-secret created # 更新Prometheus资源 ➜ ~ kubectl apply -f prometheus/prometheus-prometheus.yaml prometheus.monitoring.coreos.com/k8s configured稍等片刻，浏览器打开prometheus Web，可以看到自定义的两个Target已经在线自定义告警规则在prometheus目录下创建一个新目录AdditionalAlertRules用来存放一些自定义告警规则配置文件。 ➜ ~ mkdir -p prometheus/AdditionalAlertRules在目录AdditionalAlertRules中新建一个文件：additional-alert-rules-prometheusRule.yaml，示例如下：apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: app.kubernetes.io/component: additional-alert-rules app.kubernetes.io/name: additional-alert-rules app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.3.0 prometheus: k8s role: alert-rules name: additional-alert-rules namespace: monitoring spec: groups: - name: additional-alert-rules rules: ## 以下内容自定义 # example 1 - alert: PrometheusTargetMissing expr: up == 0 for: 0m labels: severity: critical annotations: summary: Prometheus target missing (instance {{ $labels.instance }}) description: "A Prometheus target has disappeared. An exporter might be crashed.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" # example 2 - alert: HostOutOfDiskSpace expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and ON (instance, device, mountpoint) node_filesystem_readonly == 0 for: 2m labels: severity: warning annotations: summary: Host out of disk space (instance {{ $labels.instance }}) description: "Disk is almost full (< 10% left)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"使用kubectl apply创建相关资源# 创建定义的PrometheusRule资源 ➜ ~ kubectl apply -f prometheus/AdditionalAlertRules prometheusrule.monitoring.coreos.com/additional-alert-rules created浏览器打开prometheus Web，可以看到自定义的两条规则已经在线为了测试规则是否生效，在根目录创建一个大文件，等待片刻，就会在alertmanager Web看到有一条磁盘使用率的告警产生自定义route和receivers首先备份alertmanager目录下的alertmanager-secret.yaml ➜ ~ cp alertmanager/alertmanager-secret.yaml alertmanager/alertmanager-secret.yaml.backup然后修改alertmanager-secret.yaml，将自定义的route和receivers添加到该文件中，示例如下：apiVersion: v1 kind: Secret metadata: labels: app.kubernetes.io/component: alert-router app.kubernetes.io/instance: main app.kubernetes.io/name: alertmanager app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.23.0 name: alertmanager-main namespace: monitoring stringData: alertmanager.yaml: |- # 以下内容自定义 global: resolve_timeout: 1m route: group_by: ['instance'] group_wait: 10s group_interval: 5m repeat_interval: 5m receiver: 'PrometheusAlert' receivers: - name: 'PrometheusAlert' webhook_configs: - url: http://192.168.33.3:8080/prometheusalert?type=fs&tpl=custom-prometheus-feishu&fsurl=https://open.feishu.cn/open-apis/bot/v2/hook/000000-0000-0000-0000-000000000000 type: Opaque使用kubectl apply更新相关资源# 更新Secret资源 ➜ ~ kubectl apply -f alertmanager/alertmanager-secret.yaml secret/alertmanager-main configured浏览器打开alertmanager Web，点击菜单Status，可以看到自定义的路由和告警接受方式[collapse status="false" title="目录结构 (12 directories, 89 files)"]. ├── alertmanager │ ├── alertmanager-alertmanager.yaml │ ├── alertmanager-podDisruptionBudget.yaml │ ├── alertmanager-prometheusRule.yaml │ ├── alertmanager-secret.yaml │ ├── alertmanager-secret.yaml.backup # 备份文件 │ ├── alertmanager-serviceAccount.yaml │ ├── alertmanager-serviceMonitor.yaml │ └── alertmanager-service.yaml ├── blackboxExporter │ ├── blackboxExporter-clusterRoleBinding.yaml │ ├── blackboxExporter-clusterRole.yaml │ ├── blackboxExporter-configuration.yaml │ ├── blackboxExporter-deployment.yaml │ ├── blackboxExporter-serviceAccount.yaml │ ├── blackboxExporter-serviceMonitor.yaml │ └── blackboxExporter-service.yaml ├── grafana │ ├── grafana-config.yaml │ ├── grafana-dashboardDatasources.yaml │ ├── grafana-dashboardDefinitions.yaml │ ├── grafana-dashboardSources.yaml │ ├── grafana-deployment.yaml │ ├── grafana-PersistentVolumeClaim.yaml # 创建Grafana PVC │ ├── grafana-serviceAccount.yaml │ ├── grafana-serviceMonitor.yaml │ └── grafana-service.yaml ├── kubernetesControlPlane │ ├── kubernetesControlPlane-prometheusRule.yaml │ ├── kubernetesControlPlane-serviceMonitorApiserver.yaml │ ├── kubernetesControlPlane-serviceMonitorCoreDNS.yaml │ ├── kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml │ ├── kubernetesControlPlane-serviceMonitorKubelet.yaml │ └── kubernetesControlPlane-serviceMonitorKubeScheduler.yaml ├── kubeStateMetrics │ ├── kubeStateMetrics-clusterRoleBinding.yaml │ ├── kubeStateMetrics-clusterRole.yaml │ ├── kubeStateMetrics-deployment.yaml │ ├── kubeStateMetrics-prometheusRule.yaml │ ├── kubeStateMetrics-serviceAccount.yaml │ ├── kubeStateMetrics-serviceMonitor.yaml │ └── kubeStateMetrics-service.yaml ├── nodeExporter │ ├── nodeExporter-clusterRoleBinding.yaml │ ├── nodeExporter-clusterRole.yaml │ ├── nodeExporter-daemonset.yaml │ ├── nodeExporter-prometheusRule.yaml │ ├── nodeExporter-serviceAccount.yaml │ ├── nodeExporter-serviceMonitor.yaml │ └── nodeExporter-service.yaml ├── prometheus │ ├── AdditionalAlertRules # 自定义抓取目标配置 │ │ └── additional-alert-rules-prometheusRule.yaml │ ├── AdditionalScrapeConfigs # 自定义告警规则 │ │ ├── additional-scrape-configs-ConfigMap.yaml │ │ └── additional-scrape-configs-Secret.yaml │ ├── kubePrometheus-prometheusRule.yaml │ ├── prometheus-clusterRoleBinding.yaml │ ├── prometheus-clusterRole.yaml │ ├── prometheus-podDisruptionBudget.yaml │ ├── prometheus-prometheusRule.yaml │ ├── prometheus-prometheus.yaml │ ├── prometheus-roleBindingConfig.yaml │ ├── prometheus-roleBindingSpecificNamespaces.yaml │ ├── prometheus-roleConfig.yaml │ ├── prometheus-roleSpecificNamespaces.yaml │ ├── prometheus-serviceAccount.yaml │ ├── prometheus-serviceMonitor.yaml │ └── prometheus-service.yaml ├── prometheusAdapter │ ├── prometheusAdapter-apiService.yaml │ ├── prometheusAdapter-clusterRoleAggregatedMetricsReader.yaml │ ├── prometheusAdapter-clusterRoleBindingDelegator.yaml │ ├── prometheusAdapter-clusterRoleBinding.yaml │ ├── prometheusAdapter-clusterRoleServerResources.yaml │ ├── prometheusAdapter-clusterRole.yaml │ ├── prometheusAdapter-configMap.yaml │ ├── prometheusAdapter-deployment.yaml │ ├── prometheusAdapter-podDisruptionBudget.yaml │ ├── prometheusAdapter-roleBindingAuthReader.yaml │ ├── prometheusAdapter-serviceAccount.yaml │ ├── prometheusAdapter-serviceMonitor.yaml │ └── prometheusAdapter-service.yaml ├── prometheusOperator │ ├── prometheusOperator-clusterRoleBinding.yaml │ ├── prometheusOperator-clusterRole.yaml │ ├── prometheusOperator-deployment.yaml │ ├── prometheusOperator-prometheusRule.yaml │ ├── prometheusOperator-serviceAccount.yaml │ ├── prometheusOperator-serviceMonitor.yaml │ └── prometheusOperator-service.yaml └── setup ├── 0alertmanagerConfigCustomResourceDefinition.yaml ├── 0alertmanagerCustomResourceDefinition.yaml ├── 0podmonitorCustomResourceDefinition.yaml ├── 0probeCustomResourceDefinition.yaml ├── 0prometheusCustomResourceDefinition.yaml ├── 0prometheusruleCustomResourceDefinition.yaml ├── 0servicemonitorCustomResourceDefinition.yaml ├── 0thanosrulerCustomResourceDefinition.yaml └── namespace.yaml 12 directories, 89 files[/collapse]
介绍Kube-Prometheus官网该项目收集Kubernetes清单，Grafana仪表板和Prometheus规则，并结合文档和脚本，以使用Prometheus运算符提供易于操作的端到端Kubernetes集群监控。目前最新版本：v0.10.0，本文使用最新版本进行安装。Kube-Prometheus主要包括以下组件：Prometheus OperatorPrometheus Operator有以下功能：定义Kubernetes资源类型，比如Prometheus，Alertmanager，PrometheusRule等；简化部署和配置；基于Kubernetes标签自动生成target配置。高可用的Prometheus高可用的AlertmanagerPrometheus Node-exporterPrometheus Adapter for Kubernetes Metrics APIskube-state-metrics可视化仪表盘Grafana[scode type="yellow"]根据Kubernetes版本，选择合适的kube-prometheus版本[/scode]kube-prometheus stackKubernetes 1.19Kubernetes 1.20Kubernetes 1.21Kubernetes 1.22Kubernetes 1.23release-0.7✔✔✗✗✗release-0.8✗✔✔✗✗release-0.9✗✗✔✔✗release-0.10✗✗✗✔✔main✗✗✗✔✔环境说明操作系统：CentOS 7.9.2009Kube-Prometheus：v0.10.0Kubernetes：v1.23.5Docker：20.10.14一键部署克隆仓库到本地或者下载指定的版本的Releases包，下面命令是官方提供的部署命令，可以快速的启动。kubectl apply --server-side -f manifests/setup until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done kubectl apply -f manifests/但是会存在以下问题：官方yaml文件没有归类，不方便之后的集群管理；部分image需要从k8s.gcr.io拉取，如果是国内机器并没有合适的代理，那就会导致image无法拉取；Prometheus和Grafana没有配置数据持久化，一旦pod重启或被删除，数据也将被删除；默认的Service无法暴露服务端口到集群外的客户端。自定义部署为了针对性地解决上面几个问题，我们进行以下处理yaml文件分类Kube-Prometheus v0.10.0一共有84个yaml文件 ::aru:distressed:: 为了方便管理，根据服务类型将相应的yaml文件进行分类是很有必要的。创建目录mkdir alertmanager blackboxExporter grafana kubeStateMetrics kubernetesControlPlane nodeExporter prometheus prometheusAdapter prometheusOperator根据服务类型yaml文件进行分类mv alertmanager-* ./alertmanager mv blackboxExporter-* ./blackboxExporter mv grafana-* ./grafana mv kubeStateMetrics-* ./kubeStateMetrics mv kubernetesControlPlane-* ./kubernetesControlPlane mv nodeExporter-* ./nodeExporter mv prometheus-* ./prometheus mv prometheusAdapter-* ./prometheusAdapter mv prometheusOperator-* ./prometheusOperator mv kubePrometheus-prometheusRule.yaml ./prometheus此时目录结构如下[collapse status="false" title="目录结构 (10 directories, 84 files)"]. ├── alertmanager │ ├── alertmanager-alertmanager.yaml │ ├── alertmanager-podDisruptionBudget.yaml │ ├── alertmanager-prometheusRule.yaml │ ├── alertmanager-secret.yaml │ ├── alertmanager-service.yaml │ ├── alertmanager-serviceAccount.yaml │ └── alertmanager-serviceMonitor.yaml ├── blackboxExporter │ ├── blackboxExporter-clusterRole.yaml │ ├── blackboxExporter-clusterRoleBinding.yaml │ ├── blackboxExporter-configuration.yaml │ ├── blackboxExporter-deployment.yaml │ ├── blackboxExporter-service.yaml │ ├── blackboxExporter-serviceAccount.yaml │ └── blackboxExporter-serviceMonitor.yaml ├── grafana │ ├── grafana-config.yaml │ ├── grafana-dashboardDatasources.yaml │ ├── grafana-dashboardDefinitions.yaml │ ├── grafana-dashboardSources.yaml │ ├── grafana-deployment.yaml │ ├── grafana-service.yaml │ ├── grafana-serviceAccount.yaml │ └── grafana-serviceMonitor.yaml ├── kubeStateMetrics │ ├── kubeStateMetrics-clusterRole.yaml │ ├── kubeStateMetrics-clusterRoleBinding.yaml │ ├── kubeStateMetrics-deployment.yaml │ ├── kubeStateMetrics-prometheusRule.yaml │ ├── kubeStateMetrics-service.yaml │ ├── kubeStateMetrics-serviceAccount.yaml │ └── kubeStateMetrics-serviceMonitor.yaml ├── kubernetesControlPlane │ ├── kubernetesControlPlane-prometheusRule.yaml │ ├── kubernetesControlPlane-serviceMonitorApiserver.yaml │ ├── kubernetesControlPlane-serviceMonitorCoreDNS.yaml │ ├── kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml │ ├── kubernetesControlPlane-serviceMonitorKubeScheduler.yaml │ └── kubernetesControlPlane-serviceMonitorKubelet.yaml ├── nodeExporter │ ├── nodeExporter-clusterRole.yaml │ ├── nodeExporter-clusterRoleBinding.yaml │ ├── nodeExporter-daemonset.yaml │ ├── nodeExporter-prometheusRule.yaml │ ├── nodeExporter-service.yaml │ ├── nodeExporter-serviceAccount.yaml │ └── nodeExporter-serviceMonitor.yaml ├── prometheus │ ├── kubePrometheus-prometheusRule.yaml │ ├── prometheus-clusterRole.yaml │ ├── prometheus-clusterRoleBinding.yaml │ ├── prometheus-podDisruptionBudget.yaml │ ├── prometheus-prometheus.yaml │ ├── prometheus-prometheusRule.yaml │ ├── prometheus-roleBindingConfig.yaml │ ├── prometheus-roleBindingSpecificNamespaces.yaml │ ├── prometheus-roleConfig.yaml │ ├── prometheus-roleSpecificNamespaces.yaml │ ├── prometheus-service.yaml │ ├── prometheus-serviceAccount.yaml │ └── prometheus-serviceMonitor.yaml ├── prometheusAdapter │ ├── prometheusAdapter-apiService.yaml │ ├── prometheusAdapter-clusterRole.yaml │ ├── prometheusAdapter-clusterRoleAggregatedMetricsReader.yaml │ ├── prometheusAdapter-clusterRoleBinding.yaml │ ├── prometheusAdapter-clusterRoleBindingDelegator.yaml │ ├── prometheusAdapter-clusterRoleServerResources.yaml │ ├── prometheusAdapter-configMap.yaml │ ├── prometheusAdapter-deployment.yaml │ ├── prometheusAdapter-podDisruptionBudget.yaml │ ├── prometheusAdapter-roleBindingAuthReader.yaml │ ├── prometheusAdapter-service.yaml │ ├── prometheusAdapter-serviceAccount.yaml │ └── prometheusAdapter-serviceMonitor.yaml ├── prometheusOperator │ ├── prometheusOperator-clusterRole.yaml │ ├── prometheusOperator-clusterRoleBinding.yaml │ ├── prometheusOperator-deployment.yaml │ ├── prometheusOperator-prometheusRule.yaml │ ├── prometheusOperator-service.yaml │ ├── prometheusOperator-serviceAccount.yaml │ └── prometheusOperator-serviceMonitor.yaml └── setup ├── 0alertmanagerConfigCustomResourceDefinition.yaml ├── 0alertmanagerCustomResourceDefinition.yaml ├── 0podmonitorCustomResourceDefinition.yaml ├── 0probeCustomResourceDefinition.yaml ├── 0prometheusCustomResourceDefinition.yaml ├── 0prometheusruleCustomResourceDefinition.yaml ├── 0servicemonitorCustomResourceDefinition.yaml ├── 0thanosrulerCustomResourceDefinition.yaml └── namespace.yaml 10 directories, 84 files[/collapse]解决k8s.gcr.io镜像拉取失败k8s.gcr.io因为大陆防火墙的原因，国内网络无法直接访问，所以需要其它方法来获取镜像。这里我们使用DockerHub同步的镜像来解决这个问题。具体方法可以参考下面两种，任选其一即可：[tabs][tab name="修改yaml文件中的image地址"]推荐多节点使用！将文件./kubeStateMetrics/kubeStateMetrics-deployment.yaml中的k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0修改为buall/kube-state-metrics:v2.3.0将文件./prometheusAdapter/prometheusAdapter-deployment.yaml中的k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1修改为buall/prometheus-adapter:v0.9.1[/tab][tab name="更新DockerHub镜像 tag" active="true"]这种方法不需要修改yaml文件，推荐单节点使用！docker pull buall/kube-state-metrics:v2.3.0 docker tag buall/kube-state-metrics:v2.3.0 k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0 docker pull buall/prometheus-adapter:v0.9.1 docker tag buall/prometheus-adapter:v0.9.1 k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1[/tab][/tabs]数据持久化[scode type="blue"]在配置数据持久化之前，首先得有一个StorageClass，StorageClass创建参考官方文档，推荐使用NFS，简单，容易上手。[/scode]Grafana数据持久化首先创建一个PVC，用来存储Grafana数据。在grafana目录中新建一个名为grafana-PersistentVolumeClaim.yaml的文件，并写入以下内容（创建一个名为grafana-pvc且大小为10G的PVC）。kind: PersistentVolumeClaim apiVersion: v1 metadata: name: grafana-pvc namespace: monitoring labels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 8.3.3 spec: storageClassName: nfs-client accessModes: - ReadWriteMany resources: requests: storage: 10Gi此时文件目录结构如下：[collapse status="false" title="目录结构 (10 directories, 85 files)"]. ├── alertmanager │ ├── alertmanager-alertmanager.yaml │ ├── alertmanager-podDisruptionBudget.yaml │ ├── alertmanager-prometheusRule.yaml │ ├── alertmanager-secret.yaml │ ├── alertmanager-service.yaml │ ├── alertmanager-serviceAccount.yaml │ └── alertmanager-serviceMonitor.yaml ├── blackboxExporter │ ├── blackboxExporter-clusterRole.yaml │ ├── blackboxExporter-clusterRoleBinding.yaml │ ├── blackboxExporter-configuration.yaml │ ├── blackboxExporter-deployment.yaml │ ├── blackboxExporter-service.yaml │ ├── blackboxExporter-serviceAccount.yaml │ └── blackboxExporter-serviceMonitor.yaml ├── grafana │ ├── grafana-config.yaml │ ├── grafana-dashboardDatasources.yaml │ ├── grafana-dashboardDefinitions.yaml │ ├── grafana-dashboardSources.yaml │ ├── grafana-deployment.yaml │ ├── grafana-PersistentVolumeClaim.yaml # 创建Grafana PVC │ ├── grafana-service.yaml │ ├── grafana-serviceAccount.yaml │ └── grafana-serviceMonitor.yaml ├── kubeStateMetrics │ ├── kubeStateMetrics-clusterRole.yaml │ ├── kubeStateMetrics-clusterRoleBinding.yaml │ ├── kubeStateMetrics-deployment.yaml │ ├── kubeStateMetrics-prometheusRule.yaml │ ├── kubeStateMetrics-service.yaml │ ├── kubeStateMetrics-serviceAccount.yaml │ └── kubeStateMetrics-serviceMonitor.yaml ├── kubernetesControlPlane │ ├── kubernetesControlPlane-prometheusRule.yaml │ ├── kubernetesControlPlane-serviceMonitorApiserver.yaml │ ├── kubernetesControlPlane-serviceMonitorCoreDNS.yaml │ ├── kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml │ ├── kubernetesControlPlane-serviceMonitorKubeScheduler.yaml │ └── kubernetesControlPlane-serviceMonitorKubelet.yaml ├── nodeExporter │ ├── nodeExporter-clusterRole.yaml │ ├── nodeExporter-clusterRoleBinding.yaml │ ├── nodeExporter-daemonset.yaml │ ├── nodeExporter-prometheusRule.yaml │ ├── nodeExporter-service.yaml │ ├── nodeExporter-serviceAccount.yaml │ └── nodeExporter-serviceMonitor.yaml ├── prometheus │ ├── kubePrometheus-prometheusRule.yaml │ ├── prometheus-clusterRole.yaml │ ├── prometheus-clusterRoleBinding.yaml │ ├── prometheus-podDisruptionBudget.yaml │ ├── prometheus-prometheus.yaml │ ├── prometheus-prometheusRule.yaml │ ├── prometheus-roleBindingConfig.yaml │ ├── prometheus-roleBindingSpecificNamespaces.yaml │ ├── prometheus-roleConfig.yaml │ ├── prometheus-roleSpecificNamespaces.yaml │ ├── prometheus-service.yaml │ ├── prometheus-serviceAccount.yaml │ └── prometheus-serviceMonitor.yaml ├── prometheusAdapter │ ├── prometheusAdapter-apiService.yaml │ ├── prometheusAdapter-clusterRole.yaml │ ├── prometheusAdapter-clusterRoleAggregatedMetricsReader.yaml │ ├── prometheusAdapter-clusterRoleBinding.yaml │ ├── prometheusAdapter-clusterRoleBindingDelegator.yaml │ ├── prometheusAdapter-clusterRoleServerResources.yaml │ ├── prometheusAdapter-configMap.yaml │ ├── prometheusAdapter-deployment.yaml │ ├── prometheusAdapter-podDisruptionBudget.yaml │ ├── prometheusAdapter-roleBindingAuthReader.yaml │ ├── prometheusAdapter-service.yaml │ ├── prometheusAdapter-serviceAccount.yaml │ └── prometheusAdapter-serviceMonitor.yaml ├── prometheusOperator │ ├── prometheusOperator-clusterRole.yaml │ ├── prometheusOperator-clusterRoleBinding.yaml │ ├── prometheusOperator-deployment.yaml │ ├── prometheusOperator-prometheusRule.yaml │ ├── prometheusOperator-service.yaml │ ├── prometheusOperator-serviceAccount.yaml │ └── prometheusOperator-serviceMonitor.yaml └── setup ├── 0alertmanagerConfigCustomResourceDefinition.yaml ├── 0alertmanagerCustomResourceDefinition.yaml ├── 0podmonitorCustomResourceDefinition.yaml ├── 0probeCustomResourceDefinition.yaml ├── 0prometheusCustomResourceDefinition.yaml ├── 0prometheusruleCustomResourceDefinition.yaml ├── 0servicemonitorCustomResourceDefinition.yaml ├── 0thanosrulerCustomResourceDefinition.yaml └── namespace.yaml 10 directories, 85 files[/collapse]然后修改文件grafana-deployment.yaml，按照下面的内容重新定义grafana-storage这个volume。from... volumes: - emptyDir: {} name: grafana-storage ...to... volumes: - name: grafana-storage persistentVolumeClaim: claimName: grafana-pvc ...完整文件参考：[collapse status="false" title="grafana-deployment.yaml (仅供参考)"]apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 8.3.3 name: grafana namespace: monitoring spec: replicas: 1 selector: matchLabels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus template: metadata: annotations: checksum/grafana-config: 8a40383dc6577c8b30c5bf006ba9ab7e checksum/grafana-dashboardproviders: cf4ac6c4d98eb91172b3307d9127acc5 checksum/grafana-datasources: 2e669e49f44117d62bc96ea62c2d39d3 labels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 8.3.3 spec: containers: - env: [] image: grafana/grafana:8.3.3 name: grafana ports: - containerPort: 3000 name: http readinessProbe: httpGet: path: /api/health port: http resources: limits: cpu: 200m memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - mountPath: /var/lib/grafana name: grafana-storage readOnly: false - mountPath: /etc/grafana/provisioning/datasources name: grafana-datasources readOnly: false - mountPath: /etc/grafana/provisioning/dashboards name: grafana-dashboards readOnly: false - mountPath: /grafana-dashboard-definitions/0/alertmanager-overview name: grafana-dashboard-alertmanager-overview readOnly: false - mountPath: /grafana-dashboard-definitions/0/apiserver name: grafana-dashboard-apiserver readOnly: false - mountPath: /grafana-dashboard-definitions/0/cluster-total name: grafana-dashboard-cluster-total readOnly: false - mountPath: /grafana-dashboard-definitions/0/controller-manager name: grafana-dashboard-controller-manager readOnly: false - mountPath: /grafana-dashboard-definitions/0/k8s-resources-cluster name: grafana-dashboard-k8s-resources-cluster readOnly: false - mountPath: /grafana-dashboard-definitions/0/k8s-resources-namespace name: grafana-dashboard-k8s-resources-namespace readOnly: false - mountPath: /grafana-dashboard-definitions/0/k8s-resources-node name: grafana-dashboard-k8s-resources-node readOnly: false - mountPath: /grafana-dashboard-definitions/0/k8s-resources-pod name: grafana-dashboard-k8s-resources-pod readOnly: false - mountPath: /grafana-dashboard-definitions/0/k8s-resources-workload name: grafana-dashboard-k8s-resources-workload readOnly: false - mountPath: /grafana-dashboard-definitions/0/k8s-resources-workloads-namespace name: grafana-dashboard-k8s-resources-workloads-namespace readOnly: false - mountPath: /grafana-dashboard-definitions/0/kubelet name: grafana-dashboard-kubelet readOnly: false - mountPath: /grafana-dashboard-definitions/0/namespace-by-pod name: grafana-dashboard-namespace-by-pod readOnly: false - mountPath: /grafana-dashboard-definitions/0/namespace-by-workload name: grafana-dashboard-namespace-by-workload readOnly: false - mountPath: /grafana-dashboard-definitions/0/node-cluster-rsrc-use name: grafana-dashboard-node-cluster-rsrc-use readOnly: false - mountPath: /grafana-dashboard-definitions/0/node-rsrc-use name: grafana-dashboard-node-rsrc-use readOnly: false - mountPath: /grafana-dashboard-definitions/0/nodes name: grafana-dashboard-nodes readOnly: false - mountPath: /grafana-dashboard-definitions/0/persistentvolumesusage name: grafana-dashboard-persistentvolumesusage readOnly: false - mountPath: /grafana-dashboard-definitions/0/pod-total name: grafana-dashboard-pod-total readOnly: false - mountPath: /grafana-dashboard-definitions/0/prometheus-remote-write name: grafana-dashboard-prometheus-remote-write readOnly: false - mountPath: /grafana-dashboard-definitions/0/prometheus name: grafana-dashboard-prometheus readOnly: false - mountPath: /grafana-dashboard-definitions/0/proxy name: grafana-dashboard-proxy readOnly: false - mountPath: /grafana-dashboard-definitions/0/scheduler name: grafana-dashboard-scheduler readOnly: false - mountPath: /grafana-dashboard-definitions/0/workload-total name: grafana-dashboard-workload-total readOnly: false - mountPath: /etc/grafana name: grafana-config readOnly: false nodeSelector: kubernetes.io/os: linux securityContext: fsGroup: 65534 runAsNonRoot: true runAsUser: 65534 serviceAccountName: grafana volumes: - name: grafana-storage persistentVolumeClaim: claimName: grafana-pvc - name: grafana-datasources secret: secretName: grafana-datasources - configMap: name: grafana-dashboards name: grafana-dashboards - configMap: name: grafana-dashboard-alertmanager-overview name: grafana-dashboard-alertmanager-overview - configMap: name: grafana-dashboard-apiserver name: grafana-dashboard-apiserver - configMap: name: grafana-dashboard-cluster-total name: grafana-dashboard-cluster-total - configMap: name: grafana-dashboard-controller-manager name: grafana-dashboard-controller-manager - configMap: name: grafana-dashboard-k8s-resources-cluster name: grafana-dashboard-k8s-resources-cluster - configMap: name: grafana-dashboard-k8s-resources-namespace name: grafana-dashboard-k8s-resources-namespace - configMap: name: grafana-dashboard-k8s-resources-node name: grafana-dashboard-k8s-resources-node - configMap: name: grafana-dashboard-k8s-resources-pod name: grafana-dashboard-k8s-resources-pod - configMap: name: grafana-dashboard-k8s-resources-workload name: grafana-dashboard-k8s-resources-workload - configMap: name: grafana-dashboard-k8s-resources-workloads-namespace name: grafana-dashboard-k8s-resources-workloads-namespace - configMap: name: grafana-dashboard-kubelet name: grafana-dashboard-kubelet - configMap: name: grafana-dashboard-namespace-by-pod name: grafana-dashboard-namespace-by-pod - configMap: name: grafana-dashboard-namespace-by-workload name: grafana-dashboard-namespace-by-workload - configMap: name: grafana-dashboard-node-cluster-rsrc-use name: grafana-dashboard-node-cluster-rsrc-use - configMap: name: grafana-dashboard-node-rsrc-use name: grafana-dashboard-node-rsrc-use - configMap: name: grafana-dashboard-nodes name: grafana-dashboard-nodes - configMap: name: grafana-dashboard-persistentvolumesusage name: grafana-dashboard-persistentvolumesusage - configMap: name: grafana-dashboard-pod-total name: grafana-dashboard-pod-total - configMap: name: grafana-dashboard-prometheus-remote-write name: grafana-dashboard-prometheus-remote-write - configMap: name: grafana-dashboard-prometheus name: grafana-dashboard-prometheus - configMap: name: grafana-dashboard-proxy name: grafana-dashboard-proxy - configMap: name: grafana-dashboard-scheduler name: grafana-dashboard-scheduler - configMap: name: grafana-dashboard-workload-total name: grafana-dashboard-workload-total - name: grafana-config secret: secretName: grafana-config[/collapse]Prometheus数据持久化修改prometheus-prometheus.yaml文件，在spec对象下添加以下内容spec: storage: volumeClaimTemplate: spec: storageClassName: nfs-client resources: requests: storage: 20Gi完整文件参考：[collapse status="false" title="prometheus-prometheus.yaml (仅供参考)"]apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.32.1 name: k8s namespace: monitoring spec: storage: volumeClaimTemplate: spec: storageClassName: nfs-client resources: requests: storage: 20Gi alerting: alertmanagers: - apiVersion: v2 name: alertmanager-main namespace: monitoring port: web enableFeatures: [] externalLabels: {} image: quay.io/prometheus/prometheus:v2.32.1 nodeSelector: kubernetes.io/os: linux podMetadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.32.1 podMonitorNamespaceSelector: {} podMonitorSelector: {} probeNamespaceSelector: {} probeSelector: {} replicas: 2 resources: requests: memory: 400Mi ruleNamespaceSelector: {} ruleSelector: {} securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus-k8s serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} version: 2.32.1 [/collapse]暴露服务暴露服务到集群外有多种方法，NodePort，Ingress，自己部署Nginx对ClusterIP做反向代理，这里选择相比之下比较简单的NodePort来实现。nodePort默认允许的范围：30000-32767Prometheus Service根据下面内容修改prometheus-service.yamlapiVersion: v1 kind: Service metadata: ... name: prometheus-k8s namespace: monitoring spec: ports: - name: web port: 9090 targetPort: web # 这里可以更改为允许的任意端口 nodePort: 30090 - name: reloader-web port: 8080 targetPort: reloader-web # 指定Service type type: NodePort ...Alertmanager Service根据下面内容修改alertmanager-service.yamlapiVersion: v1 kind: Service metadata: ... name: alertmanager-main namespace: monitoring spec: ports: - name: web port: 9093 targetPort: web # 这里可以更改为允许的任意端口 nodePort: 30093 - name: reloader-web port: 8080 targetPort: reloader-web # 指定Service type type: NodePort ...Grafana Service根据下面内容修改grafana-service.yamlapiVersion: v1 kind: Service metadata: ... name: grafana namespace: monitoring spec: ports: - name: http port: 3000 targetPort: http # 这里可以更改为允许的任意端口 nodePort: 30003 # 指定Service type type: NodePort ...启动setup目录中的yaml文件会创建CRD（CustomResourceDefinition），所以需要提前执行，其余的先后次序无要求。kubectl create -f setup kubectl apply -f alertmanager -f blackboxExporter -f grafana -f kubeStateMetrics -f kubernetesControlPlane -f nodeExporter -f prometheus -f prometheusAdapter -f prometheusOperator等待片刻，查看Pod状态，所有Pod状态都是Running的时候，表示服务都已正常启动 ::aru:cheer:: ➜ ~ kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 4d alertmanager-main-1 2/2 Running 0 4d alertmanager-main-2 2/2 Running 0 4d blackbox-exporter-6b79c4588b-9gd2q 3/3 Running 0 4d grafana-7cbbbdbf77-qwdng 1/1 Running 0 4d kube-state-metrics-56b4d9b89d-nf6zv 3/3 Running 0 4d node-exporter-9rk4r 2/2 Running 0 4d node-exporter-dncwr 2/2 Running 0 4d node-exporter-mxwj5 2/2 Running 0 4d node-exporter-qgv9m 2/2 Running 0 4d node-exporter-z8tqk 2/2 Running 0 4d prometheus-adapter-849f459d9c-jvj85 1/1 Running 0 4d prometheus-adapter-849f459d9c-tjxgf 1/1 Running 0 4d prometheus-k8s-0 2/2 Running 0 4d prometheus-k8s-1 2/2 Running 0 4d prometheus-operator-6dc9f66cb7-wzgv4 2/2 Running 0 4d查看Service ➜ ~ kubectl get svc -n monitoring | grep NodePort alertmanager-main NodePort 10.101.252.191 <none> 9093:30093/TCP,8080:30698/TCP 4d1h grafana NodePort 10.107.241.77 <none> 3000:30003/TCP 4d prometheus-k8s NodePort 10.105.187.184 <none> 9090:30090/TCP,8080:31749/TCP 3d23hWeb展示可以看到，Kube-Prometheus默认已对k8s集群内的机器进行了监控，如何让kube-prometheus也可以去监控其他的集群外部的主机，下篇再说 ::aru:discovertruth::参考：kube-prometheus README.mdprometheus-operator storage