68.k8s告警规则101706.pdf
勿以恶小而为之，勿以善小而不为。刘备人人好公，则天下太平；人人营私，则天下大乱。刘鹗68.k8s告警规则apiVersion: role:alert-rules name:container-hpa namespace:kubesphere-monitoring-systemspec:groups:-name:容器伸缩 rules:-alert:Pod伸缩异常 expr:kube_hpa_status_conditioncondition=AbleToScale,status=false=1 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes 伸缩异常 description:Kubernetes【$labels.instance】伸缩异常重启,请检查 -alert:HPA收集数据异常 expr:kube_hpa_status_conditioncondition=ScalingActive,status=false=1 for:1m志不强者智不达，言不信者行不果。墨翟丹青不知老将至，贫贱于我如浮云。杜甫 labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod HPA收集监控数据异常 description:Kubernetes Pod节点【$labels.instance】收集指标异常重启,请检查 -alert:HPA伸缩能 expr:kube_hpa_status_desired_replicas=kube_hpa_spec_max_replicas for:1m labels:level:p2 resources:container annotations:value:$value summary:Kubernetes Pod HPA扩容节点已经达到最 description:Kubernetes Pod节点【$labels.instance】扩容节点已经达到最,请检查apiVersion: role:alert-rules name:container-resources namespace:kubesphere-monitoring-system忍一句，息一怒，饶一着，退一步。增广贤文宠辱不惊，看庭前花开花落；去留无意，望天上云卷云舒。洪应明spec:groups:-name:容器资源 rules:-alert:CPU使率 expr:sum(rate(container_cpu_usage_seconds_totalcontainer!=,container!=POD2m)by(pod,namespace,instance,container)/(sum(container_spec_cpu_quotacontainer!=,container!=POD/100000)by(pod,namespace,instance,container)*100 60 for:1m labels:level:p4 resources:container annotations:value:$value summary:Kubernetes Pod CPU使率过 description:Kubernetes Pod节点【$labels.pod】CPU使率过，请检查 -alert:CPU使率 expr:sum(rate(container_cpu_usage_seconds_totalcontainer!=,container!=POD2m)by(pod,namespace,instance,container)/(sum(container_spec_cpu_quotacontainer!=,container!=POD/100000)by(pod,namespace,instance,container)*100 70 for:1m labels:level:p3 resources:container annotations:value:$value summary:Kubernetes Pod CPU使率过 description:Kubernetes Pod节点【$labels.pod】CPU使率过，请检查 -alert:CPU使率 expr:sum(rate(container_cpu_usage_seconds_totalcontainer!=,container!=POD2m)by(pod,namespace,instance,container)/(sum(container_spec_cpu_quotacontainer!=,container!=POD/100000)by(pod,namespace,instance,container)*100 80以铜为镜，可以正衣冠；以古为镜，可以知兴替；以人为镜，可以明得失。旧唐书魏征列传人之为学，不日进则日退，独学无友，则孤陋而难成；久处一方，则习染而不自觉。顾炎武 for:1m labels:level:p2 resources:container annotations:value:$value summary:Kubernetes Pod CPU使率过 description:Kubernetes Pod节点【$labels.pod】CPU使率过，请检查 -alert:CPU使率 expr:sum(rate(container_cpu_usage_seconds_totalcontainer!=,container!=POD2m)by(pod,namespace,instance,container)/(sum(container_spec_cpu_quotacontainer!=,container!=POD/100000)by(pod,namespace,instance,container)*100 90 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod CPU使率过 description:Kubernetes Pod节点【$labels.pod】CPU使率过，请检查 -alert:内存使率 expr:sum(container_memory_rsscontainer!=,container!=POD)by(pod,namespace,instance,container)/sum(container_spec_memory_limit_bytescontainer!=,container!=POD)by(pod,namespace,instance,container)*100 80 for:1m labels:level:p4 resources:container annotations:value:$value summary:Kubernetes Pod 内存使率过丹青不知老将至，贫贱于我如浮云。杜甫吾日三省乎吾身。为人谋而不忠乎？与朋友交而不信乎？传不习乎？论语 description:Kubernetes Pod节点【$labels.pod】内存使率过，请检查 -alert:内存使率 expr:sum(container_memory_rsscontainer!=,container!=POD)by(pod,namespace,instance,container)/sum(container_spec_memory_limit_bytescontainer!=,container!=POD)by(pod,namespace,instance,container)*100 85 for:1m labels:level:p3 resources:container annotations:value:$value summary:Kubernetes Pod 内存使率过 description:Kubernetes Pod节点【$labels.pod】内存使率过，请检查 -alert:内存使率 expr:sum(container_memory_rsscontainer!=,container!=POD)by(pod,namespace,instance,container)/sum(container_spec_memory_limit_bytescontainer!=,container!=POD)by(pod,namespace,instance,container)*100 90 for:1m labels:level:p2 resources:container annotations:value:$value summary:Kubernetes Pod 内存使率过 description:Kubernetes Pod节点【$labels.pod】内存使率过，请检查 -alert:内存使率 expr:sum(container_memory_rsscontainer!=,container!=POD)by(pod,namespace,instance,container)/sum(container_spec_memory_limit_bytescontainer!=,container!=POD)by(pod,namespace,instance,container)*100 95 for:1m labels:level:p1 resources:container谋事在人，成事在天！增广贤文忍一句，息一怒，饶一着，退一步。增广贤文 annotations:value:$value summary:Kubernetes Pod 内存使率过 description:Kubernetes Pod节点【$labels.pod】内存使率过，请检查 -alert:磁盘I/O使 expr:sum(rate(container_fs_reads_bytes_total5m)by(pod,namespace,instance,container)10485760 for:1m labels:level:p4 resources:container annotations:value:$value summary:Kubernetes Pod 磁盘IO使过 description:Kubernetes Pod节点【$labels.pod】磁盘IO使率过，每秒达到10MB请检查 -alert:磁盘I/O使 expr:sum(rate(container_fs_reads_bytes_total5m)by(pod,namespace,instance,container)20971520 for:1m labels:level:p3 resources:container annotations:value:$value summary:Kubernetes Pod 磁盘IO使过 description:Kubernetes Pod节点【$labels.pod】磁盘IO使率过，每秒达到20MB请检查 -alert:磁盘I/O使 expr:sum(rate(container_fs_reads_bytes_total5m)by(pod,namespace,instance,container)31457280 for:1m labels:level:p2良辰美景奈何天，便赏心乐事谁家院。则为你如花美眷，似水流年。汤显祖百川东到海，何时复西归？少壮不尽力，老大徒伤悲。汉乐府长歌行 resources:container annotations:value:$value summary:Kubernetes Pod 磁盘IO使过 description:Kubernetes Pod节点【$labels.pod】磁盘IO使率过，每秒达到30MB请检查 -alert:磁盘I/O使 expr:sum(rate(container_fs_reads_bytes_total5m)by(pod,namespace,instance,container)41943040 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod 磁盘IO使过 description:Kubernetes Pod节点【$labels.pod】磁盘IO使率过，每秒达到40MB请检查 -alert:络I/O使 expr:sum(rate(container_network_receive_bytes_total5m)by(pod,namespace,instance,container)20971520 for:1m labels:level:p4 resources:container annotations:value:$value summary:Kubernetes Pod 络IO使过 description:Kubernetes Pod节点【$labels.pod】络IO使率过，每秒达到20MB请检查 -alert:络I/O使 expr:sum(rate(container_network_receive_bytes_total5m)by(pod,namespace,instance,container)31457280 for:1m labels:我尽一杯，与君发三愿：一愿世清平，二愿身强健，三愿临老头，数与君相见。白居易百学须先立志。朱熹 level:p3 resources:container annotations:value:$value summary:Kubernetes Pod 络IO使过 description:Kubernetes Pod节点【$labels.pod】络IO使率过，每秒达到30MB请检查 -alert:络I/O使 expr:sum(rate(container_network_receive_bytes_total5m)by(pod,namespace,instance,container)41943040 for:1m labels:level:p2 resources:container annotations:value:$value summary:Kubernetes Pod 络IO使过 description:Kubernetes Pod节点【$labels.pod】络IO使率过，每秒达到40MB请检查 -alert:络I/O使 expr:sum(rate(container_network_receive_bytes_total5m)by(pod,namespace,instance,container)51943040 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod 络IO使过 description:Kubernetes Pod节点【$labels.pod】络IO使率过，每秒达到50MB请检查apiVersion: role:alert-rules name:kube-slos namespace:kubesphere-monitoring-systemspec:groups:-name:接错误率 rules:-alert:KubeAPI错误预算 expr:sum(apiserver_request:burnrate3d)(1*0.01)and sum(apiserver_request:burnrate6h)(1*0.01)for:1h labels:level:p4 resources:container annotations:value:$value summary:Kubernetes API错误预算 description:Kubernetes API错误预算较多,请检查 -alert:KubeAPI错误预算 expr:sum(apiserver_request:burnrate1d)(3*0.01)and sum(apiserver_request:burnrate2h)(3*0.01)for:1h labels:level:p3 resources:container谋事在人，成事在天！增广贤文穷则独善其身，达则兼善天下。孟子 annotations:value:$value summary:Kubernetes API错误预算 description:Kubernetes API错误预算较多,请检查 -alert:KubeAPI错误预算 expr:sum(apiserver_request:burnrate6h)(6*0.01)and sum(apiserver_request:burnrate30m)(6*0.01)for:15m labels:level:p2 resources:container annotations:value:$value summary:Kubernetes API错误预算 description:Kubernetes API错误预算较多,请检查 -alert:KubeAPI错误预算 expr:sum(apiserver_request:burnrate1h)(14.4*0.01)and sum(apiserver_request:burnrate5m)(14.4*0.01)for:2m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes API错误预算 description:Kubernetes API错误预算较多,请检查 -alert:KubeStateMetricsList错误率 expr:(sum(rate(kube_state_metrics_list_totaljob=kube-state-metrics,result=error5m)/sum(rate(kube_state_metrics_list_totaljob=kube-state-metrics5m)0.01 for:2m labels:level:p1天行健，君子以自强不息。地势坤，君子以厚德载物。易经大丈夫处世，不能立功建业，几与草木同腐乎？罗贯中 resources:container annotations:value:$value summary:Kubernetes kube-state-metrics List错误率 description:Kubernetes kube-state-metrics List错误率较,请检查 -alert:KubeStateMetricsWatch错误率 expr:(sum(rate(kube_state_metrics_watch_totaljob=kube-state-metrics,result=error5m)/sum(rate(kube_state_metrics_watch_totaljob=kube-state-metrics5m)0.01 for:2m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes kube-state-metrics Watch错误率 description:Kubernetes kube-state-metrics Watch错误率较,请检查apiVersion: role:alert-rules name:container-status namespace:kubesphere-monitoring-systemspec:古之立大事者，不惟有超世之才，亦必有坚忍不拔之志。苏轼非淡泊无以明志，非宁静无以致远。诸葛亮 groups:-name:容器状态 rules:-alert:Pod异常重启 expr:rate(kube_pod_container_status_restarts_totaljob=kube-state-metrics15m)*60*5 0 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod 异常重启 description:Kubernetes Pod节点【$labels.pod】异常重启,请检查 -alert:Pod处于准备阶段 expr:sum by(pod,namespace,instance,container)(max by(pod,namespace,instance,container)(kube_pod_status_phasejob=kube-state-metrics,phase=Pending|Unknown)*on(pod,namespace,instance,container)group_left(owner_kind)max by(pod,namespace,instance,container,owner_kind)(kube_pod_ownerowner_kind!=Job)0 for:2m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod 处于Ready阶段 description:Kubernetes Pod节点【$labels.pod】启动异常，长时间处于Ready阶段,请检查 -alert:Pod处于等待状态 expr:sum by(namespace,pod,container,instance)(kube_pod_container_status_waiting_reasonjob=kube-state-metrics)0 for:2m labels:老当益壮，宁移白首之心；穷且益坚，不坠青云之志。唐王勃勿以恶小而为之，勿以善小而不为。刘备 level:p1 resources:container annotations:value:$value summary:Kubernetes Pod 处于waiting状态 description:Kubernetes Pod节点【$labels.pod】状态异常，长时间处于waiting状态,请检查 -alert:Deployment部署状态不匹配 expr:kube_deployment_status_observed_generationjob=kube-state-metrics!=kube_deployment_metadata_generationjob=kube-state-metrics for:5m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Deployment部署失败 description:Kubernetes【$labels.namespace$labels.deployment】部署状态异常，这表部署失败，但尚未回滚,请检查 -alert:Deployment副本数不匹配 expr:(kube_deployment_spec_replicasjob=kube-state-metrics!=kube_deployment_status_replicas_availablejob=kube-state-metrics)and(changes(kube_deployment_status_replicas_updatedjob=kube-state-metrics5m)=0)for:5m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Deployment期望数量不匹配 description:Kubernetes【$labels.namespace$labels.deployment】与预期的副本数不匹配,请检查 -alert:Deployment Pod扩缩容常将有日思无日，莫待无时思有时。增广贤文丹青不知老将至，贫贱于我如浮云。杜甫 expr:changes(kube_deployment_status_replicas_available2m)0 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Pod进了扩缩容 description:Kubernetes【$labels.namespace$labels.deployment】Pod进了扩缩容,请检查 -alert:Deployment Pod异常 expr:kube_deployment_status_replicas_unavailable 0 for:1m labels:level:p1 resources:container annotations:value:$value summary:Kubernetes Deployment 存在不可Pod description:Kubernetes【$labels.namespace$labels.deployment】存在不可Pod,请检查
