Kubernetes可观测性最佳实践

张

张建站

2026/5/30 12:39:34

10分钟阅读

Kubernetes可观测性最佳实践引言在云原生环境中可观测性Observability是确保系统稳定运行的关键。Kubernetes作为复杂的分布式系统需要全面的监控、日志和追踪体系来理解系统行为、定位问题和优化性能。本文将深入探讨Kubernetes可观测性的核心组件和最佳实践。可观测性三要素指标Metrics指标是数值化的数据用于描述系统在特定时间点的状态。常见的指标类型包括资源指标CPU、内存、磁盘、网络业务指标请求数、响应时间、错误率自定义指标应用特定的业务指标日志Logs日志是系统事件的记录用于追踪系统行为和调试问题。日志通常包括应用日志应用程序产生的业务日志系统日志容器运行时和操作系统日志审计日志安全相关的操作记录追踪Tracing追踪用于追踪请求在分布式系统中的完整路径帮助理解请求的执行流程和瓶颈。核心概念包括Span单个操作的时间记录Trace一组Span组成的完整请求路径采样控制追踪数据的采集比例可观测性架构┌─────────────────────────────────────────────────────────────────┐ │ Kubernetes Cluster │ ├─────────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Pod A │ │ Pod B │ │ Pod C │ │ │ │ (App │ │ (App │ │ (App │ │ │ │ Sidecar) │ │ Sidecar) │ │ Sidecar) │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ └─────────┼────────────────┼────────────────┼────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 数据收集层 │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Prometheus │ Fluentd │ Jaeger Agent │ │ │ │ - 指标采集 │ - 日志收集 │ - 追踪数据收集 │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 数据存储层 │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Prometheus │ │ Loki │ │ Jaeger │ │ │ │ 时间序列存储 │ │ 日志存储 │ │ 追踪存储 │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 可视化层 │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Grafana │ │ │ │ - 指标仪表盘 │ │ │ │ - 日志查询 │ │ │ │ - 追踪可视化 │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘Prometheus监控配置Prometheus部署apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: monitoring --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: [get, list, watch] - apiGroups: [] resources: - configmaps verbs: [get] - apiGroups: - networking.k8s.io resources: - ingresses verbs: [get, list, watch] - nonResourceURLs: [/metrics] verbs: [get] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: monitoring --- apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitoring spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: serviceAccountName: prometheus containers: - name: prometheus image: prom/prometheus:v2.47.0 args: - --config.file/etc/prometheus/prometheus.yml - --storage.tsdb.path/prometheus/ - --web.console.libraries/etc/prometheus/console_libraries - --web.console.templates/etc/prometheus/consoles - --storage.tsdb.retention.time200h - --web.enable-lifecycle ports: - containerPort: 9090 volumeMounts: - name: prometheus-config-volume mountPath: /etc/prometheus/ - name: prometheus-storage-volume mountPath: /prometheus/ volumes: - name: prometheus-config-volume configMap: name: prometheus-config - name: prometheus-storage-volume emptyDir: {}Prometheus配置apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: monitoring data: prometheus.yml: | global: scrape_interval: 15s evaluation_interval: 15s rule_files: - /etc/prometheus/rules/*.yml alerting: alertmanagers: - static_configs: - targets: - alertmanager:9093 scrape_configs: - job_name: kubernetes-apiservers kubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: kubernetes-nodes kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:])(?::\d)?;(\d) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name自定义指标采集apiVersion: apps/v1 kind: Deployment metadata: name: api-server spec: template: metadata: annotations: prometheus.io/scrape: true prometheus.io/port: 8080 prometheus.io/path: /metrics spec: containers: - name: api-server image: example/api-server:v1.0.0 ports: - containerPort: 8080日志管理Loki部署apiVersion: v1 kind: Service metadata: name: loki namespace: monitoring spec: ports: - port: 3100 name: http selector: app: loki --- apiVersion: apps/v1 kind: StatefulSet metadata: name: loki namespace: monitoring spec: serviceName: loki replicas: 1 selector: matchLabels: app: loki template: metadata: labels: app: loki spec: containers: - name: loki image: grafana/loki:2.9.0 args: - -config.file/etc/loki/loki.yml ports: - containerPort: 3100 volumeMounts: - name: config mountPath: /etc/loki - name: data mountPath: /tmp/loki volumes: - name: config configMap: name: loki-config - name: data emptyDir: {}Loki配置apiVersion: v1 kind: ConfigMap metadata: name: loki-config namespace: monitoring data: loki.yml: | auth_enabled: false server: http_listen_port: 3100 common: path_prefix: /tmp/loki storage: filesystem: chunks_directory: /tmp/loki/chunks rules_directory: /tmp/loki/rules replication_factor: 1 ring: instance_addr: 127.0.0.1 kvstore: store: inmemory schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h ruler: alertmanager_url: http://alertmanager:9093Fluentd配置apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: logging data: fluent.conf: | source type tail id in_tail_container_logs path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos read_from_head true parse type json time_format %Y-%m-%dT%H:%M:%S.%NZ /parse tag kubernetes.* /source filter kubernetes.** type kubernetes_metadata /filter match kubernetes.** type loki id out_loki url http://loki:3100 extra_labels {env: production} buffer type memory flush_mode interval flush_interval 5s /buffer /match分布式追踪Jaeger部署apiVersion: v1 kind: Service metadata: name: jaeger namespace: tracing spec: ports: - port: 16686 name: ui - port: 9411 name: zipkin - port: 14268 name: collector selector: app: jaeger --- apiVersion: apps/v1 kind: Deployment metadata: name: jaeger namespace: tracing spec: replicas: 1 selector: matchLabels: app: jaeger template: metadata: labels: app: jaeger spec: containers: - name: jaeger image: jaegertracing/all-in-one:1.49 args: - --collector.zipkin.host-port:9411 ports: - containerPort: 16686 - containerPort: 9411 - containerPort: 14268 - containerPort: 14250应用集成JaegerapiVersion: apps/v1 kind: Deployment metadata: name: api-service spec: template: metadata: labels: app: api-service spec: containers: - name: api-service image: example/api-service:v1.0.0 env: - name: JAEGER_AGENT_HOST value: jaeger-agent.tracing.svc.cluster.local - name: JAEGER_AGENT_PORT value: 6831 - name: JAEGER_SAMPLER_TYPE value: const - name: JAEGER_SAMPLER_PARAM value: 1Grafana可视化Grafana部署apiVersion: v1 kind: Service metadata: name: grafana namespace: monitoring spec: type: LoadBalancer ports: - port: 80 targetPort: 3000 selector: app: grafana --- apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: monitoring spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: containers: - name: grafana image: grafana/grafana:10.2.0 ports: - containerPort: 3000 volumeMounts: - name: grafana-config mountPath: /etc/grafana - name: grafana-storage mountPath: /var/lib/grafana volumes: - name: grafana-config configMap: name: grafana-config - name: grafana-storage emptyDir: {}Grafana配置apiVersion: v1 kind: ConfigMap metadata: name: grafana-config namespace: monitoring data: grafana.ini: | [server] http_port 3000 [database] type sqlite3 path /var/lib/grafana/grafana.db [auth.anonymous] enabled true org_role Admin数据源配置apiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources namespace: monitoring data: prometheus.yaml: | apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://prometheus:9090 access: proxy isDefault: true - name: Loki type: loki url: http://loki:3100 access: proxy - name: Jaeger type: jaeger url: http://jaeger:16686 access: proxy告警配置Alertmanager部署apiVersion: v1 kind: Service metadata: name: alertmanager namespace: monitoring spec: ports: - port: 9093 name: http selector: app: alertmanager --- apiVersion: apps/v1 kind: StatefulSet metadata: name: alertmanager namespace: monitoring spec: serviceName: alertmanager replicas: 1 selector: matchLabels: app: alertmanager template: metadata: labels: app: alertmanager spec: containers: - name: alertmanager image: prom/alertmanager:v0.25.0 args: - --config.file/etc/alertmanager/config.yml - --storage.path/alertmanager ports: - containerPort: 9093 volumeMounts: - name: config mountPath: /etc/alertmanager - name: data mountPath: /alertmanager volumes: - name: config configMap: name: alertmanager-config - name: data emptyDir: {}Alertmanager配置apiVersion: v1 kind: ConfigMap metadata: name: alertmanager-config namespace: monitoring data: config.yml: | global: resolve_timeout: 5m route: group_by: [alertname] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: slack receivers: - name: slack slack_configs: - api_url: https://hooks.slack.com/services/XXXX/XXXX/XXXX channel: #alerts send_resolved: true inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, dev, instance]Prometheus告警规则apiVersion: v1 kind: ConfigMap metadata: name: prometheus-rules namespace: monitoring data: alerting_rules.yml: | groups: - name: kubernetes.rules rules: - alert: HighCPUUsage expr: sum(rate(container_cpu_usage_seconds_total{namespace!kube-system}[5m])) / sum(kube_pod_resource_requests_cpu_cores{}) * 100 80 for: 5m labels: severity: warning annotations: summary: High CPU usage detected description: CPU usage is above 80% for 5 minutes - alert: HighMemoryUsage expr: sum(container_memory_usage_bytes{namespace!kube-system}) / sum(kube_pod_resource_requests_memory_bytes{}) * 100 85 for: 5m labels: severity: critical annotations: summary: High memory usage detected description: Memory usage is above 85% for 5 minutes - alert: PodDown expr: kube_pod_status_phase{phaseRunning} 0 for: 2m labels: severity: critical annotations: summary: Pod is down description: Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is not running - alert: ServiceUnavailable expr: kube_service_status_cluster_ip for: 1m labels: severity: warning annotations: summary: Service unavailable description: Service {{ $labels.service }} in namespace {{ $labels.namespace }} has no cluster IP可观测性最佳实践指标命名规范# 推荐的指标命名格式 # metric_type_object_action_unit # 示例 api_requests_total # 计数器API请求总数 api_request_duration_seconds # 直方图API请求持续时间 api_error_rate # 计量器API错误率日志结构化apiVersion: apps/v1 kind: Deployment metadata: name: api-service spec: template: spec: containers: - name: api-service image: example/api-service:v1.0.0 env: - name: LOG_FORMAT value: json args: - --log-formatjson采样策略apiVersion: v1 kind: ConfigMap metadata: name: tracing-config data: tracing.yaml: | sampling: # 基于概率的采样 probabilistic: 0.1 # 基于速率的采样 rateLimiting: maxTracesPerSecond: 100 # 基于QPS的采样 adaptive: maxSamplingRate: 1.0 minSamplingRate: 0.01监控仪表盘设计# 仪表盘配置示例 apiVersion: grafana.integreatly.org/v1beta1 kind: GrafanaDashboard metadata: name: kubernetes-dashboard spec: title: Kubernetes Cluster Dashboard json: | { id: null, title: Kubernetes Cluster Dashboard, tags: [kubernetes, cluster], style: dark, timezone: browser, refresh: 30s, panels: [...] }可观测性故障排除常见问题排查问题1指标采集失败# 检查Prometheus配置 kubectl get configmap prometheus-config -n monitoring -o yaml # 检查Pod注解 kubectl get pods -o jsonpath{.items[*].metadata.annotations} # 检查网络连通性 kubectl exec -it prometheus-0 -- curl http://target-pod:8080/metrics问题2日志不显示# 检查Fluentd配置 kubectl get configmap fluentd-config -n logging -o yaml # 检查Fluentd日志 kubectl logs fluentd-xyz # 检查Loki查询 kubectl exec -it loki-0 -- curl -G http://localhost:3100/loki/api/v1/query --data-urlencode query{appmyapp}问题3追踪数据缺失# 检查Jaeger配置 kubectl get deployment jaeger -n tracing -o yaml # 检查应用环境变量 kubectl exec -it api-pod -- env | grep JAEGER # 检查网络连通性 kubectl exec -it api-pod -- nc -zv jaeger-agent 6831总结本文深入探讨了Kubernetes可观测性的核心组件和最佳实践包括指标监控Prometheus部署、配置和自定义指标采集日志管理Loki和Fluentd配置分布式追踪Jaeger部署和应用集成可视化Grafana仪表盘配置告警系统Alertmanager和Prometheus规则配置最佳实践指标命名、日志结构化、采样策略通过构建完整的可观测性体系可以实现对Kubernetes集群的全面监控快速定位问题提升系统稳定性和可靠性。