applicated • Mobile backends • Continuous Integration Pipeline • Data Analytics • … Asynchronous Event Driven Stateless High dynamic Short Duration 14 - 一键部署Spark集群 - 无需管理服务器节点

Server 中数据的一份快照，并记 录 ResourceVersion 版本信息，watch 从 ResourceVersion 开始，获取后 续的增量数据。 watch 通过网络异步（asynchronous）获取增量数据，所以 cache 提供 的是最终一致性（eventual consistency）。 期间遇到网络、API Server 报错等异常时，会有重试机制 Controller-runtime

/etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": nbiot-udp: address: ":9092/udp" providers: kubernetesCRD: "" log: filePath: "" level: error format: json accessLog: filePath: lvwq Org = iSurpass Bucket = openservice [Log] Filename = /opt/tools/iSurpass/openservice/log/openservice.log MaxSize = 1 MaxBackups = 30 MaxAge = 15

/data/all-log pod-path /data/all-log kafka-cluster elasticsearch-cluster logstash-cluster kafka-groupid=es topic applog logstash kafka-groupid=file file-storage mkdir -p /data/all- log/${namespaces}/${pod-name} app-log-path pod 1 worker node pod * app-log log* daemonsets filebeat 容器启动挂载宿主/data/all-log到容器/data/all-log 容器启动时创建namespace和pod-name 创建软连接到logs目录 daemonset在每一个k8s-node节点启动 挂载/data/all-log目录 实时扫描发现新目录日志

Job Job Job Job Pod Pod Pod Pod ElasticSearch ElasticSearch Logging Service agent to collecting log data ElasticSearch ElasticSearch Monitor/Alert Service CronJob Node Pod Node Pod Unified collecting log data ElasticSearch ElasticSearch Monitor/Alert Service DevOps Operator Cluster AutoScaler k8s API DevOps Service DevOps Service DevOps Manager Restful API realtime log history history log pull metric data ElasticSearch ElasticSearch Prometheus Push Gateway push metric data • Build task can also expose custom metric data • Ephemeral build task can push metric to gateway if

Semi- structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages IoT & Devices Camera �� Log Service / LogHub Real-time Data Stream DataV Log Service / Dashboard ��� Stream Processing SparkStreaming Function Compute Hadoop HIVE Big Data Analytics Batch Processing Pig PAI MaxCompute EMR Interactive Analytics DLA Log Service

ContainerCrashLoopBackOff, FailedPostStartHook, Unhealthy… Trace system Increase of SLO Data Collect Audit log Event The unhealthy node Monitoring Isolation Recover Degrade Data Analysis Failures/Machine problems automatically。 The infrastructure Log Event End User Storage Analysis Platform Trace Report Weakness The trace system Data Collect: Collect Audit log for the whole cluster. Data analysis: Analyze Auto Human experience Improve of strategy …… 1. Collect data from metrics NPD, Trace system and Log. 2. Analyze the problem of the node, such as DiskRO, critical Daemonset is not ready. 3. Processing

/var/lib/chrony/dri� makestep 1.0 3 rtcsync keyfile /etc/chrony.keys leapsectz right/UTC logdir /var/log/chrony EOF # �medatectl set-�mezone Asia/Shanghai #设置时区 # systemctl restart chronyd # #pod容器要使用的存储卷声明 - name: alog #根据此名称去寻找对应的volume存储卷 mountPath: /var/log/nginx #容器里的挂载点 readOnly: false #非只读（即可读写） volumes: containers: - name: nginx-v1-19-5 #xxx省略其他配置项 volumeMounts: - name: alog mountPath: /var/log/nginx volumes: - name: alog nfs: server: 10.99.1.71 # nfs server ip/域名

logspout K8S DaemonSet stdout stderr K8S Pod LogMinion Kibana Kafka Cluster Log Parser ES Cluster log tail log tail stdout stderr Console 21 2017 Lenovo Internal. All rights reserved. Showcase

image: gcr.io/google_containers/testapp:v1 volumeMounts: - name: varlog mountPath: /var/log - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog