Namespace events 2. A user creates a new K8s Namespace 3. The K8s API Server notifies NCP of the change (addition) of Namespaces 4. NCP creates the network topology for the Namespace : a) Requests Context Unstructured Data Logs Messages VMware vRealize Log Insight Log analytics, aggregation, and search Virtual Applications vRealize Ops, Log Insight For Comprehensive Visibility 32 UI and API

API将会不断变化和发展。但是，在很⻓⼀段时间内并不会破坏与现有客户端的兼容性。⼀般来说，新的API资源和新的 资源字段通常可被频繁添加。消除资源或字段将需遵循 API deprecation policy 。 API change document 详细介绍了兼容更改以及如何更改API的内容。 OpenAPI与Swagger定义 完整的API详情使⽤ Swagger v1.2 和 OpenAPI 记录。Kubernetes kubectl rollout history deployment/nginx-deployment deployments "nginx-deployment" REVISION CHANGE-CAUSE 1 kubectl create -f docs/user-guide/nginx-deployment.yaml --record 2 Labels: app=nginx pod-template-hash=1159050644 Annotations: kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1 Containers: nginx:

/etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": nbiot-udp: address: ":9092/udp" providers: kubernetesCRD: "" log: filePath: "" level: error format: json accessLog: filePath: lvwq Org = iSurpass Bucket = openservice [Log] Filename = /opt/tools/iSurpass/openservice/log/openservice.log MaxSize = 1 MaxBackups = 30 MaxAge = 15

/data/all-log pod-path /data/all-log kafka-cluster elasticsearch-cluster logstash-cluster kafka-groupid=es topic applog logstash kafka-groupid=file file-storage mkdir -p /data/all- log/${namespaces}/${pod-name} app-log-path pod 1 worker node pod * app-log log* daemonsets filebeat 容器启动挂载宿主/data/all-log到容器/data/all-log 容器启动时创建namespace和pod-name 创建软连接到logs目录 daemonset在每一个k8s-node节点启动 挂载/data/all-log目录 实时扫描发现新目录日志

Job Job Job Job Pod Pod Pod Pod ElasticSearch ElasticSearch Logging Service agent to collecting log data ElasticSearch ElasticSearch Monitor/Alert Service CronJob Node Pod Node Pod Unified collecting log data ElasticSearch ElasticSearch Monitor/Alert Service DevOps Operator Cluster AutoScaler k8s API DevOps Service DevOps Service DevOps Manager Restful API realtime log history history log pull metric data ElasticSearch ElasticSearch Prometheus Push Gateway push metric data • Build task can also expose custom metric data • Ephemeral build task can push metric to gateway if

Semi- structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages IoT & Devices Camera �� Log Service / LogHub Real-time Data Stream DataV Log Service / Dashboard ��� Stream Processing SparkStreaming Function Compute Hadoop HIVE Big Data Analytics Batch Processing Pig PAI MaxCompute EMR Interactive Analytics DLA Log Service

ContainerCrashLoopBackOff, FailedPostStartHook, Unhealthy… Trace system Increase of SLO Data Collect Audit log Event The unhealthy node Monitoring Isolation Recover Degrade Data Analysis Failures/Machine problems automatically。 The infrastructure Log Event End User Storage Analysis Platform Trace Report Weakness The trace system Data Collect: Collect Audit log for the whole cluster. Data analysis: Analyze Auto Human experience Improve of strategy …… 1. Collect data from metrics NPD, Trace system and Log. 2. Analyze the problem of the node, such as DiskRO, critical Daemonset is not ready. 3. Processing

/var/lib/chrony/dri� makestep 1.0 3 rtcsync keyfile /etc/chrony.keys leapsectz right/UTC logdir /var/log/chrony EOF # �medatectl set-�mezone Asia/Shanghai #设置时区 # systemctl restart chronyd # #pod容器要使用的存储卷声明 - name: alog #根据此名称去寻找对应的volume存储卷 mountPath: /var/log/nginx #容器里的挂载点 readOnly: false #非只读（即可读写） volumes: containers: - name: nginx-v1-19-5 #xxx省略其他配置项 volumeMounts: - name: alog mountPath: /var/log/nginx volumes: - name: alog nfs: server: 10.99.1.71 # nfs server ip/域名

logspout K8S DaemonSet stdout stderr K8S Pod LogMinion Kibana Kafka Cluster Log Parser ES Cluster log tail log tail stdout stderr Console 21 2017 Lenovo Internal. All rights reserved. Showcase

image: gcr.io/google_containers/testapp:v1 volumeMounts: - name: varlog mountPath: /var/log - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog