/etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": nbiot-udp: address: ":9092/udp" providers: kubernetesCRD: "" log: filePath: "" level: error format: json accessLog: filePath: lvwq Org = iSurpass Bucket = openservice [Log] Filename = /opt/tools/iSurpass/openservice/log/openservice.log MaxSize = 1 MaxBackups = 30 MaxAge = 15

/data/all-log pod-path /data/all-log kafka-cluster elasticsearch-cluster logstash-cluster kafka-groupid=es topic applog logstash kafka-groupid=file file-storage mkdir -p /data/all- log/${namespaces}/${pod-name} app-log-path pod 1 worker node pod * app-log log* daemonsets filebeat 容器启动挂载宿主/data/all-log到容器/data/all-log 容器启动时创建namespace和pod-name 创建软连接到logs目录 daemonset在每一个k8s-node节点启动 挂载/data/all-log目录 实时扫描发现新目录日志

Job Job Job Job Pod Pod Pod Pod ElasticSearch ElasticSearch Logging Service agent to collecting log data ElasticSearch ElasticSearch Monitor/Alert Service CronJob Node Pod Node Pod Unified collecting log data ElasticSearch ElasticSearch Monitor/Alert Service DevOps Operator Cluster AutoScaler k8s API DevOps Service DevOps Service DevOps Manager Restful API realtime log history history log pull metric data ElasticSearch ElasticSearch Prometheus Push Gateway push metric data • Build task can also expose custom metric data • Ephemeral build task can push metric to gateway if

Semi- structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages IoT & Devices Camera �� Log Service / LogHub Real-time Data Stream DataV Log Service / Dashboard ��� Stream Processing SparkStreaming Function Compute Hadoop HIVE Big Data Analytics Batch Processing Pig PAI MaxCompute EMR Interactive Analytics DLA Log Service

ContainerCrashLoopBackOff, FailedPostStartHook, Unhealthy… Trace system Increase of SLO Data Collect Audit log Event The unhealthy node Monitoring Isolation Recover Degrade Data Analysis Failures/Machine problems automatically。 The infrastructure Log Event End User Storage Analysis Platform Trace Report Weakness The trace system Data Collect: Collect Audit log for the whole cluster. Data analysis: Analyze Auto Human experience Improve of strategy …… 1. Collect data from metrics NPD, Trace system and Log. 2. Analyze the problem of the node, such as DiskRO, critical Daemonset is not ready. 3. Processing

/var/lib/chrony/dri� makestep 1.0 3 rtcsync keyfile /etc/chrony.keys leapsectz right/UTC logdir /var/log/chrony EOF # �medatectl set-�mezone Asia/Shanghai #设置时区 # systemctl restart chronyd # #pod容器要使用的存储卷声明 - name: alog #根据此名称去寻找对应的volume存储卷 mountPath: /var/log/nginx #容器里的挂载点 readOnly: false #非只读（即可读写） volumes: containers: - name: nginx-v1-19-5 #xxx省略其他配置项 volumeMounts: - name: alog mountPath: /var/log/nginx volumes: - name: alog nfs: server: 10.99.1.71 # nfs server ip/域名

logspout K8S DaemonSet stdout stderr K8S Pod LogMinion Kibana Kafka Cluster Log Parser ES Cluster log tail log tail stdout stderr Console 21 2017 Lenovo Internal. All rights reserved. Showcase

image: gcr.io/google_containers/testapp:v1 volumeMounts: - name: varlog mountPath: /var/log - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog

Context Unstructured Data Logs Messages VMware vRealize Log Insight Log analytics, aggregation, and search Virtual Applications vRealize Ops, Log Insight For Comprehensive Visibility 32 UI and API

describe svc' alias dimg='docker images' alias dps='docker ps|grep -v gcr' alias mtx_log='tailf /opt/matrix/logs/application.log |grep Call' alias etcd-health='/opt/bin/etcdctl cluster-health' alias etcd-ls='/opt/bin/etcdctl 48 48 K8s技术在H3Cloud OS中的应用介绍 如果一个Pod已经Running，但是里面的服务不正常，可以通过kubectl logs 来查看这个Pod里面容器的 log日志。 49 www.h3c.com Confidential 秘密 49 49 新华三集团 www.h3c.com Thanks！