filesystem, as an environment variable, or via Kubernetes API call ● Operations with secrets are audit logged Master kube-apiserver etcd SECRET Kubernetes secrets: 1.7 EncryptionConfig ● Encrypt Rotation Isolation Node authorizer K8s audit logging In etcd, not in applications aescbc, aesgcm, or secretbox Additional KMS logs aescbc KEK only, depending on KMS KMS May be more tightly scoped Additional secret manager logs Depending on secret manager Depending on secret manager In external secret store Kubernetes secrets:

ContainerCrashLoopBackOff, FailedPostStartHook, Unhealthy… Trace system Increase of SLO Data Collect Audit log Event The unhealthy node Monitoring Isolation Recover Degrade Data Analysis Failures/Machine the cluster is healthy or there is something unexpected happened. Trace system： Collect and analyze logs in cluster. So we can known what happened about the cluster. Increase of SLO： Get the weakness of End User Storage Analysis Platform Trace Report Weakness The trace system Data Collect: Collect Audit log for the whole cluster. Data analysis: Analyze failure reason if pod is failed. Reason analysis:

��1�DaemonSet or Sidecar • �����DaemonSet • ���PASS���Sidecar DaemonSet Sidecar ��2��������� /app/data/logs Sidecar ������� ������ ����� ����� ��� ���� • �����40����40GB SSD • 5W�����2PB SSD ������� Semi- structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages IoT & Devices Camera �� Log Service / LogHub Real-time Data Stream MaxCompute EMR Interactive Analytics DLA Log Service / Analytics �� Flink Storm ����������� Audit Ingress Mesh Event HPA Kubernetes������ ��� ���� Mesh ���� Stdout �� Event … DaemonSet Sidecar

tested by Google ● Access to Container services on GCP such as Cloud Build, Container Registry, Audit Logging, and more. ● Integration with Istio, Knative, Marketplace Solutions ALPHA IN FALL Run your dashboards based on Prometheus + Grafana + EFK ● Ingest metrics and logs into Stackdriver without any instrumentation changes ● Aggregate logs from many clusters -- whether GKE or GKE On-Prem Logging and

 分隔敏感的工作負載 (Segregate sensitive workloads)  掃描容器映像 (Scan container images)  開啟稽核日誌 (Enable audit logging)  跟上最新的 Kubernetes版本 (Keep your Kubernetes version up to date) Kubernetes Security Best Trusted Image kubectl run Image Registry Image Scanning Image Signing Harbor Projects AUDIT LOGGING 如果沒有企業私有的映像倉庫而只用Internet上的映像，您真的知道裡面有什麼嗎? 只有經過簽章 的受信任映像 才能被部署 即時弱點掃描 並標示弱點， 可限制有弱點 映像無法存取

Affiliates. All rights reserved. Amazon Confidential Amazon EKS 服务路线图摘要 已发布 - Amazon EKS control plane logs - Support for public IP space in VPC - Amazon EKS: Deep Learning Benchmarking Utility - New Scaling group AZ1 Region AZ2 Auto Scaling group CloudWatch Logs Elasticsearc h Kiban a Fluentd DaemonSet Kubectl logs Elasticsearch (index), Fluentd (store), and Kibana (visualize) com/blogs/opensource/ centralized-container-logging-fluent-bit/ • 新增 AWS FluentBit 容器插件 • 优化成本. Route logs from Amazon EKS 和 Amazon ECS 集群的日志会直接发送到S3， 并且通过 Amazon Athena 进行 即席查询 • 开源工具 • 比 Fluentd效率更高，测试显示

exec –it -c /bin/bash 登录进某个Pod的某个容器 6.查看容器的日志 kubectl logs 查看某个pod的日志 kubectl logs –f -c 跟踪容器的日志，相当于tail –f 7. 在线修改pod的副本数量 kubectl scale svc' alias dimg='docker images' alias dps='docker ps|grep -v gcr' alias mtx_log='tailf /opt/matrix/logs/application.log |grep Call' alias etcd-health='/opt/bin/etcdctl cluster-health' alias etcd-ls='/opt/bin/etcdctl www.h3c.com Confidential 秘密 48 48 K8s技术在H3Cloud OS中的应用介绍 如果一个Pod已经Running，但是里面的服务不正常，可以通过kubectl logs 来查看这个Pod里面容器的 log日志。 49 www.h3c.com Confidential 秘密 49 49 新华三集团 www.h3c.com Thanks！

调度器的复杂调度逻辑，最优的调度效率。 • 支持多可用区、多规格调度。 • 支持批量调度。 • AZ亲和性和反亲和性。 • ECI Pod生命周期管理：直通ECI Pod（Pod状态查询、logs、exec、metrics链路），降低ECI管控压力。 Elastic Scalling ECI ASK-Scheduler K8S API Server • Pod（N） : Node（1） ECI关键技术选择 - 基于 Pod 的基本调度单位和标准、开放的API接口 ECI ASK ACK 云上k8s集群 线下k8s集群 Creae/Delete/Update/Describe/Logs/Exec/Metrics ECI关键技术选择 - 基于安全沙箱技术的容器运行时 ECI Elastic Container Instance Pod container agent Container

#查看镜像，默认查询k8s.io命名空间，且不可指定命名 空间 # crictl ps #查看容器，默认固定为k8s.io的命名空间 # crictl logs xxxx #查看容器日志，支持带-f参数 # crictl inspect xxx #查看镜像或容器信息 ★附、安装cri-dockerd 保存，退出 # kubectl apply -f kube-flannel.yml #应用flannel配置 如果pod启动失败了，查看日志 # kubectl logs kube-flannel-ds-24�m -n kube-system I0430 11:16:34.505952 1 main.go:518] Determining IP address

Volumes: emptyDir, NFS, SecretVolume, ConfigMapVolume • Secret, ConfigMap • ServiceAccount • Logs, Exec, Attach, Top • Scaling, HPA • Helm 面向Cloud Scale的架构设计 Etcd K8S API Server Viking