顾静, 阿里云 邓隽, 阿里云 Kubernetes 异常配置检测框架 我们来自阿里云容器服务 • 顾静，研发工程师 • 邓隽，技术专家 我们参与打造 • 容器服务（ACK/ASK） • 容器镜像服务（ACR） • 服务网格（ASM） • … 1 Kubernetes 典型异常 2 检测框架演进 3 生产实践 4 总结 Kubernetes 使用日常 • 应用部署 • Kernel 仅能检测 Kernel 相关问题 要求熟悉 bpftrace 语言 1 Kubernetes 典型异常 2 检测框架演进 3 生产实践 4 总结 我们的目标 实现 Kubernetes 集群异常检测框架 支持集群多维度异常检测能力 支持集成开源检测组件 检测框架 Ver.1 { 自动化 Ver.1 自动化 Autopilot Engine Command Policy 边缘集群、GPU 集群 各类检测场景 • 节点、组件、配置等 • 集群升级、集群巡检 版本差异 * 类型差异 * 场景差异 检测项 Ver.1 优化点 检测代码在膨胀 检测能力迭代需要加速 检测框架 Ver.2 { 动态定制 动态扩展 Ver.2 DSL DSL (Domain-Specific Language) • 领域特定语言指专注于某个应用程序领域的计算机语言 • 目标受众为非程序员、业务员或最终客户

云原生监控体系建设 秦晓辉 快猫星云 联合创始人 个人介绍 秦晓辉，常用网名龙渊秦五、UlricQin，山东人，12年 毕业自山东大学，10年经验一直是在运维研发相关方向， 是Open-Falcon、Nightingale、Categraf 等开源软件 的核心研发，快猫星云联合创始人，当前在创业，为客 户提供稳定性保障相关的产品 个人主页：https://ulricqin.github.io/ apiserver 的耗时分布，histogram类型，按照 url + verb 统计 • scheduler_framework_extension_point_duration_s econds 调度框架的扩展点延迟分布，按 extension_point 统计 • scheduler_pending_pods 调度 pending 的 pod 数量，按照 queue type 分别统计 • sch • statsd 和 prometheus sdk 都是通用方案，此外，不同语言也会有一些习惯性使用的埋点方案，比如 Java 的 micrometer • 埋点方案尽量要全公司一套，规范统一，在代码框架层面内置，减轻各个研发团队的使用成本 Pod内的业务应用的监控 - statsd 数据流向 • 推荐做法：如果是容器环境，Pod 内 sidecar 的方式部署 statsd；如果是物理机虚拟机环境，每个机器上部署一

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 speeds of different birds. >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) (continues pandas: powerful Python data analysis toolkit, Release 0.24.0 (continued from previous page) >>> s2 falcon 345.0 eagle 200.0 duck 30.0 dtype: float64 Now, to combine the two datasets and view the highest

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because

'Carnivora', 58)], ...: index=['falcon', 'parrot', 'lion', 'monkey', 'leopard'], ...: columns=('class', 'order', 'max_speed')) ...: In [2]: df Out[2]: class order max_speed falcon bird Falconiformes 389.0 >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 highest speeds of the birds across the two datasets >>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64 In the previous example, the resulting value for duck is missing, because