背景与挑战 多Beats/Logstash接入管控 多ES搜索编排系统 日志AIOps探索 ## 背景与挑战 如何降低日志接入门槛 如何保证日志实时上报 如何保障日志采集不影响业务 如何做配置标准化 如何帮助业务快速排障 如何提供方便便捷的性能分析调优能力 100+ 产品数量 ## 1000 人员规模 10000 主机规模 ## 多Beats/Logstash接入 管控 提供多产 从Consul中获取当前agent的配置组列表，并启动多个采集进程 ## 配置变更感知 watch到Consul对应的agent id路径，实时感知配置变化，并对启动的进程列表做重启清理等工作 管理多Beats/logstash Beats等以agent子进程启动其管理这些进程的 cpu/内存等资源 ![Image](/uploads/documents/2/c/4/c/2c4c78721fc7921f5fa9b1e7bb808ae9/p8_1 8、Ng-zorro 后台 Golang、Iris、Consul、微服务化设计、Pongo2模板管理 配置 Consul、Elasticsearch Agent Golang、ELK beats、Logstash、LXC资源管控、 Consul、其他自定义插件 TypeScript ![Image](/uploads/documents/2/c/4/c/2c4c78721fc7921f5fa9b1e7bb808ae9/p22_1

interface) can be used to fetch log data from containers. This data can be fed to an ELK (Elasticsearch, Logstash and Kibana) stack or Google Cloud logging for further analysis and visualization. ### 1.4 Kubernetes not cover those here. For analysis and visualization, we will use the famous ELK (Elasticsearch, Logstash, Kibana) stack. To collect and forward the logs to the logging platform, we will use LogSpout (though they can be scaled easily (there are templates and detailed documentation for Kubernetes here). For Logstash and Kibana, we will use the images and templates developed by Paulo here with some minor modifications

接入第三方 BI 工具 152 一、UDW 接入 Zeppelin 152 二、UDW 接入 SuperSet 163 UDW 使用案例 177 案例一 利用 logstash+Kafka+UDW 对日志数据分析 177 案例二 基于UDW实现网络流分析 183 PXF 扩展 190 配置 PXF 服务 190 创建 EXTENSION 191 案例一 利用 logstash+Kafka+UDW 对日志数据分析 Logstash 是目前比流行、使用较多的日志收集和管理系统，Kafka也是企业常用的分布式发布-订阅消息系统，UDW (UCloud Data Warehouse) 是大规模并行处理数据仓库产品，下面介绍一些利用 logstash+Kafka+UDW 构建日志收集-存储-分析的全套解决方案。 ## Logstash收集日志到Kafka Logstash收集日志到Kafka 1. 下载安装https://www.elastic.co/downloads/logstash 2. logstash依赖java环境、确保已经安装过java 3. 安装logstash-output-kafka插件 4. 配置logstash收集日志写入Kafka 参考配置如下(更多参数和含义请参考官方文档): ![Image](/uploads/documents/2/e

转SQL）一般（类SQL查询统计等）数据治理数据加工无支持(logstash/reindex)支持（CQ/TickScript）生命周期不直接支持支持不直接支持 a42f7fb655acc3f12481e9fe38a68e6f/p31_1.jpg) ## Elastic Stack (BELK) • Beats + Elasticsearch + Logstash + Kibana • 接入层还会搭配Kafka - 重要企业级组件都在商业组件X-Pack中 - 安全、ML、SQL、监控、告警、Transform等 - 提供一个开源免费的APM方案 library APM Agent APM Server Elasticsearch Kibana APM UI ## Kibana ## Elasticsearch Beats Logstash ## Kafka + EBLK ☑ 引入队列，解决丢数据问题 部署、维护复杂度较为复杂 Collector's ![Image](/uploads/documents/a/4

rowspan="2">fluentdForwardfluentd forward v1fluentd 1.7.4logstash 7.10.1kafkakafka 0.11kafka 2.4.1logstash 7.10.1| |Loki|使用 HTTP 和 HTTPS 的 REST|OCP 和 Grafana 实验中部署的 Loki 2.3.0| |kafka|kafka 0.11|kafka 2 |elasticsearch|elasticsearch|Elasticsearch 7.10.1| |fluentdForward|fluentd forward v1|fluentd 1.7.4 logstash 7.10.1| |Loki|使用 HTTP 和 HTTPS 的 REST|在 OCP 上部署的 Loki 2.2.1| |kafka|kafka 0.11|kafka 2.7.0| |sys

elasticsearch=""> !include logstash="" logstash=""> !include ELASTICSEARCH(ElasticSearch, "Search and Analyze", database) LOGSTASH(Logstash, "Parse and Transform" Transform", node) KIBANA(Kibana, "Visualize", agent) Logstash -right-> ElasticSearch: Transformed Data ElasticSearch -right-> Kibana: Data to View @enduml kibana="" kibana.puml=""> !include !include logstash="" logstash.puml=""> !include !include

elasticsearch=""> !include logstash="" logstash=""> !include ELASTICSEARCH(ElasticSearch, "Search and Analyze", database) LOGSTASH(Logstash, "Parse and Transform" Transform", node) KIBANA(Kibana, "Visualize", agent) Logstash -right -> ElasticSearch: Transformed Data ElasticSearch -right -> Kibana: Data to View @enduml kibana="" kibana.puml=""> !include !include logstash="" logstash.puml=""> !include !include

elasticsearch=""> !include logstash="" logstash=""> !include ELASTICSEARCH(ElasticSearch, "Search and Analyze", database) LOGSTASH(Logstash, "Parse and Transform" Transform", node) KIBANA(Kibana, "Visualize", agent) Logstash -right-> ElasticSearch: Transformed Data ElasticSearch -right-> Kibana: Data to View @enduml kibana="" kibana.puml=""> !include !include logstash="" logstash.puml=""> !include !include

elasticsearch=""> !include logstash="" logstash=""> !include ELASTICSEARCH(ElasticSearch, "Search and Analyze", database) LOGSTASH(Logstash, "Parse and Transform" Transform", node) KIBANA(Kibana, "Visualize", agent) Logstash -right -> ElasticSearch: Transformed Data ElasticSearch -right -> Kibana: Data to View @enduml kibana="" kibana.puml=""> !include !include logstash="" logstash.puml=""> !include !include

elasticsearch=""> !include logstash="" logstash=""> !include ELASTICSEARCH(ElasticSearch, "Search and Analyze", database) LOGSTASH(Logstash, "Parse and Transform" Transform", node) KIBANA(Kibana, "Visualize", agent) Logstash -right -> ElasticSearch: Transformed Data ElasticSearch -right -> Kibana: Data to View @enduml kibana="" kibana.puml=""> !include !include logstash="" logstash.puml=""> !include !include