## 并行不悖 – OLAP 在互联网公司的实践与思考 ## 赵飞祥 一 数据仓库体系架构 二 Greenplum体系架构 三 Greenplum现状说明 四 Greenplum运维体系 五 Greenplum开发规范 六 Greenplum扩展规划 ## 业务数据与数据使用归类 时间维度：过去 - 现在 - 未来 （数据的生命周期） “现在”的数据 —— OLTP “过去”的数据 “过去”的数据 —— OLAP “未来”的数据 —— 趋势分析 ## 业务数据与数据特点 • 现在的数据 —— OLTP 实时，在线系统，客户使用 事务小，频率高，并发高 · 过去的数据 —— OLAP 非实时（T+1，或小时级），离线系统，分析决策 事务大，频率相对小，并发低 · 未来的数据 —— 趋势分析 非实时，离线+在线流系统，趋势分析 算法分析，持续计算 ## 数据仓库体系架构 数据仓库体系架构 ## OLAP场景举例 ## · 业务相关场景 ➢ 用户状态（注册数，活跃数，并发量，峰值） 金币状态 道具/物品状态 对账状态 活动反馈 ## • 架构相关场景 不同数据量，不同事务特点，不同查询需求 历史数据归档与冷热分离 实时与延时需求的权衡 ## 数据流转过程 • 1 业务数据的产生 —— OLTP • 2 业务数据的中转 —— ETL服务器 • 3 数据的存储和计算

8d7dc08dc63cce8614319473794444/p2_5.jpg) ## About Me Wei Wan, work at PingCAP, as the leader of OLAP Storage team. Over 11 years of experience in game, e-commerce, mobile apps, and database development provide users with a one-stop database solution that covers OLTP (Online Transactional Processing), OLAP (Online Analytical Processing), and HTAP services. ## Agenda 1. A typical user case 2. The challenges infrastructure adaptation The challenges to storage module on HTAP scenario ## I solation between OLTP and OLAP workloads • Isolation is difficult if we mix them in the same node • TP and AP scale separately

宋涛 • Ping++ DBA • 负责Ping++数据库相关工作、数据库自动化运维建设 ## TiDB在Ping++金融聚合支付业务的实践 • TiDB在Ping++的应用场景分析 - OLAP: saas服务下实时数仓的支撑 - HTAP：基于TiDB Docker的聚合支付私有化部署方案 - 关于TiDB的线上运维 - TiDB体系 - 业务零感知运维 ## Ping++原数据架构及瓶颈 d968e25d6311815ed87beb3fc1f49/p6_2.jpg) ## 场景二：聚合支付私有化部署方案 服务私有化部署的数据库要求： - 脱离云产品：DRDS • oltp+olap • 海量交易下的单表性能：分库分表? - 高可用：MHA? Innodb Cluster? • 监控、运维方式简单高效 ## 场景二：聚合支付私有化部署方案 ## 基于TiDB Docker的HTAP一站式方案 Docker的HTAP一站式方案 • 兼容mysql，应用零改动迁移 • 节点高可用 - 自动Region分片、分布式事务 • Online DDL • SQL优先级策略，安全的oltp+olap • Grafana监控 • Docker简易运维，水平扩展 ![Image](/uploads/documents/f/6/c/d/f6cd968e25d6311815ed87beb3fc1f49/p8_1

## ClickHouse在B站海量数据场景的落地实践 胡甫旺 哔哩哔哩OLAP平台  ## 目录 ClickHouse在B站 内核 日志 用户行为数据分析 Future Work Q&A  ## 基于ClickHouse的交互式OLAP技术架构 Applications 日志&Trace 平台 报表平台 标签圈人 埋点分析 广告DMP 内容定投 APM 内容分析 HDFS/Hive 离线接入 (R

ClickHouse • Monitoring ClickHouse • Optimizing funnel analysis • Wish list for ClickHouse ### Why OLAP in Ximalaya.com? • About Ximalaya.com • Top audio sharing platform in China • Huge volume of • Application/infrastructure monitoring • Start with ClickHouse in May 2018 ## Why we choose ClickHouse as the OLAP Solution? • Amazing fast • SIMD instructions not yet generated by production JVM • Linearly scalable (small) cluster • Configurable at runtime, no need to reboot ## • Free and open source ## 3 Major OLAP Business using ClickHouse • User behavior analysis “Magic Mirror” • Funnel / Path / Retention analysis

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' You can and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' You can and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

(MySQL 互换) ✓ Distributed Transactions (分散トランザクション) Cloud Native (クラウドネイティブ志向) ☑ Minimize ETL (OLTP と OLAP のサポート) High Availability (高可用性)  ✓ Distributed Transactions (分散トランザクション) Cloud Native (クラウドネイティブ志向) ☑ Minimize ETL (OLTP と OLAP のサポート) High Availability (高可用性) ## TiDB の特徴 (Horizontal Scalability)  ✓ Distributed Transactions (分散トランザクション) Cloud Native (クラウドネイティブ志向) Minimize ETL (OLTP と OLAP のサポート) High Availability (高可用性) ## TiDB の特徴 (MySQL Compatible Syntax) ![Image](/uploads/documen

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' You can and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

HTAP) 的开源分布式关系型数据库产品，具备水平扩容或者缩容、金融级高可用、实时 HTAP、云原生的分布式数据库、兼容 MySQL 5.7 协议和 MySQL 生态等重要特性，向用户提供一站式 OLTP、OLAP、HTAP 解决方案，适用于对高可用、一致性要求高、数据规模较大等应用场景。 目前 TiDB 项目在 GitHub 上已总计获得超过 25000 颗星，累计超过 1000 位开源贡献者，是国内基础架构领域的顶级开源项目。2020 交易几乎没有影响。提供和 TiDB 保持强一致的数据读取，是真正的内核级 HTAP 分布式混合负载数据处理平台。 这套系统可以很好地解决： • 行存储和列存储的取舍问题； OLTP 负载和 OLAP 负载的资源隔离问题； • 快速批量写与事务型写操作混合模式的问题； • Adhoc 查询与 Adhoc 混合负载及批处理作业共存的问题； • 数据 Offload 到数据仓库引起的不一致风险。 引擎集群调度器2分析引擎TiSpark OLAP 分析引擎✓✓TiFlash 分布式列式存储引擎

0 码力 | 58 页 | 9.51 MB | 2 年前

3