## Apache ShardingSphere Shadow DB 及在 CyborgFlow 中的应用 侯阳 2022年1月15日 SphereEx 中间件研发工程师 Apache ShardingSphere Contributor 目前专注于 Shadow DB 及全链路压测相关的设计和研发  1. Apache ShardingSphere 整体的架构设计 2. Apache ShardingSphere Shadow DB 的架构设计与应用场景 3. CyborgFlow 中 Shadow DB 的数据路由能力  ## Shadow DB Scenes 全链路压测：流量规划，比如双 11，618 等 灰度发布：指定测试用户的体验版本发布 服务预热：预热数据隔离 对比测试：基于版本的对比测试  ## Shadow DB API ![

Badger: Fast Key-Value DB in Go Manish R Jain, Dgraph Labs Apr 14,2018 Gopher China, Shanghai Dgraph Labs Fast, Distributed graph database. Sparse data sets. Lots of relationships. https://dgraph Operations Set a key-value func set() error { fmt.Println("\nRunning SET") return db.Update(func(txn *badger.Txn) error { if err := txn.Set([]byte("foo"), []byte("bar")); }) } Get a key-value func get() error { fmt.Println("\nRunning GET") return db.View(func(txn *badger.Txn) error { item, err := txn.Get([]byte("foo")) // handle err

Layer ..... 16 L2 Feature Layer ..... 16 L3 Ecosystem Layer ..... 17 Features ..... 18 4.1 DB Compatibility ..... 18 4.1.1 Background ..... 18 4.1.2 Challenges ..... 18 4.1.3 Goal ..... 19 MySQL ..... 19 PostgreSQL ..... 20 SQLServer ..... 20 Oracle ..... 20 SQL92 ..... 20 4.1.5 DB Protocol ..... 20 4.2 Sharding ..... 20 4.2.1 Background ..... 20 Vertical Sharding ..... 21 4.8 Shadow DB ..... 63 4.8.1 Background ..... 63 4.8.2 Challenges ..... 63 4.8.3 Goal ..... 64 4.8.4 Core Concept ..... 64 Shadow DB Switch ..... 64 Production DB ..... 64 Shadow DB ....

Splitting ..... 38 3.4.7 Limitations ..... 39 Supported ..... 39 Not supported ..... 39 3.5 DB Gateway ..... 39 3.5.1 Background ..... 39 3.5.2 Challenges ..... 39 3.5.3 Goal ..... 39 3 Plain column ..... 46 3.8.7 Limitations ..... 46 3.8.8 Appendix with SQL operator ..... 46 3.9 Shadow ..... 46 3.9.1 Background ..... 46 3.9.2 Challenges ..... 47 3.9.3 Goal ..... 47 3.9.4 Application Production Database ..... 48 Shadow Database ..... 48 Shadow Algorithm ..... 48 3.9.7 Limitations ..... 48 Hint based shadow algorithm ..... 48 Column based shadow algorithm ..... 48 3.10 Observability

Splitting ..... 39 3.4.7 Limitations ..... 40 Supported ..... 40 Not supported ..... 40 3.5 DB Gateway ..... 40 3.5.1 Background ..... 40 3.5.2 Challenges ..... 40 3.5.3 Goal ..... 40 3 Plain column ..... 46 3.8.7 Limitations ..... 47 3.8.8 Appendix with SQL operator ..... 47 3.9 Shadow ..... 47 3.9.1 Background ..... 47 3.9.2 Challenges ..... 47 3.9.3 Goal ..... 48 3.9.4 Application Production Database ..... 48 Shadow Database ..... 49 Shadow Algorithm ..... 49 3.9.7 Limitations ..... 49 Hint based shadow algorithm ..... 49 Column based shadow algorithm ..... 49 3.10 Observability

database 41 Primary-Secondary synchronization 41 Load balancer policy 41 8.3.7 Limitations 41 8.4 DB Gateway 42 8.4.1 Background 42 8.4.2 Challenges 42 8.4.3 Goal 42 8.4.4 Application Scenarios .... 49 8.8.6 Core Concept ..... 50 Logic column ..... 50 8.8.7 Limitations ..... 50 8.9 Shadow ..... 50 8.9.1 Background ..... 50 8.9.2 Challenges ..... 50 8.9.3 Goal ..... 51 8.9.4 Application Production Database ..... 51 Shadow Database ..... 51 Shadow Algorithm ..... 51 8.9.7 Limitations ..... 52 Hint based shadow algorithm ..... 52 Column based shadow algorithm ..... 52 8.10 Observability

database 41 Primary-Secondary synchronization 41 Load balancer policy 41 8.3.7 Limitations 41 8.4 DB Gateway 42 8.4.1 Background 42 8.4.2 Challenges 42 8.4.3 Goal 42 8.4.4 Application Scenarios .... 49 8.8.6 Core Concept ..... 50 Logic column ..... 50 8.8.7 Limitations ..... 50 8.9 Shadow ..... 50 8.9.1 Background ..... 50 8.9.2 Challenges ..... 50 8.9.3 Goal ..... 51 8.9.4 Application Production Database ..... 51 Shadow Database ..... 51 Shadow Algorithm ..... 51 8.9.7 Limitations ..... 52 Hint based shadow algorithm ..... 52 Column based shadow algorithm ..... 52 8.10 Observability

Layer ..... 16 L2 Feature Layer ..... 16 L3 Ecosystem Layer ..... 17 Features ..... 18 4.1 DB Compatibility ..... 18 4.1.1 Background ..... 18 4.1.2 Challenges ..... 18 4.1.3 Goal ..... 19 openGauss ..... 19 PostgreSQL ..... 20 SQLServer ..... 21 Oracle ..... 21 SQL92 ..... 21 4.1.5 DB Protocol ..... 21 4.1.6 Feature Support ..... 21 MySQL ..... 21 PostgreSQL ..... 22 SQLServer Unsupported...59 4.9 Shadow DB...60 4.9.1 Background...60 4.9.2 Challenges...60 4.9.3 Goal...60 4.9.4 Core Concept...60 Production Database...60 Shadow Database...61 Shadow Algorithm...61 4

L1 Kernel Layer ..... 16 L2 Feature Layer ..... 16 L3 Ecosystem Layer ..... 17 Features 4.1 DB Compatibility ..... 18 4.1.1 Background ..... 18 4.1.2 Challenges ..... 18 4.1.3 Goal ..... 19 openGauss ..... 20 PostgreSQL ..... 20 SQLServer ..... 21 Oracle ..... 21 SQL92 ..... 21 4.1.5 DB Protocol ..... 21 4.1.6 Feature Support ..... 21 MySQL ..... 22 PostgreSQL ..... 22 SQLServer 4.9 Shadow DB ..... 62 4.9.1 Background ..... 62 4.9.2 Challenges ..... 62 4.9.3 Goal ..... 63 4.9.4 Core Concept ..... 63 Production Database ..... 63 Shadow Database ..... 63 Shadow Algorithm

指数数据表在每个数据源内呈现均匀分布的态势，例如： db0 t_order0 t_order1 db1 t_order0 t_order1 数据节点的配置如下： db0.t_order0, db0.t_order1, db1.t_order0, db1.t_order1 自定义分布 指数数据表呈现有特定规则的分布，例如： db0 t_order0 t_order1 db1 t_order2 t_order3 t_order3 t_order4 数据节点的配置如下： db0.t_order0, db0.t_order1, db1.t_order2, db1.t_order3, db1.t_order4 ## 分片 ## 分片键 用于将数据库（表）水平拆分的数据库字段。例：将订单表中的订单主键的尾数取模分片，则订单主键为分片字段。SQL 中如果无分片字段，将执行全路由，性能较差。除了对单分片字段的支持，Apache 名称、运行模式、数据源集合、规则集合以及属性配置组成。 # JDBC 逻辑库名称。在集群模式中，使用该参数来联通 ShardingSphere-JDBC 与 ShardingSphere-Proxy。 # 默认值：logic_db databaseName (?) : mode: dataSources: rules: - !FOO_XXX ... - !BAR_XXX ... props: key_1: value_1