非RowKey查询性能差 • Column Family限制(数目，Partition对齐) • Region资源消耗大，实例数目不能太多 • 无法保证服务质量 $ ^{*} $ – Split/Compaction等操作对集群性能影响极大 • 多租户隔离能力差 • 大内存(>100GB)管理差 ## Kudu的设计目标  ## 性能优化目标 • 读 vs. 写? – 读需要合并HFile，因此文件越少越好 – 写需要减少Compaction操作，因此文件越多越好 - 优化读或者写之一，而不是全部 • 顺序 vs. 随机? - 参考值——每个RegionServer吞吐率>20MB/s – 读吞吐率>3000ops/s 内存使用过大，造成GC频繁，服务Timeout – 每个Region的Memstore太小，磁盘flush频繁，HFile文件过多小文件 ## Major Compaction • HBase根据时间来计划执行Major Compaction – hbase.hregion.majorcompaction = 604800000 (缺省值为一周) – hbase.hregion.majorcompaction

Column Names 55 Managing and Accessing Data in Cassandra 55 About Writes in Cassandra 55 About Compaction 55 About Transactions and Concurrency Control 55 About Inserts and Updates 56 About Deletes commitlog_sync 72 commitlog_sync_period_in_ms 72 commitlog_total_space_in_mb 72 compaction_preheat_key_cache 72 compaction_throughput_mb_per_sec 72 concurrent_compactors 72 concurrent_reads 72 concurrent_writes at 73 in_memory_compaction_limit_in_mb 73 index_interval 73 memtable_flush_queue_size 73 memtable_flush_writers 73 memtable_total_space_in_mb 73 multithreaded_compaction 73 reduce_cache_capacity_to

|---|---|---|---| |TiKV configuration file|soft-→ pending→ -→ compaction→ -bytes-→ limit|Modified|The soft limit on the pending compaction bytes. The default value is changed from "64GB" to "192GB" requests, TiKV Write Rate Limiter smoothes the write traffic of TiKV background tasks such as GC and Compaction. The default value of TiKV background task write rate limiter is "0MB". It is recommended results when cloning shared delta index concurrently – Fix the potential panic that occurs when the Compaction Filter feature is enabled – Fix the issue that TiFlash cannot resolve the lock fallen back from

#10980 – Fix the issue of poor scan performance because MVCC Deletion versions are not dropped by compaction filter GC #11248 ## · PD – Fix the issue that PD incorrectly delete the peers with data and in per → sec- → ond gc. Whether → enable- → - able → compaction → - paction → filterer Configuration item Description gc. Whether → compaction → skip → filter → -clus- → skipper → -ver- → version → ber → nameof → }. files → level0LO → - that → filetrig- → - ger → num com- → - paction → compaction → - → trigger → {db- The → namemaxi- → }.{mum → cf num- → - ber → nameof → }. files

type|Description| |---|---|---|---| |TiKV configuration file|storage.|Newly added|When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism starts to reject some write requests The default value is “192GB”.| |TiKV configuration file|storage.|Newly added|When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism rejects all write requests and reports reduces the impact on the stability of foreground write. In specific, when the stress of RocksDB compaction accumulates, flow control is performed at the TiKV scheduler layer instead of the RocksDB layer

rowspan="7">当 KvDB 的 pending compaction bytes 达到该阈值时，流控机制开始拒绝部分写入请求并报错。默认值为“192GB”。control.soft-pending-compaction-byt rowspan="7">当 KvDB 的 pending compaction bytes 达到该阈值时，流控机制开始拒绝所有写入请求并报错。默认值为“1024GB”。control.hard-pending-compaction-by 引入了新的流控机制代替之前的 RocksDB write stall 流控机制。相比于 write stall 机制，新的流控机制通过以下改进减少了流控对前台写入稳定性的影响： - 当 RocksDB compaction 压力堆积时，通过在 TiKV scheduler 层进行流控而不是在 RocksDB 层进行流控，避免 RocksDB write stall 造成的 raftstore 卡顿并造成 Raft

accidentally delete valid data #17258 @hbisheng - Fix the issue that Ingestion picked level and Compaction Job Size(files) are displayed incorrectly in the TiKV dashboard in Grafana #15990 @Connor1996 Note that it is recommended to reserve 20% of storage space, because background tasks such as compaction and snapshot replication also consume a portion of the storage space. ## 6.3.4 Change configuration configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster

请求时因超时而导致 panic 的问题 #10852 - 修复因统计线程监控数据导致的内存泄漏 #11195 - 修复在某些平台获取 cgroup 信息导致 panic 的问题 #10980 - 修复 Compaction Filter GC 无法清除 MVCC Deletion 版本导致 scan 性能下降的问题 #11248 - 修复因超过副本配置数量而导致错误删除带有数据且处于 pending 状态的副本的问题 数| |gc.max-write-bytes-per-sec|一秒可写入RocksDE的最大字节数| |gc.enable-compaction-filter|是否使用com-paction filter| |配置项|简介| |---|---| |gc.compaction-filter-skip-version-check|是否跳过com-pactionfilter的集群版本检查（未re-lease）| max-background-jobs|RocksDB后台线程个数| |{db-name}.max-open-files|RocksDB可以打开的文件总数| |{db-name}.compaction-readahead-size|Compaction时候readahead的大小| |{db-name}.bytes-sync|异步同步的限速速率| |配置项|简介| |---|---| |{db-|WAL| |name}

to spill-dir, ensuring continuous operation of the system #17356 @LykxSassinator - Optimize the compaction trigger mechanism of RocksDB to accelerate disk space reclamation when handling a large number Note that it is recommended to reserve 20% of storage space, because background tasks such as compaction and snapshot replication also consume a portion of the storage space. ## 6.3.4 Change configuration configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster

to spill-dir, ensuring continuous operation of the system #17356 @LykxSassinator - Optimize the compaction trigger mechanism of RocksDB to accelerate disk space reclamation when handling a large number Note that it is recommended to reserve 20% of storage space, because background tasks such as compaction and snapshot replication also consume a portion of the storage space. ## 6.3.4 Change configuration regularly update the metadata. Because the time interval between the MVCC of etcd and PD's default compaction is one hour, the amount of PD storage that TiCDC uses is proportional to the amount of metadata