Go runtime related problems in TiDB production environment About me ● Arthur Mao(毛康力), Senior Engineer@PingCAP ● TiDB core developer (top3 contributor) ● GitBook about golang internals (@tiancaiamao) IO is ready => goroutine wake up == 4.3ms ○ Sometime even 10ms+ latency here! ○ The time spend on runtime schedule is not negligible ● When CPU is overload, which goroutine should be given priority? Analysis longer to be scheduled ● The runtime scheduling does not consider priority ● CPU dense workload could affect IO latency Conclusion Part II - Memory control ● Go Runtime ○ Allocated from OS (mmaped)

浅谈全链路监控： 从应用到数据库到 Runtime 黄东旭, Co-founder & CTO, PingCAP 关于我 黄东旭，联合创始人 & CTO @ PingCAP 做分布式数据库的程序员 ● 现在能写代码的时间是奢侈品 TiDB 的亲爹之一兼首席客服和新功能的第一个用户 ● 冤有头债有主，SQL 慢了来找我。。。 偶尔玩玩音乐 ● 摇滚乐->实验音乐 Go 的粉丝！！！！ tool trace go tool trace ● 优点：好用，好看（UI） ● 缺点：性能损耗太大，不能一直开着 Trace in Go runtime ● go tool trace 的原理是？ Trace 会 Go Runtime 的代码中打桩收集 CPU time，在 Goroutine 开始执行时记录 start_run_time, 在调度退出执行时记录 end_run_time，累加 goroutine 的 CPU time。 A little bit about Go runtime https://learnku.com/articles/41728 https://github.com/golang/go/blob/ma ster/src/runtime/trace.go hack runtime 的思路： follow the tracing event. PingCAP

Tag Another approach to CPU resource bind in Go Goroutine CPU Stats ● Try to collect goroutine runtime information. begin := GetGoroutineStats()
 
 executeSQL()
 
 end := GetGoroutineStats()
 
 queryCost queryCost := end.Sub(begin) // 获取执⾏ SQL 消耗的 cpu stats 信息 Trace ● Use Go trace to collect runtime information. curl http://localhost:10080/debug/pprof/trace\?seconds\=1 --output trace.out go tool trace 20ms (10-0) + (60-50) Modify Go runtime to collect Goroutine stats type g struct {
 goid int64 // goroutine id.
 . . .
 stats GStats // goroutine runtime stats
 }
 
 type GStats struct {


#48755 @winoros • Fix the issue that queries containing common table expressions (CTEs) re- port runtime error: index out of range [32] with length 32 when tidb_max_chunk_size is set to a small value #48808 E E E E E Placement Rules in SQL Y Y Y Y Y E E N N N N Cascades Planner E E E E E E E E E E E Runtime Filter Y Y N N N N N N N N N 2.3.5 Data definition language (DDL) Data definition language (DDL) +0000, which allows the jobs to be scheduled at any time. 390 4.6.4.3 Observability TiDB collects runtime information about TTL periodically and provides visualized charts of these metrics in Grafana. You

Y Y E E E E E Placement Rules in SQL Y Y Y Y E E N N N N Cascades Planner E E E E E E E E E E Runtime Filter Y N N N N N N N N N 2.3.5 Data definition language (DDL) Data definition language (DDL) +0000, which allows the jobs to be scheduled at any time. 4.6.4.3 Observability TiDB collects runtime information about TTL periodically and provides visualized charts of these metrics in Grafana. You slower you write, and the more space it consumes. In addition, too many indexes affect optimizer runtime, and inappropriate indexes can mislead the optimizer. So, more indexes do not always mean better

the maximum length specified by max-index-length #55138 @lance6716 • Fix the issue that the error runtime error: index out of range might oc- cur when executing SQL statements with tidb_enable_inl_join_inner_multi that a correlated subquery that contains WITH ROLLUP might cause TiDB to panic and return the error runtime error: index out of range #54983 @AilinKid • Fix the issue that predicates cannot be pushed down columns and the execution condition contains UnionScan #54870 @qw4990 • Fix the issue that the error runtime error: invalid memory address or nil �→ pointer dereference might occur when executing SQL statements

expecta- tions for resource management. • Controlling the resource allocation of large queries at runtime to avoid exceeding the resource group limit, combined with runaway queries COOLDOWN. This can help E E E Placement Rules in SQL Y Y Y Y Y Y Y Y E E N N Cascades Planner E E E E E E E E E E E E Runtime Filter Y Y Y Y Y N N N N N N N 2.3.5 Data definition language (DDL) Data definition language (DDL) +0000, which allows the jobs to be scheduled at any time. 407 4.6.4.3 Observability TiDB collects runtime information about TTL periodically and provides visualized charts of these metrics in Grafana. You

expecta- tions for resource management. • Controlling the resource allocation of large queries at runtime to avoid exceeding the resource group limit, combined with runaway queries COOLDOWN. This can help E E E Placement Rules in SQL Y Y Y Y Y Y Y Y E E N N Cascades Planner E E E E E E E E E E E E Runtime Filter Y Y Y Y Y N N N N N N N 2.3.5 Data definition language (DDL) Data definition language (DDL) +0000, which allows the jobs to be scheduled at any time. 412 4.6.4.3 Observability TiDB collects runtime information about TTL periodically and provides visualized charts of these metrics in Grafana. You

this feature, set the system variable tidb_redact_log to MARKER, and then the SQL text in TiDB’s runtime logs is marked. In addition, you can use the collect-log subcommand on the TiDB server to remove Advanced SQL features 8.1 7.5 7.1 6.5 6.1 5.4 5.3 5.2 5.1 Cascades Planner E E E E E E E E E Runtime Filter Y Y N N N N N N N 2.3.5 Data definition language (DDL) Data definition language (DDL) 8 +0000, which allows the jobs to be scheduled at any time. 382 4.6.4.3 Observability TiDB collects runtime information about TTL periodically and provides visualized charts of these metrics in Grafana. You

Y Y Y Y E E E Placement Rules in SQL Y Y Y Y Y Y E E N N Cascades Planner E E E E E E E E E E Runtime Filter Y Y Y N N N N N N N 2.3.5 Data definition language (DDL) Data definition language (DDL) +0000, which allows the jobs to be scheduled at any time. 382 4.6.4.3 Observability TiDB collects runtime information about TTL periodically and provides visualized charts of these metrics in Grafana. You slower you write, and the more space it consumes. In addition, too many indexes affect optimizer runtime, and inappropriate indexes can mislead the optimizer. So, more indexes do not always mean better