Go in TiDB 申砾@PingCAP Agenda • About Me • What is TiDB • TiDB Architecture • SQL Layer Internal • Golang in TiDB • Next Step of TiDB About Me • Shen Li (申砾) • Tech Lead of TiDB, VP of Engineering Why Go? • Productivity • Concurrency • Great for network programming • GC • Standard libraries && Tools • Good performance • Quick improvement Go in TiDB • More than 100k lines of Go code and session is hard • Account for large memory allocation • Account for memory consuming operators New in Go1.8 • Better GC • Pauses 50μs • sort.Slice Next Step of TiDB • Switch to gRPC • Document support

Go Practices in TiDB 姚维 PingCAP wink@pingcap.com Agenda ● How to build a stable database ○ Schrodinger-test platform ○ Failpoint injection ○ Goroutine-leak detection ● Optimization ○ Chunk vs interface{} errors.New("mock commit error") // } …. } https://github.com/pingcap/tidb/blob/master/store/tikv/txn.go#L223 The generated gofail code func (txn *tikvTxn) Commit(ctx context.Context) error { if vmockCommitError or: __fp_mockCommitError.BadType(vmockCommitError, "bool"); }; …. } $ gofail enable store/tikv/txn.go Why we need a new failpoint ● Generated code is not readable ● Concurrent testing will use the same

No builds required to add new rules. BUSINESS PROBLEM Our Answer Go Language MongoDB Mgo/Beego If we had a system that could do inser*on. Two Go programs perform all the feed management work WHY GO - DYNAMIC FEED MANAGEMENT The Feed System Is Driven By Data Go supports Data from Offer and Internal Feeds DYNAMIC RULE MANAGEMENT Our Answer Go Language MongoDB Mgo/Beego Building rules is a core piece of the

Go runtime related problems in TiDB production environment About me ● Arthur Mao(毛康力), Senior Engineer@PingCAP ● TiDB core developer (top3 contributor) ● GitBook about golang internals (@tiancaiamao) consider priority ● CPU dense workload could affect IO latency Conclusion Part II - Memory control ● Go Runtime ○ Allocated from OS (mmaped) ○ Managed Memory ■ Should the memory be returned to the OS? ○ memory footprint is abnormal ● The memory available on this node is not too much Description ● The Go Runtime thinks it does not use much memory ● The OS does not release the memory (RSS is high) Investigate

Go � TiDB ����� mongodb lixia@pingcap.com @���_go About me ��� PingCAP �� 15 ���� PingCAP ���� TiDB ��������� review ������� 12 ������� go ���������� 13 ������������������������ ���� ���� ���@���_go Email: lixia@pingcap.com Agenda mongodb introduction gonzo with memory engine gonzo with TiKV engine TiDB and TiKV Transaction Q & A mongodb introduction mongodb introduction

stars 16,000+ commits Marquee adopters 100+ contributors 700+ Slack members History ● Choosing Go ● TCP, Syscalls, GC, Scheduler 2013 2015 2016 First Kubernetes Production launch Last Sunday.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Vitess Roadmap read traffic to individual cells, Vitess currently serves reads only from the local cell. Writes will go cross-cell when necessary, to wherever the master for that shard resides. Execution Plans Vitess

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Vitess Roadmap 6824 main.go:64] W0521 01:05:40.443180 reparent.go:185] master -elect tablet zone1 -0000000100 is not the shard master, proceeding anyway as -force was used 16 W0520 18:05:40.445230 6824 main.go:64] W0521

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Vitess Roadmap read traffic to individual cells, Vitess currently serves reads only from the local cell. Writes will go cross-cell when necessary, to wherever the master for that shard resides. Execution Plans Vitess

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Learning Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Google I/O 2014 - Scaling with Go: YouTube’s Vitess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Vitess Roadmap read traffic to individual cells, Vitess currently serves reads only from the local cell. Writes will go cross-cell when necessary, to wherever the primary for that shard resides. 22 Execution Plans Vitess