Go runtime related problems in TiDB production environment About me ● Arthur Mao(毛康力), Senior Engineer@PingCAP ● TiDB core developer (top3 contributor) ● GitBook about golang internals (@tiancaiamao) case study: batching client requests ● Memory control ■ case study: transparent huge pages ● GC Related ■ case study: GC sweep caused latency jitter ■ case study: Lock and NUMA aware Agenda Part I - Compare with other nodes in the cluster with THP disabled Investigate ● So, the root cause must be related to THP (transparent huge pages) ● But … why? Analysis ● Go Runtime manage memory at 8K size granularity

Mass data real‐time analysis in OLAP scenarios . . . . . . . . . . . . . . . . . . . 23 8.1.5 Related References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 8.1.6 Core Concept . 36 Application Scenarios for ShardingSphere LOCAL Transaction . . . . . . . . . . 37 8.2.6 Related references . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 8.2.7 Core Concept 40 Complex primary‐secondary database architecture . . . . . . . . . . . . . . . . . 40 8.3.5 Related References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 8.3.6 Core Concept

Mass data real‐time analysis in OLAP scenarios . . . . . . . . . . . . . . . . . . . 23 8.1.5 Related References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 8.1.6 Core Concept . 36 Application Scenarios for ShardingSphere LOCAL Transaction . . . . . . . . . . 37 8.2.6 Related references . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 8.2.7 Core Concept 40 Complex primary‐secondary database architecture . . . . . . . . . . . . . . . . . 40 8.3.5 Related References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 8.3.6 Core Concept

Adding a background-image Writing your first Django app, part 7 Customize the admin form Adding related objects Customize the admin change list Customize the admin look and feel Customize the admin index own Models Model field reference Field attribute reference Model index reference Model _meta API Related objects reference Model class reference Model Meta options Model instance reference QuerySet API | QuerySet method reference | Lookup expressions Model instances: Instance methods | Accessing related objects Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing migrations

Adding a background-image Writing your first Django app, part 7 Customize the admin form Adding related objects Customize the admin change list Customize the admin look and feel Customize the admin index reference Field attribute reference Model index reference Constraints reference Model _meta API Related objects reference Model class reference Model Meta options Model instance reference QuerySet API | QuerySet method reference | Lookup expressions Model instances: Instance methods | Accessing related objects Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing migrations

QuerySet method reference | Lookup expressions • Model instances: Instance methods | Accessing related objects • Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing Article.objects.all() ]> # Article objects get API access to related Reporter objects. >>> r = a.reporter >>> r.full_name 'John Smith' # And vice versa: Reporter objects 0. Finally, note a relationship is defined, using ForeignKey. That tells Django each Choice is related to a single Question. Django supports all the common database relationships: many-to-one, many-to-many

Adding a background-image Writing your first Django app, part 7 Customize the admin form Adding related objects Customize the admin change list Customize the admin look and feel Customize the admin index own Models Model field reference Field attribute reference Model index reference Model _meta API Related objects reference Model class reference Model Meta options Model instance reference QuerySet API | QuerySet method reference | Lookup expressions Model instances: Instance methods | Accessing related objects Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing migrations

Adding a background-image Writing your first Django app, part 7 Customize the admin form Adding related objects Customize the admin change list Customize the admin look and feel Customize the admin index own Models Model field reference Field attribute reference Model index reference Model _meta API Related objects reference Model class reference Model Meta options Model instance reference QuerySet API | QuerySet method reference | Lookup expressions Model instances: Instance methods | Accessing related objects Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing migrations

QuerySet method reference | Lookup expressions • Model instances: Instance methods | Accessing related objects • Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing Article.objects.all() ]> # Article objects get API access to related Reporter objects. >>> r = a.reporter >>> r.full_name 'John Smith' # And vice versa: Reporter objects 0. Finally, note a relationship is defined, using ForeignKey. That tells Django each Choice is related to a single Question. Django supports all the common database relationships: many-to-one, many-to-many

QuerySet method reference | Lookup expressions • Model instances: Instance methods | Accessing related objects • Migrations: Introduction to Migrations | Operations reference | SchemaEditor | Writing Article.objects.all() ]> # Article objects get API access to related Reporter objects. >>> r = a.reporter >>> r.full_name 'John Smith' # And vice versa: Reporter objects 0. Finally, note a relationship is defined, using ForeignKey. That tells Django each Choice is related to a single Question. Django supports all the common database relationships: many-to-one, many-to-many