Go GC: Latency Problem Solved Rick Hudson Google Engineer GopherCon Denver July 8, 2015 Google Confidential and Proprietary My Codefendants: The Cambridge Runtime Gang https://upload.wikimedia.o Software++ Hardware++ Hardware++ HW++ HW++ Software++ Software++ Software++ #1 Barrier: GC Latency Google Confidential and Proprietary When is the best time to do a GC? When nobody is looking. Google Confidential and Proprietary Or Trade Throughput for Reduced GC Latency Google Confidential and Proprietary Latency Nanosecond 1: Grace Hopper Nanosecond 11.8 inches Microsecond 5

Michael A Maniscalco ○ Software Architect and Principal Developer at Lime Trading ○ We develop low latency market data and trading software for use in HFT ● Personal ○ github.com/buildingcpp ■ Work Contracts

Options Automated Trading Systems team at Optiver, a global market maker company - Worked on low-latency systems for 15 years in Trading and DefenseIntroduction 3 • About me - Leading the Options Automated Optiver, a global market maker company - Worked on low-latency systems for 15 years in Trading and Defense • Today’s talk - Engineering low-latency trading systems - Profiling techniques to find bottlenecks related to being fastBeing fast to get the trades – and to be accurate! 11 Ultra low-latency execution (< 1μs) Low-latency strategy layer (1-10 μs)Order book deep dive “The order book is the heart of any

环境 境变 变量 量 描述 描述 LATENCY_TEST_DE LAY 指定测试开始运行的时间（以秒为单位）。您可以使用变量来允许 CPU 管理器协 调循环来更新默认的 CPU 池。默认值为 0。 LATENCY_TEST_CP US 指定运行延迟测试的 pod 使用的 CPU 数量。如果没有设置变量，则默认配置包含 所有隔离的 CPU。 LATENCY_TEST_RU NTIME 指 HWLATDETECT_MA XIMUM_LATENCY 指定工作负载和操作系统的最大可接受硬件延迟（微秒）。如果您没有设置 HWLATDETECT_MAXIMUM_LATENCY 或 MAXIMUM_LATENCY 的值， 该工具会比较默认预期阈值(20μs)和工具本身中实际的最大延迟。然后，测试会失 败或成功。 CYCLICTEST_MAXI MUM_LATENCY 指定 cyclictest 您没有设置 CYCLICTEST_MAXIMUM_LATENCY 或 MAXIMUM_LATENCY 的值，该 工具会跳过预期和实际最大延迟的比较。 OSLAT_MAXIMUM_L ATENCY 指定 oslat 测试结果的最大可接受延迟（微秒）。如果您没有设置 OSLAT_MAXIMUM_LATENCY 或 MAXIMUM_LATENCY 的值，该工具会跳 过预期和实际最大延迟的比较。 MAXIMUM_LATENC

........................................................................... 14 4.12 Monitoring Latency................................................................................................. records-lag-max > threshold • millisBehindLatest > threshold 4.12 Monitoring Latency Generally speaking, latency is the delay between the creation of an event and the time at which results based which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for various reasons including the following: 1. It might take

10000000 10000000 10000000 延时latency min(ms) 1.15 2.11 32.86 0.91 1.16 延时latency avg(ms) 7.07 14.67 155.32 3.51 20.53 延时latency max(ms) 3536.90 1464.20 1977.64 359.59 3173.37 延时latency 95th percentile(ms) 22 22.28 29.19 200.47 8.74 38.25 延时latency sum(ms) 70703222.04 146665415.95 1553214351.69 35054605.48 205329856.41 线程公平性(events (avg/stddev)) 19531.2500/486.76 19531.2500/1208.70 19531.2500/1484.87 19531.2500/3230 10000000 10000000 10000000 延时latency min(ms) 1.10 2.23 39.18 0.95 1.18 延时latency avg(ms) 8.20 18.09 172.39 3.55 21.93 延时latency max(ms) 3103.19 1124.73 1276.10 79.71 3179.78 延时latency 95th percentile(ms) 26

· · · · · · · · · · · · · · · · · · · · · · · 1221 10.2.4 Troubleshoot Increased Read and Write Latency · · · · · · · · · · · · · · · · · · · 1232 10.2.5 Troubleshoot Write Conflicts in Optimistic Transactions· 1.5 TiCDC Performance Analysis and Tuning Methods · · · · · · · · · · · · · · · · · 1357 11.1.6 Latency Breakdown · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · but certain TiDB servers experience resource exhaustion leading to a significant in- crease in latency. To address this issue, TiProxy provides dynamic migration for connection, which allows connections

10 Does TiCDC support outputting data changes in the Canal protocol? · 1205 7.7.11 Why does the latency from TiCDC to Kafka become higher and higher?1206 7.7.12 When TiCDC replicates data to Kafka, can mode and BR from upstream? · 1210 7.7.26 After a changefeed resumes from pause, its replication latency gets higher and higher and returns to normal only after a few minutes. Why? · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 1655 10.2.4 Troubleshoot Increased Read and Write Latency · · · · · · · · · · · · · · · · · · · 1667 10.2.5 Troubleshoot Write Conflicts in Optimistic Transactions·

· · · · · · · · · · · · · · · · · · · · · · · 1356 10.2.4 Troubleshoot Increased Read and Write Latency · · · · · · · · · · · · · · · · · · · 1368 10.2.5 Troubleshoot Write Conflicts in Optimistic Transactions· 10 TiCDC Performance Analysis and Tuning Methods · · · · · · · · · · · · · · · · · 1527 11.1.11 Latency Breakdown · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 2.1.1 Performance • Introduce parallel batching modes for TSO requests, reducing TSO retrieval latency #54960 #8432 @MyonKeminta Before v8.4.0, when requesting TSO from PD, TiDB collects multiple TSO