������������������� StatefulSetPlus Operator Ø Keep share memory during Pod upgrade Ø Upgrade jitter (a few ms) for keep-alive services Flexible and dynamic resource management Dynamic Scheduler is

环境 境变 变量 量 描述 描述 LATENCY_TEST_DE LAY 指定测试开始运行的时间（以秒为单位）。您可以使用变量来允许 CPU 管理器协 调循环来更新默认的 CPU 池。默认值为 0。 LATENCY_TEST_CP US 指定运行延迟测试的 pod 使用的 CPU 数量。如果没有设置变量，则默认配置包含 所有隔离的 CPU。 LATENCY_TEST_RU NTIME 指 HWLATDETECT_MA XIMUM_LATENCY 指定工作负载和操作系统的最大可接受硬件延迟（微秒）。如果您没有设置 HWLATDETECT_MAXIMUM_LATENCY 或 MAXIMUM_LATENCY 的值， 该工具会比较默认预期阈值(20μs)和工具本身中实际的最大延迟。然后，测试会失 败或成功。 CYCLICTEST_MAXI MUM_LATENCY 指定 cyclictest 您没有设置 CYCLICTEST_MAXIMUM_LATENCY 或 MAXIMUM_LATENCY 的值，该 工具会跳过预期和实际最大延迟的比较。 OSLAT_MAXIMUM_L ATENCY 指定 oslat 测试结果的最大可接受延迟（微秒）。如果您没有设置 OSLAT_MAXIMUM_LATENCY 或 MAXIMUM_LATENCY 的值，该工具会跳 过预期和实际最大延迟的比较。 MAXIMUM_LATENC

........................................................................... 14 4.12 Monitoring Latency................................................................................................. records-lag-max > threshold • millisBehindLatest > threshold 4.12 Monitoring Latency Generally speaking, latency is the delay between the creation of an event and the time at which results based which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for various reasons including the following: 1. It might take

number of parameters, the amount of training resources required to train the network, prediction latency, etc. Natural language models such as GPT-3 now cost millions of dollars to train just one iteration training process in terms of computation cost, memory cost, amount of training data, and the training latency. It addresses questions like: ● How long does the model take to train? ● How many devices are needed parameters does the model have, what is the disk size, RAM consumption during inference, inference latency, etc. Using the sensitive tweet classifier example, during the deployment phase the user will be

ideas, the compression techniques. Compression techniques aim to reduce the model footprint (size, latency, memory etc.). We can reduce the model footprint by reducing the number of trainable parameters. prediction latency, RAM consumption and the quality metrics, such as accuracy, F1, precision and recall as shown in table 2-1. Footprint Metrics Quality Metrics ● Model Size ● Inference Latency on Target the model with respect to one or more of the footprint metrics such as the model size, inference latency, or training time required for convergence with a little quality compromise. Hence, it is important

lMemory access latency issues: ØDirect reclaim • Background reclaim (kswapd) • Foreground reclaim (direct reclaim) ØPage fault • Major page fault will produce latency Memory Access Latency(Page Fault) Fault) alloc mem free mem? No Latency Page Reclaiming back-reclaim No Latency Direct Reclaim Y N Enough NOT Enough Latency lDirect reclaim Øvm.min_free_kbytes: 3g Øvm.extra_free_kbytes: 8g Used add_to_page_cache_locked 自旋锁- treelock 1.4万亿 低延迟分布式存储系统 – PageCache的毛刺内核源码分析 lMemory access latency issues: ØMemory lock ØWake_up_page ØWait_on_page_locked() ØWait_on_page_writebacfk() LOCKED DIRTY

lMemory access latency issues: ØDirect reclaim • Background reclaim (kswapd) • Foreground reclaim (direct reclaim) ØPage fault • Major page fault will produce latency Memory Access Latency(Page Fault) Fault) alloc mem free mem? No Latency Page Reclaiming back-reclaim No Latency Direct Reclaim Y N Enough NOT Enough Latency lDirect reclaim Øvm.min_free_kbytes: 3g Øvm.extra_free_kbytes: 8g Used add_to_page_cache_locked 自旋锁- treelock 1.4万亿 低延迟分布式存储系统 – PageCache的毛刺内核源码分析 lMemory access latency issues: ØMemory lock ØWake_up_page ØWait_on_page_locked() ØWait_on_page_writebacfk() LOCKED DIRTY

middle boxes) ● High performance networking ○ Much higher multi-Gbps peak data speeds ○ Ultra low latency ○ And of course, reduce overheads introduced! ● High availability ● CapEx, OpEx #IstioCon Security Overheads introduced ● No high performance data path support ○ Multi-Gbps bandwidth ○ Ultra low latency #IstioCon Performance Limitations: Solutions ● Software techniques ○ (eBPF-based) TCP/IP stack co-designs #IstioCon Latency Analysis ● ~3ms P90 latency added ○ Istio v1.6 ○ More for VM usage ● Hotspots ○ 1  2 ○ 3  4: 30%~50% ● Others ○ Latency between Pods ○ Latency introduced by C/S #IstioCon

• Long tail latency is a small percentage of the overall latency • Vendors increasing shipping configurations with NVME • Bet in the right direction of hardware trends. • Low latency metadata can ~500MB/s Typically 400 MB/s - 550MB/s Up to 600MB/s Typically 3,000 MB/s-5,000 MB/s Up to 7,000 MB/s Latency (4kb) ~10 ms ~200 us ~60 us Size 1TB - 16TB each Up to 20TB 500GB - 4TB each Up to 15TB 500GB 3. DN saturation of network 4. Better leveraging benefits of NVME a. Squeezing every bit of latency from each request processing b. Better caching architectures from computation down to disk to leverage

trades-off result accuracy for sustainable performance. • Suitable for applications with strict latency constraints that can tolerate approximate results. Slow down the flow of data: • The system buffers plan • It detects overload and decides what actions to take in order to maintain acceptable latency and minimize result quality degradation. 7 ??? Vasiliki Kalavri | Boston University 2020 DSMS University 2020 Load shedding decisions • When to shed load? • detect overload quickly to avoid latency increase • monitor input rates • Where in the query plan? • dropping at the sources vs. dropping