China 2022 Python Profiling 原理深入探索与实践 主讲人：羿莉（萧羿）- 阿里云SLS Python  HELLO WORLD ## Python Profiling 大纲 ## ●背景概述 ## ●Python ●Python Profiling的场景与技术原理 ➢场景：即时 VS 持续 ➢技术原理 确定剖析 VS 采样剖析 函数粒度VS 行粒度 物理时间VS CPU 时间 技术全景概览 技术细节探索 ## ●Python Profiling的工具链和解决方案 ➢CPU: cProfile、Profile、Pyinstrument、line-profiler，py-spy ➢Memory : memory-profiler、memray ➢新兴Continuous 方案：Pyroscrope ## ●实践 & 展示 ## 背景概述 ## • 什么是Profiling? - 定义：Profiling是一种以收集程序运行时信息为手段研究动态的程序行为的分析方法。其分析对象是程序的空间或时间复杂度、特定指令的使用情形、函数调用的频率以及执行的时间等等。 • 步骤：数据采集、统计分析、可视化、推理导出

overhead ## Why BPF? - Flexibility - Low overhead - Continuous profiling ## Why BPF? - Flexibility - Low overhead - Continuous profiling - No modifications of the tracee ## rbperf ## rbperf - Profile ## rbperf - Profile Ruby programs - Trace complex Ruby programs execution ## rbperf – on-CPU profiling - $ rbperf record --pid=124 cpu - $ rbperf report [...] ## rbperf – Rails on-CPU profile ## rbperf versions - Correctness testing - BPF safety features ## Future plans - Integrate in Facebook’s profiling infra - Rewrite OSS driver program - Make the OSS version awesome - Better documentation (including

REPL formatting ..... 372 29.7 System and Package Image Building ..... 373 29.8 Debugging and profiling ..... 373 30 Embedding Julia ..... 376 30.1 High-Level Embedding ..... 376 30.2 High-Level Federation of packages ..... 389 31.3 Environments ..... 390 31.4 Conclusion ..... 400 32 Profiling ..... 401 32.1 Basic usage ..... 401 32.2 Accumulation and clearing ..... 405 32.3 Options 405 32.4 Configuration ..... 406 32.5 Memory allocation analysis ..... 406 32.6 External Profiling ..... 408 33 Stack Traces ..... 409 33.1 Viewing a stack trace ..... 409 33.2 Extracting useful

4c85b44eb3710c323581ae/p23_1.jpg) ## Performance Analysis how to do it right ## Performance Profiling how to do it better Why is this so hard?   ![Im

Resources • System Table ☐ SQL Statements Implementation ☐ SLOW QUERY Implementation • Continuous Profiling Implementation ## TopSQL: Bind SQL With CPU Resources ## Background - Database Performance is getting jpg)  ## How CPU Profiling works: Sampling  function-D(20%) function-C(20%) ## CPU Profiling By SQL Request  ## CPU Profiling By SQL Request |Request tag|Count| |---|---|

kernel.bpf_stats_enabled=1 # do profiling $ sysctl -w kernel.bpf_stats_enabled=0 ## procfs ☀️ ☀️ ☁️ $ echo 1 > /proc/sys/kernel/bpf_stats_enabled # do profiling $ echo 0 > /proc/sys/kernel/bpf_stats_enabled bpf(BPF_ENABLE_STATS, &attr, sizeof(attr)); int fd = enable_stats(); if (fd < 0) { return; } // do profiling close(fd); ## bpftool prog profile ## bpftool prog profile – Added in kernel v5.7 – Uses hardware – Available metrics: - cycles, instructions, lld loads, llc misses – Used for more in-depth profiling ## bpftool prog run / BPF_PROG_TEST_RUN ## bpftool prog run – Added in kernel v4.12 – Only for

Ensure that the --secure-port argument is not set to 0 (Automated) 52 1.2.21 Ensure that the --profiling argument is set to false (Automated) 53 1.2.22 Ensure that the --audit-log-path argument is set --terminated-pod-gc-threshold argument is set as appropriate (Automated) 77 1.3.2 Ensure that the --profiling argument is set to false (Automated) 78 1.3.3 Ensure that the --use-service-account-credentials argument is set to 127.0.0.1 (Automated) 81 1.4 Scheduler 83 1.4.1 Ensure that the --profiling argument is set to false (Automated) 83 1.4.2 Ensure that the --bind-address argument is set to

benchmarking Servo over time, with debug symbols stripped for faster initial startup • profiling builds are ideal for profiling and troubleshooting performance issues; they behave like a debug or release build as a production build ||production|production-stripped|profiling| |---|---|---|---| |mach --profile|production|production-stripped|profiling| |debug info?|no|no|yes| |symbols?|yes|no|yes| |finds resources crashes on certain lines involving the profile() function, it's not just you. Comment out the profiling code, and only keep the inner function, and that should do it. ## Reversible debugging with rr

Ensure that the --anonymous-auth argument is set to false (Scored) • 1.1.8 - Ensure that the --profiling argument is set to false (Scored) - 1.1.11 - Ensure that the admission control plugin AlwaysPullImages Look for the following options in the command section of the output: --anonymous-auth=false --profiling=false --service-account-lookup=true --enable-admission-plugins=ServiceAccount,NamespaceLifecycle the command line options should be set on the Kubernetes scheduler. • 1.2.1 - Ensure that the --profiling argument is set to false (Scored) • 1.2.2 - Ensure that the --address argument is set to 127.0

nanoseconds per zone) Cross-platform (Windows, Linux, macOS, iOS, Android, WASM $ ^{*} $ ) Hybrid profiling capabilities (sampling and/or instrumentation) (CPU and GPU instrumentation) Tracing capabilities nanoseconds per zone) Cross-platform (Windows, Linux, macOS, iOS, Android, WASM $ ^{*} $ ) Hybrid profiling capabilities (sampling and/or instrumentation) Tracing capabilities (values, messages, plots, nanoseconds per zone) Cross-platform (Windows, Linux, macOS, iOS, Android, WASM $ ^{*} $ ) Hybrid profiling capabilities (sampling and/or instrumentation) Tracing capabilities (values, messages, plots,