1/66Bridging the Gap: Writing Portable Programs for CPU and GPU using CUDA Thomas Mejstrik Sebastian Woblistin 2/66Content 1 Motivation Audience etc.. Cuda crash course Quiz time 2 Patterns Oldschool Motivation Patterns The dark path Cuda proposal Thank you Why write programs for CPU and GPU Difference CPU/GPU Algorithms are designed differently Latency/Throughput Memory bandwidth Number of cores Motivation Patterns The dark path Cuda proposal Thank you Why write programs for CPU and GPU Difference CPU/GPU Why it makes sense? Library/Framework developers Embarrassingly parallel algorithms User

http://wongmichael.com/about ● C++11 book in Chinese: https://www.amazon.cn/dp/B00ETOV2OQ We build GPU compilers for some of the most powerful supercomputers in the world 34 Nevin “:-)” Liber nliber@anl Attribution 4.0 International License SYCL Single Source C++ Parallel Programming GPU FPGA DSP Custom Hardware GPU CPU CPU CPU Standard C++ Application Code C++ Libraries ML Frameworks give better performance on complex apps and libs than hand-coding AI/Tensor HW GPU FPGA DSP Custom Hardware GPU CPU CPU CPU AI/Tensor HW Other BackendsSYCL 2020 is here! Open Standard for

阿里巴巴人工智能实验室 AiILabs & TVM PART 1 : ARM32 CPU CONTENT PART 2 : HIFI4 DSP PART 3 : _ PowervVR GPU [和| Alibaba AL.Labs 阿里巴巴人工智能实验室 ARM 32 CPU Resolution Quantization Orize Kernel ALIOS TVM Alibaba HIFI4 DSP HIFI4 DSP [和| Alibaba AL.Labs 阿里巴巴人工智能实验室 PowerVR GPU Alibaba Al.Labs 阿里巴巴人工智能实验室 PowerVR support by TVM NNVM Compiler -Execution graph -Model layers functions

CPU vFast GPU vvFast PCI Bus (or other fabric)GPUs as a First-Class Computing Resource CPU GPU PCI Bus (or other fabric) NIC - Historically, network comm. was CPU-centric 1) Direct GPU access to Infiniband allows GPU-to-GPU network transfers 2) Fast in-node fabrics like NVLink, Infinity Fabric allow very fast intra-node transfers DataGPUs as a First-Class Computing Resource CPU GPU PCI Bus (or fabric) NIC Data - Historically, network comm. was CPU-centric 1) Direct GPU access to Infiniband allows GPU-to-GPU network transfers 2) Fast in-node fabrics like NVLink, Infinity Fabric allow

B" : GPU operation 9Existing TGPSs on Heterogenous Computing - Challenge A C D B! B" 5 task_b = sched.emplace([](&){ 6 // CPU code; // GPU code; 7 }); // CPU thread blocks until GPU finishes B" : GPU operation 10Existing TGPSs on Heterogenous Computing - Challenge A C D B! B" 5 task_b = sched.emplace([](&){ 6 // CPU code; // GPU code; 7 }); // CPU thread blocks until GPU finishes operation B" : GPU operation Atomic execution per task 11Existing TGPSs on Heterogenous Computing - Challenge CPU A B! C Idle GPU D B" Runtime A C D B! B" Assume one CPU and one GPU B! : CPU operation

cudaFree(x); cudaFree(y); } An Even Easier Introduction to CUDA 5 |__global__ void add_gpu(int n, float* x, float* y) { for (int i = 0; i < n; i++) y[i] = x[i] + y[i]; } TEST_CASE("cppcon-1" TEST_CASE("cppcon-1", "[CUDA]") { // … } An Even Easier Introduction to CUDA 6 |__global__ void add_gpu(int n, float* x, float* y) { for (int i = 0; i < n; i++) y[i] = x[i] + y[i]; } TEST_CASE("cppcon-1" 20; float* x; float* y; // … add_gpu<<<1, 1>>>(N, x, y); // … } An Even Easier Introduction to CUDA 7 |__global__ void add_gpu(int n, float* x, float* y) { for (int i = 0;

MiniMax 等。  其次是由 TogetherAI、Groq、Fireworks、Replicate、硅基流动等组成的 GPU 推理集群 服务提供商，它们处理扩展与缩减等技术难题，并在基本计算费用基础上收取额外费用，从 而让应用公司无需承担构建和管理 GPU 推理集群的高昂成本，而是可以直接利用抽象化的 AI 基础设施服务。  第三类是传统的云计算平台，例如亚马逊的 Amazon Vertex AI 等，允许应用开发者轻松部署和使用标准化或定制化的 AI 模型， 并通过 API 接口调用这些模型。  最后一类是本地推理，SGLang、vLLM、TensorRT-LLM 在生产级 GPU 服务负载中表现 出色，受到许多有本地托管模型需求的应用开发者的欢迎，此外，Ollama 和 LM Studio 也 是在个人计算机上运行模型的优选方案。 62 / 111 除模型层面外，应 软件，例如：微控制处理器（MCU）会运行实时操作系统或者直接运行某个特定程序；中央处 理器（CPU）往往会运行 Windows、Linux 等复杂操作系统作为底座支撑整个软件栈；图形 处理器（GPU）一般不加载操作系统而是直接运行图形图像处理程序，神经网络处理器 （NPU）则直接运行深度学习相关程序。 处理器芯片设计是一项很复杂的任务，整个过程犹如一座冰山。冰山水面上是用户或者大 众看到

involve experimental prototypes and early research.Problem: writing parallel programs is hard - Multi-GPU, multi-CPU systems require partitioning data - Users must manually split up data amongst GPUs / execution necessary. CPU NIC GPU GPU GPU GPU Xe LinkMulti-GPU Systems - NUMA regions: - 4+ GPUs - 2+ CPUs CPU NIC GPU GPU GPU GPU Xe LinkMulti-GPU Systems - NUMA regions: - 4+ GPUs more memory domains - Software needed to reduce complexity CPU NIC GPU Tile 1 Tile 0 GPU Tile 1 Tile 0 GPU Tile 1 Tile 0 GPU Tile 1 Tile 0 Xe LinkProject Goals - Offer high-level, standard C++

数 Windows 配置要求 Mac 配置要求 适⽤场景 1.5B - RAM: 4GB - GPU: 集成显卡/现代CPU - 存储: 5GB - 内存: 8GB （M1/M2/M3） - 存储: 5GB 简单⽂本⽣成、基础代 码补全 7B - RAM: 8-10GB - GPU: GTX 1680（4-bit量 化） - 存储: 8GB - 内存: 16GB（M2 Pro/M3） Pro/M3） - 存储: 8GB 中等复杂度问答、代码 调试 14B - RAM: 24GB - GPU: RTX 3090（24GB VRAM） - 存储: 20GB - 内存: 32GB（M3 Max） - 存储: 20GB 复杂推理、技术⽂档⽣ 成 32B+ 企业级部署（需多卡并联） 暂不⽀持 科研计算、⼤规模数据 处理 2. 算⼒需求分析 模型 参数规 模 2*XE9680（16*H20 GPU） DeepSeek-R1-Distill- 70B 70B BF16 ≥180GB 4*L20 或 2*H20 GPU 三、国产芯⽚与硬件适配⽅案 1. 国内⽣态合作伙伴动态 企业 适配内容 性能对标（vs NVIDIA） 华为昇 腾 昇腾910B原⽣⽀持R1全系列，提供端到端推理优化 ⽅案 等效A100（FP16） 沐曦 GPU MXN系列⽀持70B模型BF16推理，显存利⽤率提升

Impressive61 NVIDIA AI Ecosystem Tells Over Four Years = >100% Growth in Developers / Startups / Apps Note: GPU = Graphics Processing Unit. Source: NVIDIA (2021 & 2025) NVIDIA Computing Ecosystem – 2021-2025, per Cloud vs. AI Patterns105 Tech CapEx Spend Partial Instigator = Material Improvements in GPU PerformanceNVIDIA GPU Performance = +225x Over Eight Years 106 1 GPT-MoE Inference Workload = A type of workload Source: NVIDIA (5/25) Performance of NVIDIA GPU Series Over Time – 2016-2024, per NVIDIA Tech CapEx Spend Partial Instigator = Material Improvements in GPU Performance Pascal Volta Ampere Hopper Blackwell