Growth Over Nine Years of… Compute Gains from Better Algorithms Led To… Note: Estimates how much progress comes from bigger models versus smarter algorithms, based on how much computing power you'd need to to reach top performance without any improvements. Source: Epoch AI (3/24) Impact of Improved Algorithms on AI Model Performance – 2014-2023, per Epoch AI Effective Compute (Relative to 2014) +200% electricity and Internet. - NVIDIA Co-Founder & CEO Jensen Huang, 5/2578 Research = Rapid Ramp in FDA-Approved AI Medical Devices, per Stanford HAI Note: FY21, FY22 & FY23 USA government budget figures are

parallel all-to-all communication. We also customize faster CUDA kernels for communications, routing algorithms, and fused 12 1K 12K 24K 35K 47K 58K 70K 81K 93K 104K 116K 128K Context Length (#Tokens) 0