rights reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Models with with dynamism ● Control flow (if, loop, etc) ● Dynamic shapes ○ Dynamic inputs: batch size, image size, sequence length, etc. ○ Output shape of some ops are data dependent: arange, nms, etc. ○ Control models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support Any-dim in typing ● Use shape function to compute the type at runtime ● Virtual

Efficient Mixture-of-Experts Language Model DeepSeek-AI research@deepseek.com Abstract We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models. The model checkpoints are available at h t t p s : / / g i t h u b . c o m / d e e p s e e k - a i / D e e p Work 21 A Contributions and Acknowledgments 27 B DeepSeek-V2-Lite: A 16B Model Equipped with MLA and DeepSeekMoE 29 2 B.1 Model Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

writing styles 59 For few-shot prompting with classification tasks, mix up the classes 59 Adapt to model updates 60 Experiment with output formats 60 JSON Repair 61 Working with Schemas 62 Experiment When thinking about a large language model input and output, a text prompt (sometimes accompanied by other modalities such as image prompts) is the input the model uses to predict a specific output. You can be complicated. Many aspects of your prompt affect its efficacy: the model you use, the model’s training data, the model configurations, your word-choice, style and tone, structure, and context

Change Happening Faster Than Ever? Yes, It Is • AI User + Usage + CapEx Growth = Unprecedented • AI Model Compute Costs High / Rising + Inference Costs Per Token Falling = Performance Converging + Developer 2/24 2/25 4/25 75% 60% 10% 21% 15% 0% Details on Page 293 USA – LLM #1 China USA – LLM #2 AI Model Compute Costs High / Rising + Inference Costs Per Token Falling = Performance Converging + Developer Change Happening Faster Than Ever? Yes, It Is • AI User + Usage + CapEx Growth = Unprecedented • AI Model Compute Costs High / Rising + Inference Costs Per Token Falling = Performance Converging + Developer

= multiply(%1, 4 /* ty=int32 */) /* ty=Tensor[(1, 3, 2, 3), int32] */; %3 = nn.avg_pool2d(%2, pool_size=[2, 2]) /* ty=Tensor[(1, 3, 1, 2), int32] */; %4 = sum(%3, axis=[1], keepdims=True) /* ty=Tensor[(1 Metric – Latency in ms for batch size = 1 • 1.7x speedup on Inception asymmetric quantized model • Mobilenet requires depthwise convolution VNNI schedule • Symmetric model improves the speedup to 2.8x© All rights reserved. Conclusion • TVM community is pursuing both Automatic- and Pre-quantized model support. Contributions are welcomed. • We need new/tuned TVM schedules using fast Integer operations

Overlay Processor ˃ DNN Specific Instruction Set Convolution, Max Pool etc. ˃ Any Network, Any Image Size ˃ High Frequency & High Compute Efficiency ˃ Supported on U200 – 3 Instances U250 – 4 Instances >> 4© Copyright 2018 Xilinx Inference Flow >> 5 MxNet CPU Layers FPGA Layers Runtime Image Model Weights Calibration Set Quantizer Compiler Tensor Graph Optimization Framework Tensor Graph to ins, outs: tvm.call_packed('tvm.accel.accel_fused', attrs['path'], attrs['output_layout'], attrs['model_name'], outs[0], *ins ), name=name) return out >> 10© Copyright 2018 Xilinx Example of FPGA node

on widely-used operators (e.g. conv2d, dense, ReLU, etc) Now your customer wants to run a YOLO model, but... ResNet-50 Dense Non Maximum Suppression Non Maximum Suppression (NMS) is too new to import relay 2. Load a pretrained network mod, params = relay.testing.mobilenet.get_workload(batch_size=1) 3. Partition and build the network with an external codegen mod = relay.build_extern(mod, “dnnl”) the inference exe = relay.create_executor(“vm”, mod=mod, ctx=tvm.cpu(0)) data = np.random.uniform(size=(1, 3, 224, 224)).astype(“float32”) out = exe.evaluate()(data, **params) How Would That Look Like

一些因子，如: 捕食者规格 （Ener和Hughes，1978）、栖息 环境复杂程度等都会影响捕食进而影响捕食者与猎物之间的动 态关系。 Some factors, such as predator size (Elner andHughes, 1978) and habitat complexity, can affectpredation and subsequently influence the Belk (2018). A universal material+testing machine(MTS System Corporation, Eden Prairie, MIN, USA, Model 661; Fig1,)was used to determine the shell strength. Each shell valve was placed horizontally with Belk (2018). A universal material-testing machine (MTS System Corporation, Eden Prairie, MN, USA, Model 661; Fig. 1) was used to determine the shell strength. Each shell valve was placed horizontally with

design foundations In its most fundamental form, an agent consists of three core components: 01 Model The LLM powering the agent’s reasoning and decision-making 02 Tools External functions or APIs the the workflow. Not every task requires the smartest model—a simple retrieval or intent classification task may be handled by a smaller, faster model, while harder tasks like deciding whether to approve approve a refund may benefit from a more capable model. An approach that works well is to build your agent prototype with the most capable model for every task to establish a performance baseline. From there

They started with three model evals: 01 Language translation Measuring the accuracy and quality of translations produced 
 by a model. 02 Summarization Evaluating how a model condenses information, using resilient to change. 
 Evals are built around tasks that measure 
 the quality of the output of a model against 
 a benchmark—is it more accurate? More compliant? Safer? Your key metrics will depend on more tokens. To increase efficiency, OpenAI and Indeed 
 worked together to fine-tune a smaller GPT model that was able to deliver similar results 
 with 60% fewer tokens. Helping job seekers find the