## RUST CHINA CONF 2023 第三届中国 Rust 开发者大会 6.17-6.18 @Shanghai ## 使用 Rust 构建 LLM 应用 夏歌 ## 😍  ## Bojan Tunguz ![Image 培养更广泛的 Rust 开发 围绕 LLM 生态封装相应的 Rust 框架，让开发者能够使用简单的 Rust 写应用 如何用 Rust 实现的 构建和部署 AI 相关工作流的 serverless 平台 - 上传 Rust function，平台负责将 Rust 编译成 Wasm，并运行在 WasmEdge 安全容器中 - 平台封装了一些常用 LLM 和 SaaS 的 API，并发布成了 crate，比如

## COSCHINA gitee ## 2023 中国开源开发者报告 China Open Source 2023 Annual Report ## LLM 技术报告 出品：OSCHINA & Gitee 编委会： 雨多田光，OSCHINA总编 局长，OSCHINA主编 王茜，OSCHINA主编 叶子，OSCHINA新媒体运营 鱼仔，OSCHINA新媒体运营 诺墨，Gitee开源社区产品负责人 设计：张琪 ## LLM 技术报告 大语言模型（LLM）技术作为人工智能领域的一项重要创新在今年引起了广泛的关注。 LLM 是利用深度学习和大数据训练的人工智能系统，专门设计来理解、生成和回应自然语言。这些模型通过分析大量的文本数据来学习语言的结构和用法，从而能够执行各种语言相关任务。以 GPT 系列为代表，LLM 以其在自然语言处理领域的卓越表现，成为推动语言理解、生成和应用的引擎。 LLM 在多个 在多个领域都取得了令人瞩目的成就。在自然语言处理领域，GPT 系列模型在文本生成、问答系统和对话生成等任务中展现出色的性能。在知识图谱构建、智能助手开发等方面，LLM 技术也发挥了关键作用。此外，它还在代码生成、文本摘要、翻译等任务中展现了强大的通用性。 本报告从技术人视角出发，将深入探讨 LLM 技术的背景、基础设施、应用现状，以及相关的工具和平台。 ![Image](/uploads/documents/f/4/8/5

multi-step tasks. Advances in reasoning, multimodality, and tool use have unlocked a new category of LLM-powered systems known as agents. This guide is designed for product and engineering teams exploring characteristics that allow it to act reliably and consistently on behalf of a user: 01 It leverages an LLM to manage workflow execution and make decisions. It recognizes when a workflow is complete and can rules engine works like a checklist, flagging transactions based on preset criteria. In contrast, an LLM agent functions more like a seasoned investigator, evaluating context, considering subtle patterns

Documentation 3 2 Indices and tables 135 Python Module Index 137 Index 139 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput to PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. - vLLM Meetups. ## DOCUMENTATION ## 1 the ROCm driver version. ## 1.3 Installation with OpenVINO vLLM powered by OpenVINO supports all LLM models from vLLM supported models list and can perform optimal model serving on all x86-64 CPUs with

Documentation 3 2 Indices and tables 59 Python Module Index 61 Index 63 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput to PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. ## DOCUMENTATION ## 1.1 Installation vLLM other words, we use vLLM to generate texts for a list of input prompts. Import LLM and SamplingParams from vLLM. The LLM class is the main class for running offline inference with vLLM engine. The SamplingParams

Documentation 3 2 Indices and tables 91 Python Module Index 93 Index 95 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput to PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. ## DOCUMENTATION ## 1.1 Installation vLLM other words, we use vLLM to generate texts for a list of input prompts. Import LLM and SamplingParams from vLLM. The LLM class is the main class for running offline inference with vLLM engine. The SamplingParams

Documentation 3 2 Indices and tables 113 Python Module Index 115 Index 117 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput to PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. - vLLM Meetups. ## DOCUMENTATION ## 1 other words, we use vLLM to generate texts for a list of input prompts. Import LLM and SamplingParams from vLLM. The LLM class is the main class for running offline inference with vLLM engine. The SamplingParams

Documentation 3 2 Indices and tables 135 Python Module Index 137 Index 139 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput to PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. - vLLM Meetups. ## DOCUMENTATION ## 1 other words, we use vLLM to generate texts for a list of input prompts. Import LLM and SamplingParams from vLLM. The LLM class is the main class for running offline inference with vLLM engine. The SamplingParams

Infinity系统架构 02 ## 第一部分 RAG技术实践 ## 基于向量数据库的RAG解决方案 文档 文本块 LLM  提示词 ## LLM对企业信息架构的改变  交易记录  LLM  编排 ![Image](/up 挑战三：多样化查询需求 ■ 挑战四：幻觉、胡说八道 ☐ 挑战五：定制化成本 数据加工 ☐ 文档结构识别 ☐ 文字加工 ☐ 数据库 多路召回 ☐ 融合排序 ## RAG引擎工作流程 LLM Answer 大模型答案 Prompt Template 提示模板 Query 问题 文档格式解析 文档格式转换 文档格式解析 文档布局解析 Cross Attentional Reranker

Documentation 3 2 Indices and tables 157 Python Module Index 159 Index 161 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput to PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. - vLLM Meetups. ## DOCUMENTATION ## 1 the ROCm driver version. ## 1.3 Installation with OpenVINO vLLM powered by OpenVINO supports all LLM models from vLLM supported models list and can perform optimal model serving on all x86-64 CPUs with