方案设计篇：如何设计可落地的AI解决方案 扫码试看/订阅 《 TensorFlow 2项目进阶实战》视频课程 • 行业背景：AI新零售是什么? • 用户需求：线下门店业绩如何提升？ • 长期⽬目标：货架数字化与业务智能化 • 短期目标：自动化陈列审核和促销管理 • 方案设计：基于深度学习的检测/分类的AI流水线 • 方案交付：支持在线识别和API调用的 AI SaaS 目录 行业背景：AI新零售是什么 纯度 排面 SKU 种类 数量 位置 品类 占比 货架 设计 场景 层数 编号 销售执行三板斧：分销达标 销售执行三板斧：新品上架陈列稽查 销售执行三板斧：陈列激励 方案设计： 基于深度学习的检测/分类的AI流水线 货架商品检测 货架商品检测 Bottle（瓶装） Combination（组合装） 货架商品检测 Bottle（瓶装） Combination（组合装）

分辨听到的所有 说明Hadoop卓尔不群的言论。充分使用Hadoop的神话 与现实之间存在巨大的反差，这在大数据集成方面表现尤为 突出。很多业界传言称，任何不可扩展的抽取、转换和加载 (ETL) 工具搭配Hadoop后都会得到高性能、高度可扩展 的数据集成平台。 事实上，MapReduce的设计宗旨并非是对海量数据进行 高性能处理，而是为了实现细粒度的容错。这种差异可能会 使整体性能和有效性降低一个数量级乃至更多。 一个Hadoop与大数据集成的重要用例是将大型ETL工作负载 从企业数据仓库 (EDW) 卸载下来，以便降低成本并改善查询 服务水平协议 (SLA)。该用例会引发以下问题： • 企业是否应卸载EDW中的所有ETL工作负载？ • 是否应将所有大数据集成工作负载都推送到Hadoop？ • 在没有并行关系数据库管理系统 (RDBMS) 和Hadoop 的情况下，大数据集成工作负载在ETL网格中发挥怎样 的持续作用？ 选择并行RDBMS、Hadoop和可扩展的ETL网格来运行大数据 集成工作负载。但无论选择哪种方法，信息基础架构都必须满足 一个常见的要求：全面支持大规模可扩展处理。 某些数据集成操作在RDBMS引擎内外的运行效率较高。同样， 并非所有数据集成操作均适用于Hadoop环境。设计精妙的架 构必须足够灵活，可以充分利用系统中每个环境的优势（参见 图3）。 在ETL网格中运行 在数据库中运行 在Hadoop中运行

[https://iceberg.apache.org/] to build and manage Data Lake with pure SQL for both data processing e.g. ETL, and analytics e.g. BI. All workloads can be done on one platform, using one copy of data, with one engine per session Large-scale ETL Ad hoc High Low USER One engine per user Ad hoc Small-scale ETL Medium Medium GROUP One engine per primary group Ad hoc Small-scale ETL Low High SERVER One engine able to leverage engines in different ways to handle their different workloads, such as large-scale ETL jobs and interactive ad hoc queries. Kyuubi Security Overview 1. Authentication 1.1. Using KERBEROS

[https://iceberg.apache.org/] to build and manage Data Lake with pure SQL for both data processing e.g. ETL, and analytics e.g. BI. All workloads can be done on one platform, using one copy of data, with one engine per session Large-scale ETL Ad hoc High Low USER One engine per user Ad hoc Small-scale ETL Medium Medium GROUP One engine per primary group Ad hoc Small-scale ETL Low High SERVER One engine able to leverage engines in different ways to handle their different workloads, such as large-scale ETL jobs and interactive ad hoc queries. Kyuubi Security Overview 1. Authentication 1.1. Using KERBEROS

Alluxio 服务器 Alluxio 服务器 大数据查询 大数据ETL 模型训练 Alluxio核心功能一：分布式数据缓存 Alluxio 服务器 A B /path1/file1 /path2/file2 C A B C A Alluxio 服务器 Alluxio 服务器 大数据查询 大数据ETL 模型训练 Alluxio核心功能二：灵活多样的数据访问API Alluxio Alluxio 服务器 HDFS接口客户端 POSIX接口客户端 Alluxio 服务器 Alluxio 服务器 大数据查询 大数据ETL 模型训练 Alluxio核心功能三：统一的文件系统抽象 Alluxio 服务器 Alluxio在云端AI训练场景的性能好处 • 支持大规模的数据缓存 • 本地内存加速 • 支持数据预热 • LRU缓存管理 Object storage

Big Data Platform 数字化转型过程中的数据流转 GemFire® In-Memory Data Grid and Cache ETL/CDC AI/ML AS-IS TO-BE Labs Data Services ❖ ETL: Extract, Transforming, Loading ❖ CDC: Changed Data Capture © VMware Lakes / Other Data Platforms / Public Cloud Data Lakes / External Storages 各种数据平台 结构化 & 半结构化数据 ETL , Streaming SQL, R, Python, Java,C... BI/Analytic/ML ... 数据科学 数据捕获 GPORCA 优化器 | 并行执行 | 数据分布

[https://iceberg.apache.org/] to build and manage Data Lake with pure SQL for both data processing e.g. ETL, and analytics e.g. BI. All workloads can be done on one platform, using one copy of data, with one engine per session Large-scale ETL Ad hoc High Low USER One engine per user Ad hoc Small-scale ETL Medium Medium GROUP One engine per primary group Ad hoc Small-scale ETL Low High SERVER One engine able to leverage engines in different ways to handle their different workloads, such as large-scale ETL jobs and interactive ad hoc queries. 4. The TTL Of Kyuubi Engines For a multi-tenant cluster, its

[https://iceberg.apache.org/] to build and manage Data Lake with pure SQL for both data processing e.g. ETL, and analytics e.g. BI. All workloads can be done on one platform, using one copy of data, with one engine per session Large-scale ETL Ad hoc High Low USER One engine per user Ad hoc Small-scale ETL Medium Medium GROUP One engine per primary group Ad hoc Small-scale ETL Low High SERVER One engine able to leverage engines in different ways to handle their different workloads, such as large-scale ETL jobs and interactive ad hoc queries. 4. The TTL Of Kyuubi Engines For a multi-tenant cluster, its

[https://iceberg.apache.org/] to build and manage Data Lake with pure SQL for both data processing e.g. ETL, and analytics e.g. BI. All workloads can be done on one platform, using one copy of data, with one engine per session Large-scale ETL Ad hoc High Low USER One engine per user Ad hoc Small-scale ETL Medium Medium GROUP One engine per primary group Ad hoc Small-scale ETL Low High SERVER One engine able to leverage engines in different ways to handle their different workloads, such as large-scale ETL jobs and interactive ad hoc queries. 4. The TTL Of Kyuubi Engines For a multi-tenant cluster, its

[https://iceberg.apache.org/] to build and manage Data Lakehouse with pure SQL for both data processing, e.g. ETL, and online analytics processing(OLAP), e.g. BI. All workloads can be done on one platform, using one Delta Lake. Logical data warehouse Provide a relational abstraction on top of disparate data without ETL jobs, from collecting to connecting. Run Anywhere at Any Scale Most of the Kyuubi engine types have Degree Shareability CONNECTION One engine per session Large-scale ETL Ad hoc High Low USER One engine per user Ad hoc Small-scale ETL Medium Medium Share Level Syntax Scenario Isolation Degree Shareability