stanford.edu/projects/glove 6 Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. "Efficient estimation of word representations in vector space." arXiv preprint arXiv:1301.3781 (2013). Figure 4-6: This Step 1: Vocabulary Creation In this step, we create a vocabulary of the top words10 (ordered by frequency) from the given training corpus. We would learn embeddings of dimensions each (where we can also

satisfied Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 11 / 25 Parameter Estimation in Probabilistic Models Assume data are generated via probabilistic model d ∼ p(d; θ) p(d; θ): Regularization and Bayesian Statistics September 20, 2023 12 / 25 Maximum Likelihood Estimation (MLE) Maximum Likelihood Estimation (MLE): Choose the parameter θ that maximizes the probability of the data, given parameter estimation θMLE = arg max θ ℓ(θ) = arg max θ m � i=1 log p(d(i); θ) Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 13 / 25 Maximum-a-Posteriori Estimation (MAP)

1（正样本）。 •（单词, 来自词汇表的随机单词），标签为 0（负样本）。 若要了解更多和 Skipgram 有关的知识，请参阅这份由 Mikolov 等人发表的经典论文：Efficient Estimation of Word Representations in Vector Space 参数 • sequence: 一个编码为单词索引（整数）列表的词序列（句子） 。如果使用一个 samp 中使用的采样分布生成： p(word) = (min(1, sqrt(word_frequency / sampling_factor) / (word_frequency / sampling_factor))) 我们假设单词频率遵循 Zipf 定律（s=1），来导出 frequency(rank) 的数值近似： frequency(rank) ~ 1/(rank * (log(rank) + gamma)

scale factor Arbitrary temporal frames Our Method 44 45 Data from Vid4 [Ce Liu et al.] Motion Estimation Our Method 46 ???????????????????????? ???????????? ????????????0 ???????????? ME ??????

至今仍用于解 决从保险计算到医疗诊断的许多问题。这些工具算法催生了自然科学中的一种实验方法——例如，电阻中电 流和电压的欧姆定律可以用线性模型完美地描述。 即使在中世纪，数学家对估计（estimation）也有敏锐的直觉。例如，雅各布·克贝尔 (1460–1533)18的几何学 书籍举例说明，通过平均16名成年男性的脚的长度，可以得出一英尺的长度。 图1.4.1: 估计一英尺的长度 图1 freqs = [freq for token, freq in vocab.token_freqs] d2l.plot(freqs, xlabel='token: x', ylabel='frequency: n(x)', xscale='log', yscale='log') 通过此图我们可以发现：词频以一种明确的方式迅速衰减。将前几个单词作为例外消除后，剩余的所有单词 大致遵循双对数坐 trigram_vocab.token_freqs] d2l.plot([freqs, bigram_freqs, trigram_freqs], xlabel='token: x', ylabel='frequency: n(x)', xscale='log', yscale='log', legend=['unigram', 'bigram', 'trigram']) 8.3. 语言模型和数据集 307

another equally sized range. It creates equal sized quantization ranges (bins), regardless of the frequency of data. Clustering helps solve that problem by adapting the allocation of precision to match the regions? Recall that huffman encoding does this by trying to create a huffman tree based on symbol frequency. As a result it comes up with a variable-length code, where a smaller length code is assigned to picked (orange dots). Notice that the centroids are densely distributed around the ranges where the frequency of x is high. How satisfying is that? You can rely on clustering to put its centroids where the

Partitions for Memory-Efficient Recommendation Systems Twiiter [RecSys21] Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems 9 千 万 key hash1(key) hash2(key) 千 万 业界⽅案：Double

(or for that matter any problem with sequential data), we can consider heuristics like vocabulary frequency (sequences with rare tokens are likely harder as shown in the language model task by Bengio et al

is Huffman Coding, where we assign unique strings of bits (codes) to the symbols based on their frequency in the data. More frequent symbols are assigned smaller codes, and less frequent symbols are assigned

通常可以假设?(?)符合已知的分布，比如?(0,1)。在?(?)已知的条件下，我们的目的 就是希望能学会生成概率模型?(?|?)。这里可以采用最大似然估计(Maximum Likelihood Estimation)方法：一个好的模型，应该拥有很大的概率生成真实的样本? ∈ ?。如果我们的 生成模型?(?|?)是用?来参数化，那么我们的神经网络的优化目标是： max ? ? (?) = ∫ ?(