从零开始用 Go 实现 Lexer & Parser 何源 yuan@liulishuo.com 流利说 流利说 Help everyone become a global citizen! aithub/lingochamp 流利说 Empower everyone to achieve their full potential 何源 yuan@liulishuo.com Platform DOWNLOAD A FRESH COPY. 撸起袖子！ — 撸起袖子加油干！ 三步走 写一个 Parser: course file -> pb file • 写一个 Lexer • 串起来 1. 定义格式 ===== Group 1526034270606411 [TYPE Pre] ID: 1766229869997200808 DF1: Pic(i func(*lexer) stateFn The run loop // run lexes the input by executing state functions // until the state is nil. func run() { for state := startState; state != nil; { state = state(lexer)

构造可组合的解析器 1. // V 代表解析成功后获得的值 2. // Lexer[V] == (String) -> Option[(V, String)] 3. type Lexer[V] (String) -> Option[(V, String)] 4. 5. fn parse[V](self : Lexer[V], str : String) -> Option[(V, String)] 最简单的解析器 ☐ 判断下一个待读取的字符是否符合条件，符合则读取并前进 1. fn pchar(predicate : (Char) -> Bool) -> Lexer[Char] { 2. Lexer(fn(input) { 3. if input.length() > 0 && predicate(input[0]) { 4. let symbol: Lexer[Char] = pchar(fn{ 2. '+' | '-' | '*' | '/' | '(' | ')' => true 3. => false 4. }) 5. let whitespace : Lexer[Char] = pchar(fn{

parentheses. This is used to figure out if an empty tuple is a valid expression or not. #### class jinja2.lexer.TokenStream(generator, name, filename) A token stream is an iterable that yields Tokens. The parser expect(expr) Expect a given token type and return it. This accepts the same argument as jinja2.lexer.Token.test(). look() Look at the next token. next_if(expr) Perform the token test and return the Got n tokens ahead. skip_if(expr) Like next_if() but only returns True or False. class jinja2.lexer.Token Token class. ## lineno The line number of the token ## type The type of the token. This string

string="">: 41 EQL { lexeme_to_token($lexer, $1) } 42 | NEQ { lexeme_to_token($lexer, $1) } 43 | EQL_REGEX { lexeme_to_token($lexer, $1) } 44 | NEQ_REGEX { lexeme_to_token($lexer, $1) } 45 ; 46 47 /* 48 * = parse_str_radix($lexer.span_str($span)); 54 Ok(Expr::from(num?)) 55 } 56 ; 57 58 string_literal -> Result: 59 STRING { Ok(Expr::from(span_to_string($lexer, $span))) } 60 ;

0 码力 | 39 页 | 6.95 MB | 2 年前

3

"name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" }, "nbformat": 4, "nbformat_minor": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.1" } } { "cell_type":

"name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } } } }, "nbformat": "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" } }, "nbformat":

"name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } } } }, "nbformat": "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" } }, "nbformat":

"name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" }, "nbformat": 4, "nbformat_minor": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.1" } } { "cell_type":

- Mixing.patterns and co-patterns - Core language - Grammar - Syntax overview - Lexer - Parser • Concrete Syntax • Nice Concrete Syntax • Abstract Syntax • Internal Syntax translating an $ * $ .agda file into an executable has several stages: *.agda file ==[ parser (Lexer.x + Parser.y)]==> Concrete syntax ==[ nicifier (Syntax.Concrete.Definitions)]==> 'Nice' detail: ## Lexer Lexical analysis (aka tokenization) is the process of converting a sequence of characters (the raw *.agda file) into a sequence of tokens (strings with a meaning). The lexer in Agda is

- Mixing.patterns and co-patterns - Core language - Grammar - Syntax overview - Lexer - Parser • Concrete Syntax • Nice Concrete Syntax • Abstract Syntax • Internal Syntax translating an $ * $ .agda file into an executable has several stages: *.agda file ==[ parser (Lexer.x + Parser.y) ]==> Concrete syntax ==[ nicifier (Syntax.Concrete.Definitions) ]==> 'Nice' detail: ## Lexer Lexical analysis (aka tokenization) is the process of converting a sequence of characters (the raw *.agda file) into a sequence of tokens (strings with a meaning). The lexer in Agda is