Token (translator construction)
A token ( Art .: "That"; Pl .: 'Tokens') is a character string that is assigned a type by a formal grammar . The token forms the basic lexical unit for the parser . In the grammar of the parser, a token corresponds to a terminal symbol .
In the trivial case, tokens are the characters of the original input : For example, the letter A is recognized as a type A token. Several characters can be combined into tokens in the course of a lexical analysis . The character string on which a token is based is called a lexeme . Examples:
Token | |
---|---|
Lexeme | Type |
123 | number |
foobar | Identifier |
begin | keyword |