Token (translator construction)

from Wikipedia, the free encyclopedia

A token ( Art .: "That"; Pl .: 'Tokens') is a character string that is assigned a type by a formal grammar . The token forms the basic lexical unit for the parser . In the grammar of the parser, a token corresponds to a terminal symbol .

In the trivial case, tokens are the characters of the original input : For example, the letter A is recognized as a type A token. Several characters can be combined into tokens in the course of a lexical analysis . The character string on which a token is based is called a lexeme . Examples:

Token
Lexeme Type
123 number
foobar Identifier
begin keyword