Token Identifiers and the Token Class

There are two sections two customize the tokens to be used as output for lexical analysis. The section token defines token identifier names and optionally associates them with numeric values. If the default token class is not sufficient, a customized token class can be defined in the section token_type.

Token Identifier

The keyword token opens a token identifier definition section. A token identifier is an integer value representing a type of lexeme. In a program they are, usually, referred to by named constants. The token section associates names of token identifiers with numerical values. In the following, the term ‘token-id’ shall comprise the terms token identifier and token identifier name.

The token section is optional [#fx]_. It contains a list of definitions separated by ;.

token {
    FUNCTION;
    OP_MINUS = 0x4711;
    OP_PLUS  = 0x4712;
}

Note

Token identifiers in the token section are specified without prefix. By default the prefix is QUEX_TKN_. It can be adapted with the command line option --token-id-prefix. This option also provides a means to place token ids in a specific C++ name space. For example:

> quex ... --token-id-prefix example::bison::token::

defines a prefix which consist of a name space reference. The setup

> quex … –token-id-prefix example::bison::token::TK_

specifies further that only tokens in the given name space are considered which start with TK_.

A definition starts with a token identifier name. Here they are FUNCTION, OP_MINUS, OP_PLUS. A spefic numeric value is assigned by adding a = followed by a number (sec:basics-number-format:). By default, numeric values are assigned distinctly. More on token identifier definitions is provided in sec:token-token-id-definition.

The Token Class

The keyword token_type opens a section to design a token class. Quex generates a default token class/struct containing the members id, text and number. For the case that this is not sufficient, section sec:token_class explains in detail how to customize a token class. A freely implemented external token class can be specified by --external-token-class.

Footnotes

[1]

Quex warns if token identifers are sent that have not been defined in the token section. This helps to avoid dubious effects of typos where the analyzer sends unexpected token-ids. A parser might exprect a QUEX_TKN_KEY, for example, but the lexer sends an auto-generated QUEX_TKN_KEV instead. The tiny typo Y vs. V is overseen easily and the effects may be subtle.