Stream Navigation

Section sec_lexical_analysis introduced the concept of a lexical analysis step as the procedure that determines a token for a given position in the input stream. The position of a lexatom is given as a natural number. Navigating in the input stream means to set the position of the initial lexatom of the subsequent lexical analysis step. The interactions to implement navigation on lexatom-index level can be complex in the context of byte loaders and converters. The complete handling of this, however, happens behind the facade of very few interface functions, as they are the following:

pos_type tell(Lexer* me);
void     seek(Lexer* me, const pos_type LexatomIndex);
void     seek_forward(Lexer* me, const ptrdiff_t LexatomN);
void     seek_backward(Lexer* me, const ptrdiff_t LexatomN);
void     undo(Lexer* me);
void     undo_n(Lexer* me, ptrdiff_t LexatomN);

where pos_type is a shorthand for QUEX_TYPE_STREAM_POSITION. In C++ by the member functions have the same form, except that the lexer does not need to be passed as first argument.

The function tell() returns the position from where the next analysis step will take off. The seek functions set the position to a specific value. seek() sets an absolute position starting from the beginning. seek_forward() increments the current position by a certain number. seek_backward() decrements it.

undo() resets the input pointer to the beginning of the currently matched lexeme. undo_n() goes a given number of lexatoms backwards. The undo-functions differ from the seek functions. They reset a possible terminating zero in the buffer and set the terminating zero to the newly set position. Upon the beginning of the next analysis step, the zero-ing is undone automatically.

The result of a lexical analysis step is uniquely defined by the language, i.e. the mode in which it acts, and the start position in the input stream. By setting the lexer at a given position in a given mode allows to dynamically analyse sub-sequences of the input stream. This is, for example, interesting for text editors that re-highlight content which has been changed by the user.