Advanced Usage

The previous chapter was concerned with the implementation of lexical analyzer. This chapter focussed on advanced topics concerned with memory management and computation time. It not only sheds light on some internal mechanisms of the engine that quex generates, but also gives a toolset at hand that allows for the optimization of memory consumption and analysis speed.

• Token Passing Policies
  • Automatic Token Memory Management
    • Queue
    • Single
  • User Controlled Token Memory Management
    • Token Construction and Destruction
  • Remark on Token Passing
  • Summary
• User defined Token Classes
  • Customized Token Classes
  • Formal Requirements on Token Classes.
• Token Repetition
• Line and Column Number Counting
• Stamping Tokens
• Stream Navigation
• Include Stack
  • Memento Extensions
  • HOWTO
  • Infinite Recursion Protection
  • Caveats
• Indentation Based Blocks
• Indentation Parameters
• Customized Indentation Event Handlers
• Remarks on Hand-Written Indentation Management
  • Caveat
• Deriving from Lexical Analyser
• Multiple Lexical Analyzers
• Reset and Change of Input Source
• The Accumulator
• The Post Categorizer