Summary

This section elaborated on everything related to input. The concept of a buffer has been discussed and how it is filled. It can be filled automatically by a chain of byte loaders and lexatom loaders, or manually using a Gavager, a Feeder, or by pointing to memory. In the frame of the discussion of byte loading, it has been discussed how the lexer may be adapted to different input APIs. In the frame of lexatom loaders, the subject of encoding conversion has been explored using existing converters such as GNU’s IConv, or IBM’s ICU. Finally, a section presented means to navigate in the input stream on the level of lexatoms.