FA as Nested Cases for Scanners
From M.L. Scott's book:
Outline of Pascal scanner written as explicit FA, using nested
cases in a loop.
Groups chars into tokens, removes comments and ``white space'', saves text where needed (e.g. identifiers, strings, numbers), adds line or col numbers for better err. msgs.
Ad Hoc Code, often used in production, fast, efficient.
FA or RE → FA: easy to write code from automatically, in fact can automate much of the process.
Code: handwritten: generate switch (case, goto) statements, as seen rather earlier.
OR: automatically generated: use tables and a driver (as in lex/flex.) Or maybe produce tables for handwritten driver.
Outer cases covers FA states. Inner cases cover transitions out of each state: most go to new state, some return with current token.
Two additions to pure FA: keywords and ``peek-aheads'' (e.g. for 3.14 vs 3..14 in Pascal).
Key (reserved) words look like identifiers. Telling them apart with scanner takes lots of states. So easiest to depart from formalism, ignore distinction until token is generated, then look it up (hash or trie) to see if it's a keyword.
We've seen the FORTRAN peek-ahead problem DO100I=1.10. If not too common, can treat these problems as special case hacks. But some languages just need more lookahead: scanner assumes longer token possible but remembers shorter ones along the way, buffers chars. Good syntax design can minimize these complications!
From M.L. Scott's book:
Outline of Pascal scanner written as explicit FA, using nested
cases in a loop.
Represent FA by its two-dimensional (state and character) transition table. Table entries say whether to transition to new state and if so to which one, whether to return a token or error.
A second table indicates for each state whether we might be at the end of a token (and if so which). That lets us notice when pass a state that might have been end of a token can back up if hit error later on.
Other issues: Lexical error handling (often toss invalid token, skip to possible start of next, and keep scanning, thus punting to the parser for bad syntax. Pragmas are significant comments that control or hint to compiler (turn profiling on, put variable in register, routine not recursive...).
From M.L. Scott's book:
