TYPE 3 Regular Expressions FSAs Again...
S → aB | λ
A → bB | cC
B → bB | bS
C → cC | cS
((abb*)|(acc*))*
S → aB | λ
A → bB | cC
B → bB | bS
C → cC | cS
((abb*)|(acc*))*
Every production is of form:
A → %OK
A → aBAcdE % OK
Ab → aBAcdE % BAD: Context SENSITIVE LHS.
Allows recursion (needs stack, so PDA): Nonterminals defined in terms
of themselves. Rule: Only single nonterminal, no terminals on LHS of
productions. Supports structure like nested (()). G=(S,N,T,P):
(Start, NonTerm, Term, Prods).
e.g. in BNF:
expr → id | number | - expr | ( expr)
| expr op expr
op → + | - | * | /
(note paren-matching trick). More powerful than RE: easy to do
concat.
and alternate, thus Kleene closure, but can also define things in
terms of themselves, which RE's can't.
BNF sometimes augmented with new useful symbols: e.g.
id-list → id (, id)*
(e.g. of new id, Kleene+, and note | is not needed.)
Parser doesn't distinguish one terminal symbol (e.g. id)
from another, but semantics has to, so scanner saves spellings.
<NP> → the <Noun>
<NP> → the <Noun> <NP> <Verb>
the man
the boy the man saw
the dog the boy the man saw kicked
the flea the cat ... saw kicked chased painted shot scratched bit...
Starting with grammar's start symbol and rewriting non-terminals
using the rules lets you generate (derive) syntactically valid strings
of terminals and non-terminals, called sentential forms, which are the
yield of the derivation. Here's a rightmost or
canonical derivation.
expr → expr op expr
→ expr op id
→ expr + id
→ expr op expr + id
→ expr op id + id
→ expr * id + id
→ id * id + id
(slope) * (x) + (intercept).
Leftmost Derivation: replace leftmost non-terminal at each step.
Rightmost Derivation: replace rightmost non-terminal at each step.
Ambiguous Grammar: has multiple left- or rightmost derivations for a single sentential form.
Calls scanner to get tokens, assembles tokens into syntax tree, passes the tree to later phases of compiler for semantic analysis, code generation. The parser is ``in charge'' and hence this style is syntax-directed translation.
CFG can be a generator for language. A compiler, say, wants to recognize language:
Parsing from arbitrary CFG turns out to be
LL are often thought simpler or clearer, can be written by hand or with parser-generator tool. LR grammars are a larger class, possibly have more intuitive structure. esp. arith. exps. LR almost always created automatically, more common for commercial products.
Represent a derivation as a parse tree, with root the start symbol, internal nodes are nonterminals, leaves are terminals, children of node T are symbols on RHS of some production for T in grammar. Every tree represents a string (or yield generated by grammar. The simple grammar
expr → id | number | - expr
| ( expr ) | expr op expr
op → + | - | * | /
allows two possible parse trees for
slope * x + intercept.
It is thus an ambiguous grammar. There are infinitely many
grammars for any CFL, and ambiguous grammars are much less useful in
CS than unambiguous ones.
Also to be avoided are grammars with useless symbols -- nonterminals that can't generate any string of terminals (e.g. A → B.) or terminals that never appear in any yield. Grammars may be put into canonical form, which simplifies applications and analysis, but we won't follow that up.
Rightmost and Leftmost derivations: which is which?
Relevant for scanner-parser project!
expr → term | expr add-op term
term → factor | term mult-op factor
factor → id | number |
- factor | ( expr )
add-op → + | -
mult-op → * | /
This unambiguous grammar also captures arithmetic precedence in the way the productions use each other, and it captures the usual left-associativity by building sub-exprs to the left of the operator. Note: precedence is not a property of CFG, it's a property of the semantics we choose to apply to the strings. BUT if the grammar reflects what we want, easier for compiler!
Top-Down (LL) Parsers
Bottom-up (LR Parsers):
Like many forms or search, parsing can be done tops-down (LL) or bottoms-up (LR). We'll be concerned with LL (predictive) parsers in 173, in fact recursive descent. As in scanning, there are table-driven parsers too.
The grammar may be nasty and require a long look-ahead to do a parse. The length of the lookahead (in tokens) appears in parens after the grammar type, hence LL(1) or LR(2). (1) is most common.
One of simplest practical methods (more importantly, it's used in our assignment).
Each non-terminal has associated procedure whose goal is to read a sequence of input characters that can be generated by that non-terminal. Also return a pointer to the parse tree for that non-terminal.
Like the cases in a scanner, each procedure structure mirrors its production structure. It attempts to ``match'' the RHS of some production.
To match a terminal symbol, compare it to input. Succeed if agree, consume the symbol from input.
To match non-terminal, call its corresponding procedure (may be a recursive call, hence the name of the technique).
Rec. desc. parsing hard to change if change the grammar a little.
Replace the implicit stack of procedure calls with an explicit stack.
Hugely important implementation of parsing: both commercial and research/experimental: separates the grammar from the code.
In a derivation from a grammar, an epsilon production removes nonterminal from string being generated: Balanced ()s:
P → ( P )
P → P P
P → ε (or more commonly P →)
or
P → ( P ) | P P | ε
Note this is ambiguous grammar: rule 2 once, then rule 1 on either P,
then rule 3 on other. Unambiguous one is
P → ( P ) P | ε
Find main:
<program> → <letter*>
m a i n <letter*>
<letter*> → <letter> <letter*>
| epsilon
<letter> → A | B | ... | Z | a | b ... | z
Pascal IDs:
<id> → <L> <LorD*>
<LorD*> → <L> <LorD*>
| <D> <LorD*> | ε
<L> → A|B| ... |Z|a|b ... |z
<D> → 0|1|2|3|4|5|6|7|8|9
Pascal Real Numbers:
<real> → <dig> <dig*>
<dec part> <exp>
<dig*> → <dig> <dig*> | ε
<dec part> → '.' <dig> <dig*> | ε
<exp> → 'E' <sign> <dig> <dig*> | ε
<sign> → + | - | ε
<dig> → 0 | ... | 9
C compound statements (more or less):
<compound stmt> → { <stmt list> }
<stmt list> → <stmt> <stmt list>
| ε
<stmt> → <compound stmt>
<stmt> → id : <stmt>
<stmt> → if ( <expr> ) <stmt>
<stmt> → if ( <expr> ) <stmt>
else <stmt>
<stmt> → while ( <expr> ) <stmt>
<stmt> → do <stmt>
while ( <expr> );
<stmt> →
for ( <stmt> <expr> ; <expr> )
<stmt>
<stmt> → switch ( <expr> )
{ <mult case stmt> }
<mult case stmt> → <case stmt>
<mult case stmt> | ε
<case stmt> → case <expr> : <stmt>
| default: <stmt>
<stmt> → break ; | continue ; | ;
<stmt> → return <expr> ; | goto <id>
Actions:
Push as yet unseen portions of productions onto a stack
Use:
which we'll learn about later.
Table
Code->Scanner->Tokens->Driven<->Stack
Parser
^
|
Grammar->Parser-----> Parse
Rules Generator Table
The strategy is to expand (parse) the next (leftmost) nonterminal: ideally, we don't immediately drop into an infinite loop, and we don't have a choice of how to expand it (i.e. the grammar is unambiguous). Thus ...
!! No left recursion or common prefixes !!
A grammar is left recursive if there exists A in NonTerms
such that
A → A σ for some string σ .
A grammar has common prefixes if there are productions of the
form
A → B δ
A → B μ
Transform the grammar to remove left recursion:
A → Aa1 | Aa2|...| b1 | b2 | ...
⇒
A → b1A' | b2A'...
A' → a1A' | a2A'...| ε,
where A' is new nonterminal.
This substitution of right-recursion for left means that the resultant parse tree does not enjoy the "built-in left associativity" property for arithmetic expressions that left recursion gives.
Want no embarrassing choice of what to do next. Factoring the
expression means deferring the choice of productions until after we
deal
with the
common prefix.
A → B δ
A → B μ
⇒
A → B Btail
Btail → δ | μ
Recursive descent parsing:
Here's a version of our unambiguous grammar for (E)xpressions,
with (T)erms and (F)actors.
E → E + T | E - T | T
T → T * F | T / F | F
F → ( E ) | number
But it's a cess-pool of left recursion and can't be used in top-down parsing!
BIG LESSON: there are an infinite number of grammars for any language and they're not all equally good for everything. (motivates Canonical Grammars). What's wrong here is the ``left recursion''. The Rule for E predicts an E, which means we would call the rule associated with E, etc. forever.
MORAL: language design is important (e.g. minimize lookahead) AND grammar choice is important!
Different Grammar -- simply use the left recursion and common prefix
removal rewriting tricks
seen a while back. (Details coming up!)
E → T Etail
Etail → + T Etail | - T Etail | ε
T → F Ttail
Ttail → * F Ttail | / F Ttail | ε
F → ( E ) | num
This works top-down. Create procedures for the nonterminals, like
procedure E
T()
Etail()
For a rule that has to examine the input:
Assume a global variable next_token and a utility routine
match:
procedure Etail
switch next_token
case +
match(+)
T()
Etail()
case -
match(-)
T()
Etail()
default
return
procedure match(expected)
if next_token != expected
error()
else
next_token = scan()
% read next terminal symbol
% into global variable
F → ( E ) | num
procedure F
switch next_token
case (
match(()
E()
match())
case num
match(num)
default
error()
Here the default case is an error, whereas the default case in Etail and Ttail is to return without doing anything. Etail and Ttail have epsilon productions (OK to have empty subtree under them in the parse tree.) F has to be either a number or a parenthesized expression, not epsilon.
Note the check inside match is useful if, for instance, there is no ``)'' for the 2nd call to match --- an error.
Could make eof a sort of token so match can help us with
error detection here...
procedure main
E
match(eof)
Then, with input 1 +(2*3)/4:
next_token = "1"
Call E
Call T
Call F
next_token = "+"
/* Match 1 with F */
Call Ttail
/* Match ε */
Call Etail
next_token = "("
/* Match + */
Call T
Call F
/*Match (, looking for E)*/
next_token = "2"
Call E
Call T
Call F
/* Match 2 with F */
next_token = "*"
Call Ttail
/* Match * */
next_token = "3"
Call F
/* Match 3 with F */
next_token = ")"
Call Ttail
/* Match epsilon */
Call Etail /* Match epsilon */
next_token = "/" /* Match ")" */
Call Ttail
next_token = "4" /* Match "/" */
Call F
/* Match 4 with F */
next_token = eof
Call Ttail /* Match epsilon */
Call Ttail /* Match epsilon */
Call Etail /* Match epsilon */
/* Match eof */
In procedure Etail and Ttail, we match one of the productions with an arithmetic operator if we see such an operator in the input; otherwise we simply return, (choosing the epsilon production.)
Here we only choose the epsilon production if the next_token doesn't match the first token on the right hand side of the production.
We never attempt to read beyond the end marker (eof), which is matched only at the end of an outermost expression. In all other circumstances, the presence of the end marker signals a syntax error.
Our recursive-descent parser only determines whether or not the input string is in the language of the grammar; it does not give the structure of the string according to the grammar. We could easily build a parse tree incrementally during parsing.
MORE on the web...Google recursive descent parser, Danno!
Recursive descent only works for CSGs with disjoint PREDICT sets for productions that share a common left hand side. Such CFG's are called LL(1). Practically, programming languages are usually LL(1). So we need to learn about PREDICT sets.
We have seen how these
problems keep grammars from being LL(1):
Left Recursion: A → A X1...Xm.
Common Prefix: B → C X1...Xm,
B → C X1...Xm.
(Symptomatic of the trouble they'll cause a tops-down parser,
LR and CP yield
non-disjoint predict sets.)
We know how to fix these problems, and doing that derives our example grammar from the (more intuitive) non-LL(1) version.
For Expressions:
1. E → E + T
2. E → E - T
3. E → T
4. T → T * F
5. T → T / F
6. T → F
7. F → ( E )
8. F → number
Must replace left recursion with right recursion...
Converting with Left Recursion to Right Recursion is easy:
1. E → T + E
2. E → T - E
3. E → T
4. T → F * T
5. T → F / T
6. T → F
7. F → ( E )
8. F → number
But 1.--3., 4.--6. have common prefix. Factoring the common prefix means deferring decision of which production to pick until we see the common prefix.
That leads to
1. E → T Etail
2. Etail → + T Etail | - T Etail | ε
3. T → F Ttail
4. Ttail → * F Ttail | / F Ttail | ε
5. F → ( E ) | number
That's our familiar top-down grammar for expressions.
Eliminating left recursion and common prefixes is not enough to guarantee LL(1). Here's an ingeniously snotty grammar for the simple language
``all strings of a's followed by equal number of b's or by equal
number of c's''
(
G → a B b
G → a C c
B → a B b
B →
C → a C c
C →
Left factoring doesn't help.
But it's not hard to find an LL(1) version:
G → B | C
B → aBb | ε
C → aCc | ε
To write our parser, we need to answer this:
Q: What are the cases in a production's case statement??
Example:
A → B C Ω
B → bB
B → ε
C → c
C → ε
Regular language:
Regular Grammar? (There's a moral here!)
Recursive Descent Parser has Functions like
A()
case: b
match(b)
B();
case: c
C();
case: OMEGA
match(OMEGA)
otherwise:
error
Likewise, for
B → bB
B → ε
Get
B() % Parse Nonterminal B
case: b %cases are over nxt token
match(b) % accept and delete
B() % recurse
case: c, OMEGA % B -> eps! c,
% OMEGA are from
return % C-> c and C -> eps.
otherwise:
error
and for
C → c
C → ε
get
C()
case: c
match(c)
case: OMEGA
return
otherwise
error
PROBLEM: How figure out what goes in the cases???
What tokens should be in the arms of the case statements in a Rec. Desc. parser? It's a token X that predicts the production, but they don't just jump out at you from the grammar.
The RHS of production, recursively expanded, might yield string beginning with X.
The RHS may yield nothing ( ε or string of nonterminals that recursively give ε), and X may begin the yield of what comes next in the sentential form.
What we need is a PREDICT set of tokens that predicts what production could be responsible for the current token. PREDICT can be made from the FIRST and FOLLOW token sets, which can be extracted automatically from an LL(1) CFG.
In fact, FIRST sets are often enough. But in all but the simplest grammars we need to know them to write a rec. desc. parser. Thus the class was given FIRST SETS in the source code directory for the 2006 parser assignment to save deriving them. Our sets are a lot simpler!
You're the parser: you need to figure out what's going on (what
production
you're parsing), given the stream of tokens from the scanner.
Here's a grammar:
S → A B C
A → a | ε
B → D ! b | ε
C → c
D → d
Here's a related tree: the arc-connected productions mean AND (in
order),
the straight arcs are OR (pick any production).
We see:
1. We can think of the dashed triangle subtrees disappearing from the tree if
the epsilon (emit nothing) is chosen by the root (LHS of production).
2. After that, the possible token streams are ordered choices of terminals.
3. So the possible sentences are:
adc, abc if A,B,C productions all used
ac if B → ε
dc, bc if A → ε
c if A → ε, B → ε
Possible sentences are: adc, abc, ac, dc, bc c. We're the parser; We
see tokens one at a time and from them must identify which production
we are parsing. e.g.
see: q, x, <eof>, etc. so: ERROR
see: a, so: A
see: d or b, so: B
see: c, so: C.
That is, FIRST(S) is {a, d, b, c}. These are all we need to make up
the three cases for what production to parse: a, (b or d), c.
Visually we can find them in the tree by imagining the dashed
triangles
to vanish if epsilon is taken in the generation of the incoming sentence.
Likewise, FOLLOW(A) is seen to be {d, b, c}: c means B → ε and we're in the C production, else we're in B.
Consider a grammar with a production like
(1) A → x y B z A C w.
There could be other A → productions in the grammar.
If we are trying to parse (find, explain) an A in a Rec. Desc. parser, we are in the A() function. If it is an LL(1) parser, the current token determines which production we're in. Thus If A() finds an x, it knows it is in production (1), and we say x is in the FIRST set of A.
Suppose the grammar has A → ε. Say we're parsing production (1) as before, and we get to the z, match that, and call A(). But the string we're parsing was generated with the A → ε production at that point.
So there's no evidence for A except that evidence for C is evidence for A → ε. So we should look for the first token produced by some C → production (!), and that token FOLLOWs A. That's where things get non-obvious.
Hence the FIRST and FOLLOW sets.
Algorithm on page 76 of M. Scott's e-Reserve reading (not the BB Course Materials nor the equivalent link in 173 Schedule).
FIRST(X) is the set of terminal symbols that begin strings derived from X
To build FIRST(X):
For a non-terminal A, FOLLOW(A) is the set of terminals that can appear immediately to the right of A in some sentential form.
To build FOLLOW(B) (α and β are arbitrary strings of terminals and nonterminals.)
FIRST sets distinguish two productions with same nonterminal on LHS: 3 steps (iterated)
Steps 1 and 2 are simply ``obvious'' facts from the grammar, then we
iterate over what we know until we don't learn any more. E.g.
...
A → ε
B → ε
...
C → A B
D → C A B
A and B can generate ε; 2nd pass figure out C can too, as can D (found on 3rd pass), etc.
...
1. A → b C D
2. B → c D e
3. C → B A d
...
b is in FIRST(A) and c is in FIRST(B), immediately from 1 and 2. Pass 2 would show c is in FIRST(C). Continue working through each RHS adding tokens to the FIRST set of RHS until we learn nothing new.
Here use FIRST for individual symbols to get FIRST for whole RHS:
if
A → X1 ... Xm
we need FIRST(X1...Xm).
Start with X1:
Etc. If X1,X2,...Xm can all produce ε, then we should predict A → X1 ... Xm if the lookahead symbol can come after an A in some line of the derivation: that brings up FOLLOW sets.
E → T Etail
Etail → + T Etail | - T Etail | ε
T → F Ttail
Ttail → * F Ttail | / F Ttail | ε
F → ( E ) | num
Then:
First(E) = First(T Etail)
First(T Etail) = First(T)
First(T) = First(F Ttail)
First(F Ttail) = First(F) = {(,num}
A production's PREDICT set is the set of all initial (first) tokens derivable
from it.
Grammar:
(1) A → B C Ω
(2, 3) B → bB | ε
(4, 5) C → c | ε
FIRST(A) = {b (1,2), c (1,3,4), Ω (1,3,5)}
FOLLOW(A) = { ε }
PREDICT(A → BC) = { b, c, Ω }
FIRST(B) = {b (2), ε (3) }
FOLLOW(B) = { c (2 3 4), Ω (2,3,5) }
PREDICT(B → bB) = { b }
PREDICT(B → ε) = { c, Ω }
FIRST(C) = {c (4), ε (5) }
FOLLOW(C) = { Ω }
PREDICT(C → c) = { c }
PREDICT(C → ε) = { Ω }
Claim: Cases in A() are
{b, c, Ω}.
in B(): are {b, c, Ω}
in C(): {c, Ω}.
We claimed that the cases
in A() are
{b, c, Ω}.
In B(): are {b, c, Ω}
In C(): {c, Ω}.
B() % Parse Nonterminal B
case: b % cases are over the next token
match(b) % accept and delete
B() % recurse
case: c, Ω % B -> eps! c, Ω from
return % C-> c and C -> eps.
otherwise:
error
Also, and obviously: If ``some token belongs to the PREDICT set of more than one production with the same left-hand side, then the grammar is not LL(1), because [M. Scott]'' the single token we're seeing could be produced by more than one production. In a recursive descent parser, it would appear in more than one branch of the case statement that deals with that left-hand-side (non-terminal).
If there are multiple choices, the grammar is not LL(1) (predictive).
Scott's Chapter 2 has First, Follow, Predict for an entire calculator language, nice treatment (pp. 72-76).
Algorithm to compute F,F, and P relies on these formal definitions;
( if α → * ε then { ε } else φ)
( if S → * α A then { ε} else φ)
(if α → * ε then FOLLOW(A) else φ)
If grammar has disjoint PREDICT sets it is LL(1): otherwise not: there is a predict-predict conflict.
In building rec. desc. parser... say G, H produce ε in:
A → B c D
A → e f
A → G H
Parsing routine:
A() {
switch (next_token) {
case First(BcD):
B
match(c)
D
case e:
match(e)
match(f)
default: % if next not in First(GH)
G % or Follow(A), just get
H } } % syntax error later...
A Grammar G is LL(1) if and only if, for all
non-terminals A, each distinct pair of
productions
A → a and A → b satisfy the
condition
FIRST(a) ∩ FIRST(b) = φ, i.e.,
For each set of productions
A → a1 | a2 | . . .
| an:
Expect linear since no back-up (look-ahead).
Instructions inside main loop are bounded by constant (function of symbols on RHS)
How many times does the main loop execute?
stmt --> if condition then-clause
else-clause
| other_stmt
then-clause → then stmt
else-clause → else stmt | e
if C1 then if C2 then S1 else S2 --- the else can be paired with either then. Neither LL nor LR. Rather famous problem.
stmt → balanced-stmt | unbalanced-stmt
balanced-stmt →
if condition then balanced-stmt
else balanced-stmt | other-stmt
unbalanced_stmt → if condition then stmt
| if condition then balanced-stmt
else unbalanced-stmt
OR: Use special disambiguating rules, like ``use production that occurs first in case of conflict''.
OR: Use a better syntax (like elseif).
For pure parser, simply halt without accepting.
In a compiler, you should print a nice helpful diagnostic message and then do something potentially really complicated (not covered here) to patch up the parse tree and the input and continue looking for further errors. OR you could do something really stupid like PL/I did and try to ``correct'' errors...
It is sometimes possible to avoid using FIRST and FOLLOW in parsing code.
Consider the program for non-terminal A()
If there is only one non-epsilon production in the grammar with
A as a left-hand side, say:
A → B C D
One could figure out the FIRST and FOLLOW for B, C and D and
explicitly
build in all the cases in A's program. But why bother? Why not do this?:
A()
B();
C();
D();
end
Not using FIRST: If B(),C(), or D() doesn't like
what it sees,
let it complain by calling a parse error.
Sure, A() could have done a lot of work
to create cases for when it seemed B() was appropriate to call or
if B could derive ε then whether it seemed like
C() was the thing to do, etc. etc.
but not checking just costs some more
recursion.
Not using FOLLOW:
Now consider adding the production
A → ε to the grammar,
so there are now two productions with A as LHS, one
being the ε production. Nothing has to change in
A() (!).
If BCD is matched, A() is happy, and if not A may have produced ε -- that's enough excuse for A() to declare success and return upward. What's coming up is either grammatical or not. If it's not, sometime later the parser will fail to match a token: that's where it finds the error.
Sure, FOLLOW(A) would tell us that A "really" has produced ε, but this version of A() just assumes it has and kicks any possible problems upstairs.
If a production really needs a switch statement, we
need a case for the
ε production,
but it can always be "accept and don't match
anything":
default:
break;
More than one production with a non-ε RHS actually do need
the FIRST sets:
A → B
A → D
We need to know whether to call B or D, so we need their FIRST sets.
Also, if we have something like
B → aC
D → aD
We would not be LL(1): two productions for A have the same
FIRST sets.
Last: Our expression grammar is so simple the FIRST sets can be figured out easily "by inspection": + and - are the cases in ETAIL, * and / are the cases in TTAIL, and ( and <number> are the cases in FACTOR.
Transform the grammar to remove left recursion:
A → Aa1 | Aa2|...| b1 | b2 | ...
⇒
A → b1A' | b2A'...
A' → a1A' | a2A'...| ε,
where A' is new nonterminal.
This substitution of right-recursion for left means that the resultant parse tree does not enjoy the "built-in left associativity" property for arithmetic expressions that left recursion gives.
Various declarations from a .h file for CB's parser... this is about creating parse trees (of ptnodes).
typedef enum
{
EXPR, TERM, TERM_TAIL, FACTOR,
FACTOR_TAIL, IDENTIFIER, NUM }
node_type;
/* only for factor tails and term tails */
typedef enum
{
TIMES, DIVIDE, PLUS, MINUS, NONE}
op_type;
typedef struct ptnode{
char * ptname;
node_type pttype;
char * ID_name;
double ptvalue;
op_type optype;
struct ptnode * first; /* <= 3 descs */
struct ptnode * second;
struct ptnode * third;
} * ptnode_t;
Using the facts above, here a couple of CB's functions used by the
parser: first, a program for a nonterminal with a single production.
It must produce a parse-tree node, of course.
static ptnode_t parse_expr()
{
ptnode_t res; % making this node
ptnode_t ter; % which points to these
ptnode_t tertail; % two nodes
/* printf("\n parse_expr\n"); */
ter =parse_term();
% each prog. rtns a parse-tree node
tertail = parse_term_tail();
res = make_ptnode("expr", EXPR,"",
0.0,NONE,ter, tertail, NULL);
return(res);
}
Next, a version of the FACTOR() program; Two cases are tokens that aren't terminal characters: one is an identifier (its value is a string of chars in the input line) and one a number (string of digit chars in input line). These non-terminals are dealt with in the original scanner code (e.g. with got_dot(), got_dig(), got_ft_dot()), which CB did not change from the original.
Again, this code makes a parse-tree node (with lots of debugging
info along the way). It also shows one way to evaluate a number in
the input and put the resulting float in the parse tree, and how
to copy a string (here, an identifier name).
static ptnode_t parse_factor()
{
ptnode_t res;
char * idcopy;
char * idloc;
location_t loc;
double anum;
printf("\n parse_factor\n");
switch (tok.tc) {
case T_LPAREN:
match(T_LPAREN);
res = parse_expr();
match(T_RPAREN);
break;
case T_IDENTIFIER:
/* printf("\n ID found\n"); */
loc = tok.location;
/* cbprint_location(&tok);
print_token(&tok); */
idloc = (char *)
&(loc.line->data[loc.column]);
idcopy = strndup(idloc,tok.length);
/*printf("\n ID copied to: %s\n",
idcopy);*/
match(T_IDENTIFIER);
res = make_ptnode("an ID",
IDENTIFIER, idcopy, 0.0, NONE,
NULL, NULL, NULL);
break; /* accept */
case T_NUM:
/* printf("\n NUM found\n"); */
loc = tok.location;
/* cbprint_location(&tok);
print_token(&tok); */
idloc = (char *)
&(loc.line->data[loc.column]);
idcopy = strndup(idloc,tok.length);
sscanf(idcopy, "%lf", &anum);
/*printf("\n NUM scanned as:
%f\n", anum);*/
res = make_ptnode("a NUM", NUM, "",
anum, NONE, NULL, NULL,
NULL);
match(T_NUM);
break;
}
return(res);
}
static ptnode_t parse_factor_tail()
{
ptnode_t fac;
ptnode_t factail;
ptnode_t res;
/* printf("\n parse_factortail\n"); */
switch (tok.tc) {
case T_STAR:
match(T_STAR);
fac = parse_factor();
factail = parse_factor_tail();
res = make_ptnode("factortail",
FACTOR_TAIL, "", 0.0,TIMES,fac,
factail, NULL);
break;
case T_SLASH:
match(T_SLASH);
fac = parse_factor();
factail = parse_factor_tail();
res = make_ptnode("factortail",
FACTOR_TAIL, "", 0.0,DIVIDE,fac,
factail, NULL);
break;
default:
res = NULL;
break; /* just return */
}
return(res);}
I'm giving up on this example...too hairy. Best let sleeping dogs lie. This got out of hand and lost original point about FOLLOW sets. Never got good e.g. of that -- this turned into an e.g. of not even using FIRST sets (A->BCD). 9/25/14
To FOLLOW or not? Claim: FOLLOW sets are nice symmetric idea,
given FIRST sets, but:
FIRST sets needed if a nonterminal has more than one non-ε
production: (Which One Should I Do??).
FOLLOW set is a luxury:
a. Only useful if nonterminal has an ε
production.
b. All FOLLOW gives you is a chance to report an error slightly
earlier.
A → c c % must deal with this rhs, not
A → aAx % get mixed up with this one
A → ε
A → BCD;
FIRST(A) = {a,c,ε} ∪ FIRST(BCD) (hard?)
FOLLOW(A) = {x} ∪ FOLLOW(BCD) (hard?)
PREDICT(A → cc) = {c} (FIRST)
PREDICT(A → aAx) = {a} (FIRST)
PREDICT(A → ε) = {x} (FOLLOW)
PREDICT(A → BCD) = ???
A() % Use FOLLOW
case: c
match(c);
match(c);
case: a
match(a);
A();
match(x);
case: x
return(); % A -> e; match(x) eats it
case: PREDICT(BCD)
B()
C()
D()
otherwise:
error(); % non-FOLLOW token detected
end;
That is, ignore what can go wrong...
declare success, ignore what's coming next.
% Leave it to the parser to notice that when
% it reads next token.
A() % Don't Use FOLLOW
case: c
match(c);
match(c);
case: a
match(a);
A();
match(x);
otherwise:
return();
end;