Lecture 2
.
......
Syntactic Formalisms for Parsing
Natural Languages
Aleš Horák, Miloš Jakubíček, Vojtěch Kovář
(based on slides by Juyeon Kang)
ia161@nlp.fi.muni.cz
Autumn 2013
IA161 Syntactic Formalisms for Parsing Natural Languages 1 / 50
Lecture 2
.
...... Basic parsing methods
IA161 Syntactic Formalisms for Parsing Natural Languages 2 / 50
Lecture 2
Main points
Context-free grammar
Parsing methods
Top-down or bottom-up
Directional or non-directional
Basic parsing algorithms
Unger
CKY (or CYK)
Left-corner parsing
Earley
IA161 Syntactic Formalisms for Parsing Natural Languages 3 / 50
Lecture 2
Ambiguity in Natural Language
Notion of ambiguity
Essential ambiguity: same syntactic structure but the semantics
differ
Spurious ambiguity: different syntactic structure but no change in
semantics
There is no unambiguous languages!
An input may have exponentially many parses
Should identify the “correct” parse
IA161 Syntactic Formalisms for Parsing Natural Languages 4 / 50
Lecture 2
Ambiguity in Natural Language
Main idea of parsing
Parsing (syntactic structure)
Input: sequence of tokens
John ate an apple
Output: parse tree
S
NP VP
NAME VERB NP
ART NOUN
John ate an apple
IA161 Syntactic Formalisms for Parsing Natural Languages 5 / 50
Lecture 2
Ambiguity in Natural Language
Basic connection between a sentence and the grammar it
derives from is the “parse tree”, which describes how the
grammar was used to produce the sentences.
For the reconstruction of this connection we need
a “parsing techniques”
IA161 Syntactic Formalisms for Parsing Natural Languages 6 / 50
Lecture 2
Ambiguity in Natural Language
Word categories: Traditional parts of speech
Noun Names of things boy, cat, truth
Verb Action or state become, hit
Pronoun Used for noun I, you, we
Adverb Modifies V, Adj, Adv sadly, very
Adjective Modifies noun happy, clever
Conjunction Joins things and, but, while
Preposition Relation of N to, from, into
Interjection An outcry ouch, oh, alas, psst
IA161 Syntactic Formalisms for Parsing Natural Languages 7 / 50
Lecture 2
Formal language
Symbolic string set which describe infinitely unlimited language
as mathematical tool for recognizing and generating languages.
Topic of formal language: finding finitely infinite languages
using rewriting system.
Three basic components of formal language: finite symbol set,
finite string set, finite formal rule set
IA161 Syntactic Formalisms for Parsing Natural Languages 8 / 50
Lecture 2
Constituency
Sentences have parts, some of which appear to have subparts.
These groupings of words that go together we will call
constituents.
(How do we know they go together?)
I hit the man with a cleaver
I hit [the man with a cleaver]
I hit [the man] with a cleaver
You could not go to her party
You [could not] go to her party
You could [not go] to her party
IA161 Syntactic Formalisms for Parsing Natural Languages 9 / 50
Lecture 2
The Chomsky hierarchy
Type 0 Languages / Grammars (LRE: Recursively enumerable
grammar)
Rewrite rules α → β
where α and β are any string of terminals and non-terminals
Type 1 Context-sensitive Languages / Grammars (LCS)
Rewrite rules αXβ → αϒβ
where X is a non-terminal, and α, ϒ, β are any string of terminals and
non-terminals, (ϒ must be non-empty but strings α and β can be
empty).
Type 2 Context-free Languages / Grammars (LCF)
Rewrite rules X → ϒ
where X is a non-terminal and ϒ is any string of terminals and
non-terminals
Type 3 Regular Languages / Grammars (LREG)
Rewrite rules X → αY
where X, Y are single non-terminals, and α is a string of terminals; Y
might be missing.
IA161 Syntactic Formalisms for Parsing Natural Languages 10 / 50
Lecture 2
The Chomsky hierarchy
Type 0 > 1 > 2 > 3
according to generative power
Superior language can generate inferior language but superior
language is more inefficient and slow than inferior language.
IA161 Syntactic Formalisms for Parsing Natural Languages 11 / 50
Lecture 2
The Chomsky hierarchy






Figure : Chomsky hierarchy
IA161 Syntactic Formalisms for Parsing Natural Languages 12 / 50
Lecture 2
Context-free grammar (Type 2)
The most common way of modeling constituency.
The idea of basing a grammar on constituent structure dates
back to Wilhem Wundt (1890), but not formalized until
Chomsky (1956), and, independently, by Backus (1959).
CFG = Context-Free Grammar = Phrase Structure Grammar=
BNF = Backus-Naur Form
IA161 Syntactic Formalisms for Parsing Natural Languages 13 / 50
Lecture 2
Context-free grammar (Type 2)
CFG rewriting rule
X →ϒ
where X is a non-terminal symbol and ϒ is string consisting of
terminals/non-terminals.
The term “Context-free” expresses the fact that the
non-terminal v can always be replaced by w, regardless of the
context in which it occurs.
IA161 Syntactic Formalisms for Parsing Natural Languages 14 / 50
Lecture 2
Context-free grammar (Type 2)
G = < T, N, S, R>
T is set of terminals (lexicon)
N is set of non-terminals (written in capital letter). S is start
symbol (one of the non-terminals)
R is rules/productions of the form X →ϒ , where X is a
non-terminal and ϒ is a sequence of terminals and
non-terminals (may be empty).
A grammar G generates a language L
IA161 Syntactic Formalisms for Parsing Natural Languages 15 / 50
Lecture 2
Example1 of Context-Free Grammar
G = < T, N, S, R>
T = { that, this, a, the, man, book, flight, meal, include, read, does }
N = { S, NP, NOM, VP, DET ,N, V, AUX }
S = S
R = {
S → NP VP Det → that | this | a | the
S → Aux NP VP N → book | flight | meal | man
S → VP V → book | include | read
NP → Det NOM AUX → does
NP → N
VP → V
VP → V NP
}
IA161 Syntactic Formalisms for Parsing Natural Languages 16 / 50
Lecture 2
Example2 of Context-Free Grammar
R1: S -> NP VP R13: DET -> his|her
R2: NP -> DET N R14: DET -> the
R3: NP -> NP PNP R15: V -> eat|serve
R4: NP -> PN R16: V -> give
R5: VP -> V R17: V -> speak|speaks
R6: VP -> V NP R18: V -> discuss
R7: VP -> V PNP R19: PN -> John|Mark
R8: VP -> V NP PNP R20: PN -> Mary|Juliette
R9: VP -> V PNP PNP R21: N -> daugther|mother
R10: PNP -> PP NP R22: N -> son|boy
R11: PP-> to|from|of R23: N -> salad|soup|meat
R12: DET -> an|a R24: N -> desert|cheese|bread
R25: ADJ -> small|kind
Simplified example of CFG = GD
IA161 Syntactic Formalisms for Parsing Natural Languages 17 / 50
Lecture 2
Example2 of Context-Free Grammar
Using the presented grammar, we make a first derivation for
the sentence “John speaks”,
S -> GD NP VP (by R1)
S -> GD PN VP (by R4)
-> GD John VP (by R19)
-> GD John V (by R5)
-> GD John speaks (by R17)
IA161 Syntactic Formalisms for Parsing Natural Languages 18 / 50
Lecture 2
Example2 of Context-Free Grammar
Another derivation of “John speaks” from GD using rule 5
before rule 4
S -> GD NP VP
S -> GD NP V
-> GD NP speaks
-> GD PN speaks
-> GD John speaks
IA161 Syntactic Formalisms for Parsing Natural Languages 19 / 50
Lecture 2
Production Rule 3
NP -> NP PNP
Because it contains the same symbol in his left and his right,
we say that the production having this property is recursive.
IA161 Syntactic Formalisms for Parsing Natural Languages 20 / 50
Lecture 2
Production Rule 3
This property of R3 involves that the language generated by
the grammar GD is infinite, because we can create the
sentences of arbitrary length by iterative application of R3.
Test
NP -> GD NP PNP -> GD NP PNP PNP -> GD NP PNP PNP
PNP….
The son of John speaks
The son of the mother of John speaks
The son of the daughter of the daughter ….of John speaks.
IA161 Syntactic Formalisms for Parsing Natural Languages 21 / 50
Lecture 2
Production Rule 3
Last remark concerning this grammar (GD)
This grammar can generate sentences which are ambiguous.
“John speaks to the daughter of Mark”
Example
1 A conversation between John and the daughter of Mark (R7)
2 A conversation about Mark between John and the daughter
(R9)
IA161 Syntactic Formalisms for Parsing Natural Languages 22 / 50
Lecture 2
Production Rule 3
VP VP
Speaks PNP V PNP PNP
Pto NP to the D of Mark
NP the daughter PNP of Mark
IA161 Syntactic Formalisms for Parsing Natural Languages 23 / 50
Lecture 2
Commonly used non-terminal abbreviations
S sentence
NP noun phrase
PP prepositional phrase
VP verb phrase
XP X phrase
N noun
PREP preposition
V verb
DET/ART determiner / article
ADJ adjective
ADV adverb
AUX auxiliary verb
PN proper noun
IA161 Syntactic Formalisms for Parsing Natural Languages 24 / 50
Lecture 2
Parsing methods
Classification of parsing methods
Top-down parsing vs. Bottom-up parsing
Directional vs. non-directional parsing
IA161 Syntactic Formalisms for Parsing Natural Languages 25 / 50
Lecture 2
Top-down or bottom-up
Top-down parsing
The sentence from the start symbol, the production tree is
reconstructed from the top downwards
Identify the production rules in prefix order
Never explores a tree that cannot result in an S
BUT Wastes time generating trees inconsistent with the input
Bottom-up parsing
The sentence back to the start symbol
Identify the production rules in postfix order
Never generates trees that are not grounded in the input
BUT Wastes time generating trees that do not lead to an S
IA161 Syntactic Formalisms for Parsing Natural Languages 26 / 50
Lecture 2
Top-down parsing
Top-down parsing is goal-directed.
A top-down parser starts with a list of constituents to be built.
It rewrites the goals in the goal list by matching one against the
LHS of the grammar rules,
and expanding it with the RHS,
...attempting to match the sentence to be derived.
If a goal can be rewritten in several ways, then there is a choice
of which rule to apply (search problem)
Can use depth-first or breadth-first search, and goal ordering.
IA161 Syntactic Formalisms for Parsing Natural Languages 27 / 50
Lecture 2
Top-down parsing
Simulation of the operation of parser in top-down
methods
The son speaks
1 S
2 NP VP
3 DET N VP
4 4. a N VP. Fail: input begin by the. We return to DET N VP
5 the N VP
6 the daughter VP. New fail α=le N VP
……
7 the son VP
8 the son V
9 the son speaks.
……
IA161 Syntactic Formalisms for Parsing Natural Languages 28 / 50
Lecture 2
Top-down parsing
Top-down parsing example
S → NP VP
→ NAME VP
→ “John” VP
→ “John” VERV NP
→ “John” “ate” NP
→ “John” “ate” DET NOUN
→ “John” “ate” “an” NOUN
→ “John” “ate” “an” “apple”
IA161 Syntactic Formalisms for Parsing Natural Languages 29 / 50
Lecture 2
Top-down parsing
S
NP VP
(1) (2) (3)
(4) (5)
(7) (8)
(6)
NAME
S
NP VP
NAME
S
NP VP
John
VERB NPNAME
S
NP VP
John
VERB NPNAME
S
NP VP
John ate
VERB NP
ART NOUN
NAME
S
NP VP
John ate
VERB NP
ART NOUN
NAME
S
NP VP
John ate an
VERB NP
ART NOUN
NAME
S
NP VP
John ate an apple
IA161 Syntactic Formalisms for Parsing Natural Languages 30 / 50
Lecture 2
Top-down parsing
Algorithm of top-down left-right (LR) parsing
α is a primal current word, u input to be recognized.
tdlrp = main function
tdlrp (α, u)
begin
if (α = u) then return (true) fi
Α = u1……ukAΥ
while (∃A− > β) do
(β = uk+1……….uk+1
δ
) with δ = ϵ ou δ = A…
if (tdlrp(u1……uk+1
δ
Υ) = true) then return(true) fi
od
return (false)
end
IA161 Syntactic Formalisms for Parsing Natural Languages 31 / 50
Lecture 2
Top-down parsing
Problems in top-down parsing
Left recursive rules... e.g. NP → NP PP... lead to infinite recursion
Will do badly if there are many different rules for the same LHS.
Consider if there are 600 rules for S, 599 of which start with NP,
but one of which starts with a V, and the sentence starts with a
V.
Top-down parsers do well if there is useful grammar-driven
control: search is directed by the grammar.
Top-down is hopeless for rewriting parts of speech
(preterminals) with words (terminals).
IA161 Syntactic Formalisms for Parsing Natural Languages 32 / 50
Lecture 2
Bottom-up parsing
Bottom-up parsing is data-directed.
The initial goal list of a bottom-up parser is the string to be parsed.
If a sequence in the goal list matches the RHS of a rule, then this
sequence may be replaced by the LHS of the rule.
Parsing is finished when the goal list contains just the start
symbol.
If the RHS of several rules match the goal list, then there is a
choice of which rule to apply (search problem)
Can use depth-first or breadth-first search, and goal ordering.
IA161 Syntactic Formalisms for Parsing Natural Languages 33 / 50
Lecture 2
Bottom-up parsing
Let’s suppose that we have a sentence “the son eats his soup”
in the grammar GD.
Question
How we can do to verify that the word belong to the language
generated by the grammar GD and if the answer is positive to
assign a tree?
→ The first idea can be given in the following algorithms:
IA161 Syntactic Formalisms for Parsing Natural Languages 34 / 50
Lecture 2
Bottom-up parsing
Bottom-up parsing example
“John” “ate” “an” “apple”
→ NAME “ate” “an” “apple”
→ NAME VERV “an” “apple”
→ NAME VERV DET “apple”
→ NAME VERV DET NOUN
→ NP VERV DET NOUN
→ NP VERV NP
→ NP VP
→ S
IA161 Syntactic Formalisms for Parsing Natural Languages 35 / 50
Lecture 2
Bottom-up parsing
(1) (2) (3)
(4) (5)
(7) (8)
(6)
NP
John ate an apple
NAME
John ate an apple
NAME VERB
John ate an apple
NAME VERB
ART
John ate an apple
NAME VERB
ART NOUN
John ate an apple
NAME VERB
ART NOUN
NP
VP
John ate an apple
NAME
NP
VERB
ART NOUN
NP
VP
John ate an apple
NAME
NP
VERB
ART NOUN
NP
VP
S
John ate an apple
NAME
NP
VERB
ART NOUN
IA161 Syntactic Formalisms for Parsing Natural Languages 36 / 50
Lecture 2
Bottom-up parsing
Problems with bottom-up parsing
Unable to deal with empty categories: termination problem,
unless rewriting empties as constituents is somehow restricted
(but then it’s generally incomplete)
Inefficient when there is great lexical ambiguity
(grammar-driven control might help here). Conversely, it is
data-directed: it attempts to parse the words that are there.
Both Top-down (LL) and Bottom-up (LR) parsers can (and
frequently do) do work exponential in the sentence length on
NLP problems.
IA161 Syntactic Formalisms for Parsing Natural Languages 37 / 50
Lecture 2
Left-corner parsing
Left-corner parsing
Bottom-up with top-down filtering:
combine top-down processing with bottom-up processing in order
to avoid going wrong in the ways that we are prone to go wrong
with pure top-down and pure bottom-up techniques
IA161 Syntactic Formalisms for Parsing Natural Languages 38 / 50
Lecture 2
Left-corner parsing
.
Going wrong with top-down parsing
..
......
S -> NP VP
NP -> DET N
NP -> PN
VP -> IV
DET -> the
N -> robber
PN -> Vincent
IV -> died
Vincent died.
IA161 Syntactic Formalisms for Parsing Natural Languages 39 / 50
Lecture 2
Left-corner parsing
.
Going wrong with bottom-up parsing
..
......
S -> NP VP
NP -> DET N
VP -> IV
VP -> TV NP
TV -> plant
IV -> died
DET-> the
N -> plant
The plant died.
1 DET plant died
2 DET TV IV Fail
3 DET N IV OK
4 NP VP OK
5 S
IA161 Syntactic Formalisms for Parsing Natural Languages 40 / 50
Lecture 2
Left-corner parsing
.
Combining Top-down and Bottom-up Information
..
......
S -> NP VP
NP -> DET N
NP -> PN
VP -> IV
DET -> the
N -> robber
PN -> Vincent
IV -> died
Vincent died.
IA161 Syntactic Formalisms for Parsing Natural Languages 41 / 50
Lecture 2
Left-corner parsing
Now, let’s look at how a left-corner recognizer would proceed
to recognize Vincent died.
1 Input: Vincent died. Recognize an S. (Top-down prediction.)
S
vincent died
2 The category of the first word of the input is PN. (Bottom-up
step using a lexical rule.)
S
PN
vincent died
IA161 Syntactic Formalisms for Parsing Natural Languages 42 / 50
Lecture 2
Left-corner parsing
3 Select a rule that has at its left corner : NP-> PN. (Bottom-up
step using a phrase structure rule.)
S
NP
PN
vincent died
4 Select a rule that has at its left corner: S->NP VP. (Bottom-up
step.)
5 Match! The left hand side of the rule matches with S, the
category we are trying to recognize.
S
NP
PN
vincent died
VP
IA161 Syntactic Formalisms for Parsing Natural Languages 43 / 50
Lecture 2
Left-corner parsing
6 Input: died. Recognize a VP. (Top-down prediction.)
7 The category of the first word of the input is IV. (Bottom-up
step.) S
NP
PN IV
vincent died
VP
8 Select a rule that has at its left corner: VP->IV. (Bottom-up step.)
9 Match! The left hand side of the rule matches with VP, the
category we are trying to recognize.
S
NP
PN IV
vincent died
VP
IA161 Syntactic Formalisms for Parsing Natural Languages 44 / 50
Lecture 2
Left-corner parsing
What is a left-corner of a rule:
the first symbol on the right hand side. For example, NP is the left
corner of the rule S → NP VP, and IV is the left corner of the rule VP
→ IV. Similarly, we can say that Vincent is the left corner of the
lexical rule PN → Vincent.
IA161 Syntactic Formalisms for Parsing Natural Languages 45 / 50
Lecture 2
Left-corner parsing
What is a left-corner of a rule:
“Predictive” parser : it uses grammatical knowledge to predict
what should come next, given what it has found already.
4 operations creating new items from old:
“Shift”, “Predict”, “Match” and “Reduce”
IA161 Syntactic Formalisms for Parsing Natural Languages 46 / 50
Lecture 2
Left-corner parsing
Definition (Corner relation)
The relation ∠ between non-terminals A and B such that B ∠ A if
and only if there is a rule A → Bα, where α denotes some
sequence of grammar symbols
Definition (Left corner relation)
The transitive and reflexive closure of ∠ is denoted by ∠∗
,
which is called left-corner relation
IA161 Syntactic Formalisms for Parsing Natural Languages 47 / 50
Lecture 2
Left-corner parsing
.
Left-corner table
..
......
Non Terminal Left-corners
S S NP time an VorN files
NP NP time an VorN files
VP VP VorN files VorP like
PP PP VorP like
VorN VorN files
VorP VorP like
Grammar
S → NP VP
S → S PP
NP → time
NP → an arrow
NP → VorN
VP → VorN
VP → VorP NP
PP → VorP NP
VorN → files
VorP → like
IA161 Syntactic Formalisms for Parsing Natural Languages 48 / 50
Lecture 2
How to deal with ambiguity?
Backtracking
Try all variants subsequently.
Determinism
Just choose one variant and keep it (i.,e. greedy).
Parallelism
Try all variants in parallel.
Underspecification
Do not desambiguate, keep ambiguity.
IA161 Syntactic Formalisms for Parsing Natural Languages 49 / 50
Lecture 2
Summary
One view on parsing: parsing as a phrase-structure formal
grammar recognition task
Parsing approaches: top-down, bottom-up, left-corner
IA161 Syntactic Formalisms for Parsing Natural Languages 50 / 50