coupling semi-supervised learning of categories and relations
andrew carlson1 justin betteridge1 estevam r hruschka jr 1 2 and tom m mitchell1
1school of computer science
carnegie mellon university
pittsburgh pa 15213
2federal university of sao carlos
sao carlos sp brazil
abstract
we consider semi-supervised learning of
information extraction methods especially
for extracting instances of noun categories e g ‘athlete ’ ‘team’ and relations e g ‘playsforteam athlete team ’ semi-
supervised approaches using a small number
of labeled examples together with many un-
labeled examples are often unreliable as they
frequently produce an internally consistent but nevertheless incorrect set of extractions we propose that this problem can be over-
come by simultaneously learning classifiers
for many different categories and relations
in the presence of an ontology defining
constraints that couple the training of these
classifiers experimental results show that
simultaneously learning a coupled collection
of classifiers for 30 categories and relations
results in much more accurate extractions
than training classifiers individually 1 introduction
a great wealth of knowledge is expressed on the web
in natural language translating this into a struc-
tured knowledge base containing facts about enti-
ties e g ‘disney’ and relations between those en-
tities e g companyindustry ‘disney’ ‘entertain-
ment’ would be of great use to many applications although fully supervised methods for learning to
extract such facts from text work well the cost
of collecting many labeled examples of each type
of knowledge to be extracted is impractical re-
searchers have also explored semi-supervised learn-
ing methods that rely primarily on unlabeled data figure 1 we show that significant improvements in ac-
curacy result from coupling the training of information
extractors for many inter-related categories and relations b compared with the simpler but much more difficult
task of learning a single information extractor a but these approaches tend to suffer from the fact that
they face an under-constrained learning task result-
ing in extractions that are often inaccurate we present an approach to semi-supervised learn-
ing that yields more accurate results by coupling the
training of many information extractors the intu-
ition behind our approach summarized in figure 1 is that semi-supervised training of a single type of
extractor such as ‘coach’ is much more difficult than
simultaneously training many extractors that cover
a variety of inter-related entity and relation types in particular prior knowledge about the relation-
ships between these different entities and relations e g that ‘coach x ’ implies ‘person x ’ and ‘not
sport x ’ allows unlabeled data to become a much
more useful constraint during training although previous work has coupled the learning
of multiple categories or used static category rec-
ognizers to check arguments for learned relation ex-
tractors our work is the first we know of to couple
the simultaneous semi-supervised training of multi-
ple categories and relations our experiments show
that this coupling results in more accurate extrac-
tions based on our results reported here we hy-
pothesize that significant accuracy improvements in
information extraction will be possible by coupling
the training of hundreds or thousands of extractors 2 problem statement
it will be helpful to first explain our use of common
terms an ontology is a collection of unary and bi-
nary predicates also called categories and relations respectively 1 an instance of a category or a cate-
gory instance is a noun phrase an instance of a rela-
tion or a relation instance is a pair of noun phrases instances can be positive or negative with respect
to a specific predicate meaning that the predicate
holds or does not hold for that particular instance a promoted instance is an instance which our algo-
rithm believes to be a positive instance of some pred-
icate also associated with both categories and rela-
tions are patterns strings of tokens with placehold-
ers e g ‘game against arg1’ and ‘arg1 head coach
of arg2’ a promoted pattern is a pattern believed
to be a high-probability indicator for some predicate the challenge addressed by this work is to learn
extractors to automatically populate the categories
and relations of a specified ontology with high-
confidence instances starting from a few seed pos-
itive instances and patterns for each predicate and
a large corpus of sentences annotated with part-of-
speech pos tags we focus on extracting facts that
are stated multiple times in the corpus which we
can assess probabilistically using corpus statistics we do not resolve strings to real-world entities— the
problems of synonym resolution and disambiguation
of strings that can refer to multiple entities are left
for future work 3 related work
work on multitask learning has demonstrated that
supervised learning of multiple related” functions
together can yield higher accuracy than learning the
functions separately thrun 1996 caruana 1997 semi-supervised multitask learning has been shown
1we do not consider predicates of higher arity in this work to increase accuracy when tasks are related allow-
ing one to use a prior that encourages similar pa-
rameters liu et al 2008 our work also involves
semi-supervised training of multiple coupled func-
tions but differs in that we assume explicit prior
knowledge of the precise way in which our multi-
ple functions are related e g that the values of the
functions applied to the same input are mutually ex-
clusive or that one implies the other in this paper we focus on a ‘bootstrapping’
method for semi-supervised learning bootstrap-
ping approaches start with a small number of la-
beled ‘seed’ examples use those seed examples to
train an initial model then use this model to la-
bel some of the unlabeled data the model is
then retrained using the original seed examples plus
the self-labeled examples this process iterates gradually expanding the amount of labeled data such approaches have shown promise in applica-
tions such as web page classification blum and
mitchell 1998 named entity classification collins
and singer 1999 parsing mcclosky et al 2006 and machine translation ueffing 2006 bootstrapping approaches to information extrac-
tion can yield impressive results with little initial
human effort brin 1998 agichtein and gravano 2000 ravichandran and hovy 2002 pasca et al 2006 however after many iterations they usu-
ally suffer from semantic drift where errors in label-
ing accumulate and the learned concept ‘drifts’ from
what was intended curran et al 2007 coupling
the learning of predicates by using positive exam-
ples of one predicate as negative examples for oth-
ers has been shown to help limit this drift riloff and
jones 1999 yangarber 2003 additionally ensur-
ing that relation arguments are of certain expected
types can help mitigate the promotion of incorrect
instances pas¸ca et al 2006 rosenfeld and feld-
man 2007 our work builds on these ideas to cou-
ple the simultaneous bootstrapped training of multi-
ple categories and multiple relations our approach to information extraction is based
on using high precision contextual patterns e g ‘is
mayor of arg1’ suggests that arg1 is a city an early
pattern-based approach to information extraction ac-
quired ‘is a’ relations from text using generic con-
textual patterns hearst 1992 this approach was
later scaled up to the web by etzioni et al 2005 other research explores the task of ‘open informa-
tion extraction’ where the predicates to be learned
are not specified in advance shinyama and sekine 2006 banko et al 2007 but emerge instead from
analysis of the data in contrast our approach re-
lies strongly on knowledge in the ontology about the
predicates to be learned and relationships among
them in order to achieve high accuracy chang et al 2007 present a framework for
learning that optimizes the data likelihood plus
constraint-based penalty terms than capture prior
knowledge and demonstrate it with semi-supervised
learning of segmentation models constraints that
capture domain knowledge guide bootstrap learn-
ing of a structured model by penalizing or disallow-
ing violations of those constraints while similar in
spirit our work differs in that we consider learning
many models rather than one structured model and
that we are consider a much larger scale application
in a different domain 4 approach
4 1 coupling of predicates
as mentioned above our approach hinges on the no-
tion of coupling the learning of multiple functions
in order to constrain the semi-supervised learning
problem we face our system learns four different
types of functions for each category c 1 fc inst np c → 0 1 2 fc patt pattc c → 0 1 and for each relation r 1 fr inst np c ×np c → 0 1 2 fr patt pattr c → 0 1 where c is the input corpus np c is the set of
valid noun phrases in c pattc c is the set of valid
category patterns in c and pattr c is the set of
valid relation patterns in c valid” noun phrases category patterns and relation patterns are defined
in section 4 2 2 the learning of these functions is coupled in two
ways 1 sharing among same-arity predicates according
to logical relations
2 relation argument type-checking
these methods of coupling are made possible by
prior knowledge in the input ontology beyond the
lists of categories and relations mentioned above we provide general descriptions of these methods
of coupling in the next sections while the details are
given in section 4 2 4 1 1 sharing among same-arity predicates
each predicate p in the ontology has a list of other
same-arity predicates with which p is mutually
exclusive where mutuallyexclusive p p ′ ≡ p arg1 ⇒ ¬p ′ arg1 ∧ p ′ arg1 ⇒
¬p arg1 and similarly for relations these mu-
tually exclusive relationships are used to carry out
the following simple but crucial coupling if predi-
cate a is mutually exclusive with predicate b a’s
positive instances and patterns become negative in-
stances and negative patterns for b for example if ‘city’ having an instance ‘boston’ and a pattern
‘mayor of arg1’ is mutually exclusive with ‘scien-
tist’ then ‘boston’ and ‘mayor of arg1’ will become
a negative instance and a negative pattern respec-
tively for ‘scientist ’ such negative instances and
patterns provide negative evidence to constrain the
bootstrapping process and forestall divergence some categories are declared to be a subset of
one of the other categories being populated where
subset p p ′ ≡ p arg1 ⇒ p ′ arg1 e g ‘ath-
lete’ is a subset of ‘person’ this prior knowledge
is used to share instances and patterns of the subcat-
egory e g ‘athlete’ as positive instances and pat-
terns for the super-category e g ‘person’ 4 1 2 relation argument type-checking
the last type of prior knowledge we use to couple
the learning of functions is type checking informa-
tion which couples the learning of relations with cat-
egories for example the arguments of the ‘ceoof’
relation are declared to be of the categories ‘person’
and ‘company’ our approach does not promote a
pair of noun phrases as an instance of a relation un-
less the two noun phrases are classified as belonging
to the correct argument types additionally when a
relation instance is promoted the arguments become
promoted instances of their respective categories 4 2 algorithm description
in this section we describe our algorithm cbl coupled bootstrap learner in detail the inputs to cbl are a large corpus of pos-
tagged sentences and an initial ontology with pre-
algorithm 1 cbl algorithm
input an ontology o and text corpus c
output trusted instances patterns for each
predicate
share initial instances patterns among
predicates for i = 1 2 ∞ do
foreach predicate p ∈ o do
extract candidate instances patterns filter candidates train instance pattern classifiers assess candidates using classifiers promote highest-confidence candidates end
share promoted items among predicates end
defined categories relations mutually exclusive re-
lationships between same-arity predicates subset re-
lationships between some categories seed instances
for all predicates and seed patterns for the cate-
gories categories in the input ontology also have
a flag indicating whether instances must be proper
nouns common nouns or whether they can be ei-
ther e g instances of ‘city’ are proper nouns algorithm 1 gives a summary of the cbl algo-
rithm first seed instances and patterns are shared
among predicates using the available mutual exclu-
sion subset and type-checking relations then for an indefinite number of iterations cbl expands
the sets of promoted instances and patterns for each
predicate as detailed below cbl was designed to allow learning many pred-
icates simultaneously from a large sample of text
from the web in each iteration of the algorithm the
information needed from the text corpus is gathered
in two passes through the corpus using the mapre-
duce framework dean and ghemawat 2008 this
allows us to complete an iteration of the system in
1 hour using a corpus containing millions of web
pages see section 5 3 for details on the corpus 4 2 1 sharing
at the start of execution seed instances and pat-
terns are shared among predicates according to the
mutual exclusion subset and type-checking con-
straints newly promoted instances and patterns are
shared at the end of each iteration 4 2 2 candidate extraction
cbl finds new candidate instances by using
newly promoted patterns to extract the noun phrases
that co-occur with those patterns in the text corpus to keep the size of this set manageable cbl lim-
its the number of new candidate instances for each
predicate to 1000 by selecting the ones that occur
with the most newly promoted patterns an analo-
gous procedure is used to extract candidate patterns candidate extraction is performed for all predicates
in a single pass through the corpus using the mapre-
duce framework the candidate extraction procedure has defini-
tions for valid instances and patterns that limit ex-
traction to instances that look like noun phrases and
patterns that are likely to be informative here we
provide brief descriptions of those definitions category instances in the placeholder of a cate-
gory pattern cbl looks for a noun phrase it uses
part-of-speech tags to segment noun phrases ignor-
ing determiners proper nouns containing prepo-
sitions are segmented using a reimplementation of
the lex algorithm downey et al 2007 cate-
gory instances are only extracted if they obey the
proper common noun specification of the category category patterns if a promoted category in-
stance is found in a sentence cbl extracts the pre-
ceding words as a candidate pattern if they are verbs
followed by a sequence of adjectives prepositions or determiners e g ‘being acquired by arg1’ or
nouns and adjectives followed by a sequence of ad-
jectives prepositions or determiners e g ‘former
ceo of arg1’ cbl extracts the words following the instance as
a candidate pattern if they are verbs followed option-
ally by a noun phrase e g ‘arg1 broke the home run
record’ or verbs followed by a preposition e g ‘arg1 said that’ relation instances if a promoted relation pattern e g ‘arg1 is mayor of arg2’ is found a candi-
date relation instance is extracted if both placehold-
ers are valid noun phrases and if they obey the
proper common specifications for their categories relation patterns if both arguments from a pro-
moted relation instance are found in a sentence then
the intervening sequence of words is extracted as a
candidate relation pattern if it contains no more than
5 tokens has a content word has an uncapitalized
word and has at least one non-noun 4 2 3 candidate filtering
candidate instances and patterns are filtered to
maintain high precision and to avoid extremely spe-
cific patterns an instance is only considered for as-
sessment if it co-occurs with at least two promoted
patterns in the text corpus and if its co-occurrence
count with all promoted patterns is at least three
times greater than its co-occurrence count with neg-
ative patterns candidate patterns are filtered in the
same manner using instances all co-occurrence counts needed by the filtering
step are obtained with an additional pass through
the corpus using mapreduce this implementa-
tion is much more efficient than one that relies on
web search queries cbl typically requires co-
occurrence counts of at least 10 000 instances with
any of at least 10 000 patterns which would require
100 million hit count queries 4 2 4 candidate assessment
next for each predicate cbl trains a discretized
naı¨ve bayes classifier to classify the candidate in-
stances its features include pointwise mutual infor-
mation pmi scores turney 2001 of the candidate
instance with each of the positive and negative pat-
terns associated with the class the current sets of
promoted and negative instances are used as training
examples for the classifier attributes are discretized
based on information gain fayyad and irani 1993 patterns are assessed using an estimate of the pre-
cision of each pattern p precision p =
∑
i∈i count i p count p where i is the set of promoted instances for the
predicate currently being considered count i p is
the co-occurrence count of instance i with pattern p and count p is the hit count of the pattern p this
is a pessimistic estimate because it assumes that the
rest of the occurrences of pattern p are not with pos-
itive examples of the predicate we also penalize
extremely rare patterns by thresholding the denomi-
nator using the 25th percentile candidate pattern hit
count mcdowell and cafarella 2006 all of the co-occurrence counts needed for the as-
sessment step are collected in the same mapreduce
pass as those required for filtering candidates 4 2 5 candidate promotion
cbl then ranks the candidates according to their
assessment scores and promotes at most 100 in-
stances and 5 patterns for each predicate 5 experimental evaluation
we designed our experimental evaluation to try to
answer the following questions can cbl iterate
many times and still achieve high precision how
helpful are the types of coupling that we employ can we extend existing semantic resources 5 1 configurations of the algorithm
we ran our algorithm in three configurations full the algorithm as described in section 4 2 no sharing among same-arity predicates ns this configuration couples predicates only us-
ing type-checking constraints it uses the full
algorithm except that predicates of the same ar-
ity do not share promoted instances and patterns
with each other seed instances and patterns are
shared though so each predicate has a small fixed pool of negative evidence no category relation coupling ncr this
configuration couples predicates using mutual
exclusion and subset constraints but not type-
checking it uses the full algorithm except
that relation instance arguments are not fil-
tered or assessed using their specified categories and arguments of promoted relations are not
shared as promoted instances of categories the
only type-checking information used is the com-
mon proper noun specifications of arguments for
filtering out implausible instances 5 2 initial ontology
our ontology contained categories and relations re-
lated to two domains companies and sports ex-
tra categories were added to provide negative evi-
dence to the domain-related categories ‘hobby’ for
‘economic sector’ ‘actor ’ ‘politician ’ and ‘scien-
tist’ for ‘athlete’ and ‘coach’ and ‘board game’ for
‘sport’ table 1 lists each predicate in the leftmost
column categories were started with 10 20 seed
5 iterations 10 iterations 15 iterations
predicate full ns ncr full ns ncr full ns ncr
actor 93 100 100 93 97 100 100 97 100
athlete 100 100 100 100 93 100 100 73 100
board game 93 76 93 89 27 93 89 30 93
city 100 100 100 100 97 100 100 100 100
coach 100 63 73 97 53 43 97 47 47
company 100 100 100 97 90 97 100 90 100
country 60 40 60 30 43 27 40 23 40
economic sector 77 63 73 57 67 67 50 63 40
hobby 67 63 67 40 40 57 20 23 30
person 97 97 90 97 93 97 93 97 93
politician 93 93 97 73 53 90 90 53 87
product 97 87 90 90 87 100 97 90 77
product type 93 93 90 70 73 97 77 80 67
scientist 100 90 97 97 63 97 93 60 100
sport 100 90 100 93 67 83 97 27 90
sports team 100 97 100 97 70 100 90 50 100
category average 92 84 89 82 70 84 83 63 79
acquired company company 77 77 80 67 80 47 70 63 47
ceoof person company 97 87 100 90 87 97 90 80 83
coachesteam coach sports team 100 100 100 100 100 97 100 100 90
competesin company econ sector 97 97 80 100 93 67 97 63 60
competeswith company company 93 80 60 77 70 37 70 60 43
hasofficesin company city 97 93 40 93 90 27 93 57 30
hasoperationsin company country 100 95 50 100 97 40 90 83 13
headquarteredin company city 77 90 20 70 77 27 70 60 7
locatedin city country 90 67 57 63 50 43 73 50 30
playsfor athlete sports team 100 100 0 100 97 7 100 43 0
playssport athlete sport 100 100 27 93 80 10 100 40 30
teamplayssport sports team sport 100 100 77 100 97 80 93 83 67
produces company product 91 83 90 83 93 67 93 80 57
hastype product product type 73 63 17 33 67 33 40 57 27
relation average 92 88 57 84 84 48 84 66 42
all 92 86 74 83 76 68 84 64 62
table 1 precision for each predicate results are presented after 5 10 and 15 iterations for the full no sharing ns and no category relation coupling ncr configurations of cbl note that we expect full and ncr to
perform similarly for categories but for full to outperform ncr on relations and for full to outperform ns on both
categories and relations instances and 5 seed patterns the seed instances
were specified by a human and the seed patterns
were derived from the generic patterns of hearst
for each predicate hearst 1992 relations were
started with similar numbers of seed instances and
no seed patterns it is less obvious how to gener-
ate good seed patterns from relation names most
predicates were declared as mutually exclusive with
most others except for special cases e g ‘hobby’
and ‘sport’ ‘university’ and ‘sports team’ and ‘has
offices in’ and ‘headquartered in’ 5 3 corpus
our text corpus was from a 200-million page web
crawl we parsed the html filtered out non-
english pages using a stop word ratio threshold then
filtered out web spam and adult content using a ‘bad
word’ list the pages were then segmented into sen-
tences tokenized and tagged with parts-of-speech
using the opennlp package finally we filtered
the sentences to eliminate those that were likely to
be noisy and not useful for learning e g sentences
without a verb without any lowercase words with
too many words that were all capital letters this
yielded a corpus of roughly 514-million sentences 5 4 experimental procedure
we ran each configuration for 15 iterations to eval-
uate the precision of promoted instances we sam-
pled 30 instances from the promoted set for each
predicate in each configuration after 5 10 and 15 it-
erations pooled together the samples for each pred-
icate and then judged their correctness the judge
did not know which run an instance was sampled
from we estimated the precision of the promoted
instances from each run after 5 10 and 15 itera-
tions as the number of correct promoted instances
divided by the number sampled while samples of
30 instances do not produce tight confidence inter-
vals around individual estimates they are sufficient
for testing for the effects in which we are interested 5 5 results
table 1 shows the precision of each of the three al-
gorithm configurations for each category and rela-
tion after 5 10 and 15 iterations as is apparent
in this table fully coupled training full outper-
forms training when coupling is removed between
categories and relations ncr and also when cou-
pling is removed among predicates of the same ar-
ity ns the net effect is substantial as is appar-
ent from the bottom row of table 1 which shows
that the precision of full outperforms ns by 6 and
ncr by 18 after the first 5 iterations and by an
even larger 20 and 22 after 15 iterations this
increasing gap in precision as iterations increase re-
flects the ability of coupled learning to constrain the
system to reduce the otherwise common drift asso-
ciated with self-trained classifiers using student’s paired t-test we found that for
categories the difference in performance between
full and ns is statistically significant after 5 10 and 15 iterations p-value 0 05 2 no significant
difference was found between full and ncr for cat-
egories but this is not a surprise because ncr still
uses mutually exclusive and subset constraints the
same test finds that the differences between full and
ns are significant for relations after 15 iterations and the differences between full and ncr are sig-
nificant after 5 10 and 15 iterations for relations the worst-performing categories after 15 itera-
tions of full are ‘country ’ ‘economic sector ’ and
‘hobby ’ the full configuration of cbl promoted
1637 instances for ‘country ’ far more than the num-
ber of correct answers many of these are general
geographic regions like ‘bayfield peninsula’ and
‘baltic republics ’ in the ‘hobby’ case promoting
patterns like ‘the types of arg1’ led to the category
drifting into a general list of plural common nouns ‘economic sector’ drifted into academic fields like
‘behavioral science’ and ‘political sciences ’ we
expect that the learning of these categories would
be significantly better if there were even more cat-
egories being learned to provide additional negative
evidence during the filtering and assessment steps of
the algorithm at this stage of development obtaining high re-
call is not a priority because our intent is to create
a continuously running and continuously improving
system it is our hope that high recall will come with
time however to very roughly convey the com-
pleteness of the current results we show in table 2
the average number of instances promoted for cate-
2our selection of the paired t-test was motivated by the work
of smucker et al 2007 but the wilcoxon signed rank test
gives the same results categories relations
configuration instances prec instances prec full 970 83 191 84
ns 1337 63 307 66
ncr 916 79 458 42
table 2 average numbers of promoted category and re-
lation instances and estimates of their precision for each
configuration of cbl after 15 iterations figure 2 extracted facts for two companies discovered
by cbl full these two companies were extracted by
the learned ‘company’ extractor and the relations shown
were extracted by learned relation extractors gories and relations for each of the three configura-
tions of cbl after 15 iterations for categories not
sharing examples results in fewer negative examples
during the filtering and assessment steps this yields
more promoted instances on average for relations not using type checking yields higher relative recall but at a much lower level of precision figure 2 gives one view of the type of information
extracted by the collection of learned category and
relation classifiers note the initial seed examples
provided to cbl did not include information about
either company or any of these relation instances 3
5 6 comparison to an existing database
to estimate the capacity of our algorithm to con-
tribute additional facts to publicly available seman-
tic resources we compared the complete lists of in-
stances promoted during the full 15 iteration run
for certain categories to corresponding lists in the
freebase database metaweb technologies 2009 excluding the categories that did not have a di-
rectly corresponding freebase list we computed for
each category precision × cblinstances − matches where precision is the estimated pre-
cision from our random sample of 30 instances cblinstances is the total number of instances
promoted for that category and matches is the
3see http rtw ml cmu edu sslnlp09 for re-
sults from a full run of the system est cbl freebase est new
category prec instances matches instances
actor 100 522 465 57
athlete 100 117 54 63
board game 89 18 6 10
city 100 1799 1665 134
company 100 1937 995 942
econ sector 50 1541 137 634
politician 90 962 74 792
product 97 1259 0 1221
sports team 90 414 139 234
sport 97 613 134 461
table 3 estimated numbers of new instances” correct
instances promoted by cbl in the full 15 iteration run
which do not have a match in freebase and the values
used in calculating them number of promoted instances that had an exact
match in freebase while exact matches may under-
estimate the number of matches it should be noted
that rather than make definitive claims our intent
here is simply to give rough estimates which are
shown in table 3 these approximate numbers in-
dicate a potential to use cbl to extend existing se-
mantic resources like freebase 6 conclusion
we have presented a method of coupling the semi-
supervised learning of categories and relations and
demonstrated empirically that the coupling forestalls
the problem of semantic drift associated with boot-
strap learning methods we suspect that learning
additional predicates simultaneously will yield even
more accurate learning an approximate compari-
son with an existing repository of semantic knowl-
edge freebase suggests that our methods can con-
tribute new facts to existing resources acknowledgments
this work is supported in part by darpa google a yahoo fellowship to andrew carlson and the
brazilian research agency cnpq we also gratefully
acknowledge jamie callan for making available his
collection of web pages yahoo for use of their m45
computing cluster and the anonymous reviewers for
their comments references
eugene agichtein and luis gravano 2000 snowball extracting relations from large plain-text collections in jcdl michele banko michael j cafarella stephen soderland matt broadhead and oren etzioni 2007 open infor-
mation extraction from the web in ijcai avrim blum and tom mitchell 1998 combining la-
beled and unlabeled data with co-training in colt sergey brin 1998 extracting patterns and relations
from the world wide web in webdb workshop at
6th international conference on extending database
technology rich caruana 1997 multitask learning machine
learning 28 41 75 ming-wei chang lev-arie ratinov and dan roth 2007 guiding semi-supervision with constraint-
driven learning in acl michael collins and yoram singer 1999 unsupervised
models for named entity classification in emnlp james r curran tara murphy and bernhard scholz 2007 minimising semantic drift with mutual exclu-
sion bootstrapping in pacling jeffrey dean and sanjay ghemawat 2008 mapreduce simplified data processing on large clusters commun acm 51 1 107 113 doug downey matthew broadhead and oren etzioni 2007 locating complex named entities in web text in ijcai oren etzioni michael cafarella doug downey ana-
maria popescu tal shaked stephen soderland daniel s weld and alexander yates 2005 unsu-
pervised named-entity extraction from the web an ex-
perimental study artif intell 165 1 91 134 usama m fayyad and keki b irani 1993 multi-
interval discretization of continuous-valued attributes
for classification learning in uai marti a hearst 1992 automatic acquisition of hy-
ponyms from large text corpora in coling qiuhua liu xuejun liao and lawrence carin 2008 semi-supervised multitask learning in nips david mcclosky eugene charniak and mark johnson 2006 effective self-training for parsing in naacl luke k mcdowell and michael cafarella 2006 ontology-driven information extraction with on-
tosyphon in iswc metaweb technologies 2009 freebase data dumps http download freebase com datadumps marius pas¸ca dekang lin jeffrey bigham andrei lif-
chits and alpa jain 2006 names and similarities on
the web fact extraction in the fast lane in acl marius pasca dekang lin jeffrey bigham andrei lif-
chits and alpa jain 2006 organizing and search-
ing the world wide web of facts step one the one-
million fact extraction challenge in aaai deepak ravichandran and eduard hovy 2002 learning
surface text patterns for a question answering system in acl ellen riloff and rosie jones 1999 learning dictionar-
ies for information extraction by multi-level bootstrap-
ping in aaai benjamin rosenfeld and ronen feldman 2007 us-
ing corpus statistics on entities to improve semi-
supervised relation extraction from the web in acl yusuke shinyama and satoshi sekine 2006 preemp-
tive information extraction using unrestricted relation
discovery in hlt-naacl mark d smucker james allan and ben carterette 2007 a comparison of statistical significance tests for
information retrieval evaluation in cikm sebastian thrun 1996 is learning the n-th thing any
easier than learning the first in nips peter d turney 2001 mining the web for synonyms pmi-ir versus lsa on toefl in emcl nicola ueffing 2006 self-training for machine trans-
lation in nips workshop on machine learning for
multilingual information access roman yangarber 2003 counter-training in discovery
of semantic patterns in acl