entity disambiguation for knowledge base population
†mark dredze and †paul mcnamee and †delip rao and †adam gerber and tim finin
†human language technology center of excellence center for language and speech processing
johns hopkins university university of maryland baltimore county
mdredze mcnamee delip adam gerber@jhu edu finin@umbc edu
abstract
the integration of facts derived from information extraction
systems into existing knowledge bases requires a system to
disambiguate entity mentions in the text this is challeng-
ing due to issues such as non-uniform variations in entity
names mention ambiguity and entities absent from a knowl-
edge base we present a state of the art system for entity dis-
ambiguation that not only addresses these challenges but also
scales to knowledge bases with several million entries using
very little resources further our approach achieves perfor-
mance of up to 95 on entities mentioned from newswire
and 80 on a public test set that was designed to include
challenging queries 1 introduction
the ability to identify entities like people orga-
nizations and geographic locations tjong kim
sang and de meulder 2003 extract their at-
tributes pasca 2008 and identify entity rela-
tions banko and etzioni 2008 is useful for sev-
eral applications in natural language processing
and knowledge acquisition tasks like populating
structured knowledge bases kb however inserting extracted knowledge into a
kb is fraught with challenges arising from nat-
ural language ambiguity textual inconsistencies and lack of world knowledge to the discern-
ing human eye the bush” in mr bush left
for the zurich environment summit in air force
one ” is clearly the us president further con-
text may reveal it to be the 43rd president george
w bush and not the 41st president george h w bush the ability to disambiguate a polyse-
mous entity mention or infer that two orthograph-
ically different mentions are the same entity is
crucial in updating an entity’s kb record this
task has been variously called entity disambigua-
tion record linkage or entity linking when per-
formed without a kb entity disambiguation is
called coreference resolution entity mentions ei-
ther within the same document or across multi-
ple documents are clustered together where each
cluster corresponds to a single real world entity the emergence of large scale publicly avail-
able kbs like wikipedia and dbpedia has spurred
an interest in linking textual entity references to
their entries in these public kbs bunescu and
pasca 2006 and cucerzan 2007 presented im-
portant pioneering work in this area but suffer
from several limitations including wikipedia spe-
cific dependencies scale and the assumption of
a kb entry for each entity in this work we in-
troduce an entity disambiguation system for link-
ing entities to corresponding wikipedia pages de-
signed for open domains where a large percent-
age of entities will not be linkable further our
method and some of our features readily general-
ize to other curated kb we adopt a supervised
approach where each of the possible entities con-
tained within wikipedia are scored for a match to
the query entity we also describe techniques to
deal with large knowledge bases like wikipedia which contain millions of entries furthermore our system learns when to withhold a link when
an entity has no matching kb entry a task that
has largely been neglected in prior research in
cross-document entity coreference our system
produces high quality predictions compared with
recent work on this task 2 related work
the information extraction oeuvre has a gamut of
relation extraction methods for entities like per-
sons organizations and locations which can be
classified as open- or closed-domain depending
on the restrictions on extractable relations banko
and etzioni 2008 closed domain systems ex-
tract a fixed set of relations while in open-domain
systems the number and type of relations are un-
bounded extracted relations still require process-
ing before they can populate a kb with facts namely entity linking and disambiguation motivated by ambiguity in personal name
search mann and yarowsky 2003 disambiguate
person names using biographic facts like birth
year occupation and affiliation when present
in text biographic facts extracted using regular
expressions help disambiguation more recently the web people search task artiles et al 2008 clustered web pages for entity disambiguation the related task of cross document corefer-
ence resolution has been addressed by several
researchers starting from bagga and baldwin 1998 poesio et al 2008 built a cross document
coreference system using features from encyclo-
pedic sources like wikipedia however success-
ful coreference resolution is insufficient for cor-
rect entity linking as the coreference chain must
still be correctly mapped to the proper kb entry previous work by bunescu and pasca 2006 and cucerzan 2007 aims to link entity men-
tions to their corresponding topic pages in
wikipedia but the authors differ in their ap-
proaches cucerzan uses heuristic rules and
wikipedia disambiguation markup to derive map-
pings from surface forms of entities to their
wikipedia entries for each entity in wikipedia a context vector is derived as a prototype for the
entity and these vectors are compared via dot-
product with the context vectors of unknown en-
tity mentions his work assumes that all entities
have a corresponding wikipedia entry but this as-
sumption fails for a significant number of entities
in news articles and even more for other genres like blogs bunescu and pasca on the other hand
suggest a simple method to handle entities not in
wikipedia by learning a threshold to decide if the
entity is not in wikipedia both works mentioned
rely on wikipedia-specific annotations such as
category hierarchies and disambiguation links we just recently became aware of a system
fielded by li et al at the tac-kbp 2009 eval-
uation 2009 their approach bears a number
of similarities to ours both systems create candi-
date sets and then rank possibilities using differing
learning methods but the principal difference is in
our approach to nil prediction where we simply
consider absence i e the nil candidate as an-
other entry to rank and select the top-ranked op-
tion they use a separate binary classifier to decide
whether their top prediction is correct or whether
nil should be output we believe relying on fea-
tures that are designed to inform whether absence
is correct is the better alternative 3 entity linking
we define entity linking as matching a textual en-
tity mention possibly identified by a named en-
tity recognizer to a kb entry such as a wikipedia
page that is a canonical entry for that entity an
entity linking query is a request to link a textual
entity mention in a given document to an entry in
a kb the system can either return a matching en-
try or nil to indicate there is no matching entry in this work we focus on linking organizations geo-political entities and persons to a wikipedia
derived kb 3 1 key issues
there are 3 challenges to entity linking name variations an entity often has multiple
mention forms including abbreviations boston
symphony orchestra vs bso shortened forms osama bin laden vs bin laden alternate
spellings osama vs ussamah vs oussama and aliases osama bin laden vs sheikh al-
mujahid entity linking must find an entry de-
spite changes in the mention string entity ambiguity a single mention like
springfield can match multiple kb entries as
many entity names like people and organizations tend to be polysemous absence processing large text collections vir-
tually guarantees that many entities will not ap-
pear in the kb nil even for large kbs the combination of these challenges makes
entity linking especially challenging consider
an example of william clinton ” most read-
ers will immediately think of the 42nd us pres-
ident however the only two william clintons in
wikipedia are william de clinton” the 1st earl
of huntingdon and william henry clinton” the
british general the page for the 42nd us pres-
ident is actually bill clinton” an entity link-
ing system must decide if either of the william
clintons are correct even though neither are ex-
act matches if the system determines neither
matches should it return nil or the variant bill
clinton” if variants are acceptable then perhaps clinton iowa” or dewitt clinton” should be
acceptable answers 3 2 contributions
we address these entity linking challenges robust candidate selection our system is
flexible enough to find name variants but suffi-
ciently restrictive to produce a manageable can-
didate list despite a large-scale kb features for entity disambiguation we de-
veloped a rich and extensible set of features based
on the entity mention the source document and
the kb entry we use a machine learning ranker
to score each candidate learning nils we modify the ranker to learn
nil predictions which obviates hand tuning and
importantly admits use of additional features that
are indicative of nil our contributions differ from previous efforts bunescu and pasca 2006 cucerzan 2007 in
several important ways first previous efforts de-
pend on wikipedia markup for significant perfor-
mance gains we make no such assumptions al-
though we show that optional wikipedia features
lead to a slight improvement second cucerzan
does not handle nils while bunescu and pasca
address them by learning a threshold our ap-
proach learns to predict nil in a more general
and direct way third we develop a rich fea-
ture set for entity linking that can work with any
kb finally we apply a novel finite state machine
method for learning name variations 1
the remaining sections describe the candidate
selection system features and ranking and our
novel approach learning nils followed by an
empirical evaluation 4 candidate selection for name variants
the first system component addresses the chal-
lenge of name variants as the kb contains a large
number of entries 818 000 entities of which 35 are per org or gpe we require an efficient se-
lection of the relevant candidates for a query previous approaches used wikipedia markup
for filtering only using the top-k page categories
1http www clsp jhu edu markus fstrain bunescu and pasca 2006 which is limited to
wikipedia and does not work for general kbs we consider a kb independent approach to selec-
tion that also allows for tuning candidate set size this involves a linear pass over kb entry names wikipedia page titles a naive implementation
took two minutes per query the following sec-
tion reduces this to under two seconds per query for a given query the system selects kb entries
using the following approach titles that are exact matches for the mention titles that are wholly contained in or contain
the mention e g nationwide and nationwide in-
surance the first letters of the entity mention match the
kb entry title e g oa and olympic airlines the title matches a known alias for the entity aliases described in section 5 2 the title has a strong string similarity score
with the entity mention we include several mea-
sures of string similarity including character
dice score 0 9 skip bigram dice score 0 6 and hamming distance = 2 we did not optimize the thresholds for string
similarity but these could obviously be tuned to
minimize the candidate sets and maximize recall all of the above features are general for any
kb however since our evaluation used a kb
derived from wikipedia we included a few
wikipedia specific features we added an entry if
its wikipedia page appeared in the top 20 google
results for a query on the training dataset section 7 the selection
system attained a recall of 98 8 and produced
candidate lists that were three to four orders of
magnitude smaller than the kb some recall er-
rors were due to inexact acronyms abc arab
banking ‘corporation’ is missing asg abu
sayyaf ‘group’ is missing and pcf french
communist party french reverses the order of the
pre-nominal adjectives we also missed interna-
tional police interpol and becks david beck-
ham mr beckham and his wife are collectively
referred to as ‘posh and becks’ 4 1 scaling candidate selection
our previously described candidate selection re-
lied on a linear pass over the kb but we seek
more efficient methods we observed that the
above non-string similarity filters can be pre-
computed and stored in an index and that the skip
bigram dice score can be computed by indexing
the skip bigrams for each kb title we omitted
the other string similarity scores and collectively
these changes enable us to avoid a linear pass over
the kb finally we obtained speedups by serving
the kb concurrently2 recall was nearly identical
to the full system described above only two more
queries failed additionally more than 95 of
the processing time was consumed by dice score
computation which was only required to cor-
rectly retrieve less than 4 of the training queries omitting the dice computation yielded results in
a few milliseconds a related approach is that of
canopies for scaling clustering for large amounts
of bibliographic citations mccallum et al 2000 in contrast our setting focuses on alignment vs clustering mentions for which overlapping parti-
tioning approaches like canopies are applicable 5 entity linking as ranking
we select a single correct candidate for a query
using a supervised machine learning ranker we
represent each query by a d dimensional vector
x where x ∈ rd and we aim to select a sin-
gle kb entry y where y ∈ y a set of possible
kb entries for this query produced by the selec-
tion system above which ensures that y is small the ith query is given by the pair xi yi where
we assume at most one correct kb entry to evaluate each candidate kb entry in y we
create feature functions of the form f x y de-
pendent on both the example x document and en-
tity mention and the kb entry y the features
address name variants and entity disambiguation we take a maximum margin approach to learn-
ing the correct kb entry y should receive a
higher score than all other possible kb entries
ŷ ∈ y ŷ 6= y plus some margin γ this learning
2our python implementation with indexing features and
four threads achieved up to 80× speedup compared to naive
implementation constraint is equivalent to the ranking svm algo-
rithm of joachims 2002 where we define an or-
dered pair constraint for each of the incorrect kb
entries ŷ and the correct entry y training sets pa-
rameters such that score y ≥ score ŷ γ we
used the library svmrank to solve this optimiza-
tion problem 3 we used a linear kernel set the
slack parameter c as 0 01 times the number of
training examples and take the loss function as
the total number of swapped pairs summed over
all training examples while previous work used
a custom kernel we found a linear kernel just as
effective with our features this has the advan-
tage of efficiency in both training and prediction 4 important considerations in a system meant to
scale to millions of kb entries 5 1 features for entity disambiguation
200 atomic features represent x based on each
candidate query kb pair since we used a lin-
ear kernel we explicitly combined certain fea-
tures e g acroynym-match and known-alias to
model correlations this included combining each
feature with the predicted type of the entity al-
lowing the algorithm to learn prediction functions
specific to each entity type with feature combina-
tions the total number of features grew to 26 569 the next sections provide an overview for a de-
tailed list see mcnamee et al 2009 5 2 features for name variants
variation in entity name has long been recog-
nized as a bane for information extraction sys-
tems poor handling of entity name variants re-
sults in low recall we describe several features
ranging from simple string match to finite state
transducer matching string equality if the query name and kb en-
try name are identical this is a strong indication of
a match and in our kb entry names are distinct however similar or identical entry names that
refer to distinct entities are often qualified with
parenthetical expressions or short clauses as
an example london kentucky” is distinguished
3
www cs cornell edu people tj svm_light svm_rank html
4bunescu and pasca 2006 report learning tens of thou-
sands of support vectors with their taxonomy” kernel while
a linear kernel represents all support vectors with a single
weight vector enabling faster training and prediction from london ontario” london arkansas” london novel ” and london” therefore other string equality features were used such as
whether names are equivalent after some transfor-
mation for example baltimore” and baltimore
city” are exact matches after removing a common
gpe word like city university of vermont” and university of vt” match if vt is expanded approximate string matching many entity
mentions will not match full names exactly we
added features for character dice skip bigram
dice and left and right hamming distance scores features were set based on quantized scores these were useful for detecting minor spelling
variations or mistakes features were also added if
the query was wholly contained in the entry name or vice-versa which was useful for handling ellip-
sis e g united states department of agricul-
ture” vs department of agriculture” we also
included the ratio of the recursive longest com-
mon subsequence christen 2006 to the shorter
of the mention or entry name which is effective at
handling some deletions or word reorderings e g li gong” and gong li” finally we checked
whether all of the letters of the query are found in
the same order in the entry name e g univ wis-
consin” would match university of wisconsin” acronyms features for acronyms using dic-
tionaries and partial character matches enable
matches between mit” and madras institute of
technology” or ministry of industry and trade ”
aliases many aliases or nicknames are non-
trivial to guess for example java is the
stock symbol for sun microsystems and gin-
ger spice” is a stage name of geri halliwell a
reasonable way to do this is to employ a dictio-
nary and alias lists that are commonly available
for many domains5 fst name matching another measure of sur-
face similarity between a query and a candidate
was computed by training finite-state transducers
similar to those described in dreyer et al 2008 these transducers assign a score to any string pair
by summing over all alignments and scoring all
5we used multiple lists including class-specific lists i e for per org and gpe lists extracted from freebase bol-
lacker et al 2008 and wikipedia redirects per org and
gpe are the commonly used terms for entity types for peo-
ple organizations and geo-political regions respectively contained character n-grams we used n-grams of
length 3 and less the scores are combined using a
global log-linear model since different spellings
of a name may vary considerably in length e g j miller vs jennifer miller we eliminated the
limit on consecutive insertions used in previous
applications 6
5 3 wikipedia features
most of our features do not depend on wikipedia
markup but it is reasonable to include features
from kb properties our feature ablation study
shows that dropping these features causes a small
but statistically significant performance drop wikigraph statistics we added features de-
rived from the wikipedia graph structure for an
entry like indegree of a node outdegree of a node and wikipedia page length in bytes these statis-
tics favor common entity mentions over rare ones wikitology kb entries can be indexed with hu-
man or machine generated metadata consisting of
keywords or categories in a domain-appropriate
taxonomy using a system called wikitology syed et al 2008 investigated use of ontology
terms obtained from the explicit category system
in wikipedia as well as relationships induced from
the hyperlink graph between related wikipedia
pages following this approach we computed top-
ranked categories for the query documents and
used this information as features if none of the
candidate kb entries had corresponding highly-
ranked wikitology pages we used this as a nil
feature section 6 1 5 4 popularity
although it may be an unsafe bias to give prefer-
ence to common entities we find it helpful to pro-
vide estimates of entity popularity to our ranker
as others have done fader et al 2009 apart
from the graph-theoretic features derived from the
wikipedia graph we used google’s pagerank to
by adding features indicating the rank of the kb
entry’s corresponding wikipedia page in a google
query for the target entity mention 6without such a limit the objective function may diverge
for certain parameters of the model we detect such cases and
learn to avoid them during training 5 5 document features
the mention document and text associated with a
kb entry contain context for resolving ambiguity entity mentions some features were based on
presence of names in the text whether the query
appeared in the kb text and the entry name in the
document additionally we used a named-entity
tagger and relation finder serif boschee et al 2005 identified name and nominal mentions that
were deemed co-referent with the entity mention
in the document and tested whether these nouns
were present in the kb text without the ne anal-
ysis accuracy on non-nil entities dropped 4 5 kb facts kb nodes contain infobox attributes or facts we tested whether the fact text was
present in the query document both locally to a
mention or anywhere in the text although these
facts were derived from wikipedia infoboxes they could be obtained from other sources as well document similarity we measured similarity
between the query document and the kb text in
two ways cosine similarity with tf idf weight-
ing salton and mcgill 1983 and using the dice
coefficient over bags of words idf values were
approximated using counts from the google 5-
gram dataset as by klein and nelson 2008 entity types since the kb contained types
for entries we used these as features as well as
the predicted ne type for the entity mention in
the document text additionally since only a
small number of kb entries had per org or
gpe types we also inferred types from infobox
class information to attain 87 coverage in the
kb this was helpful for discouraging selection
of eponymous entries named after famous enti-
ties e g the former u s president vs john f kennedy international airport” 5 6 feature combinations
to take into account feature dependencies we cre-
ated combination features by taking the cross-
product of a small set of diverse features the
attributes used as combination features included
entity type a popularity based on google’s rank-
ings document comparison using tf idf cov-
erage of co-referential nouns in the kb node
text and name similarity the combinations were
cascaded to allow arbitrary feature conjunctions thus it is possible to end up with a feature kbtype-
is-org and high-tfidf-score and low-name-
similarity the combined features increased the
number of features from roughly 200 to 26 000 6 predicting nil mentions
so far we have assumed that each example has a
correct kb entry however when run over a large
corpus such as news articles we expect a signifi-
cant number of entities will not appear in the kb hence it will be useful to predict nils we learn when to predict nil using the svm
ranker by augmenting y to include nil which
then has a single feature unique to nil answers it can be shown that modulo slack variables this
is equivalent to learning a single threshold τ for
nil predictions as in bunescu and pasca 2006 incorporating nil into the ranker has several
advantages first the ranker can set the thresh-
old optimally without hand tuning second since
the svm scores are relative within a single exam-
ple and cannot be compared across examples set-
ting a single threshold is difficult third a thresh-
old sets a uniform standard across all examples whereas in practice we may have reasons to favor
a nil prediction in a given example we design
features for nil prediction that cannot be cap-
tured in a single parameter 6 1 nil features
integrating nil prediction into learning means
we can define arbitrary features indicative of nil
predictions in the feature vector corresponding to
nil for example if many candidates have good
name matches it is likely that one of them is cor-
rect conversely if no candidate has high entry-
text article similarity or overlap between facts
and the article text it is likely that the entity is
absent from the kb we included several features such as a the max mean and difference between
max and mean for 7 atomic features for all kb
candidates considered b whether any of the can-
didate entries have matching names exact and
fuzzy string matching c whether any kb en-
try was a top wikitology match and d if the top
google match was not a candidate micro-averaged macro-averaged
best median all features best features best median all features best features
all 0 8217 0 7108 0 7984 0 7941 0 7704 0 6861 0 7695 0 7704
non-nil 0 7725 0 6352 0 7063 0 6639 0 6696 0 5335 0 6097 0 5593
nil 0 8919 0 7891 0 8677 0 8919 0 8789 0 7446 0 8464 0 8721
table 1 micro and macro-averaged accuracy for tac-kbp data compared to best and median reported performance results are shown for all features as well as removing a small number of features using feature selection on development data 7 evaluation
we evaluated our system on two datasets the
text analysis conference tac track on knowl-
edge base population tac-kbp mcnamee and
dang 2009 and the newswire data used by
cucerzan 2007 microsoft news data since our approach relies on supervised learn-
ing we begin by constructing our own training
corpus 7 we highlighted 1496 named entity men-
tions in news documents from the tac-kbp doc-
ument collection and linked these to entries in
a kb derived from wikipedia infoboxes 8 we
added to this collection 119 sample queries from
the tac-kbp data the total of 1615 training ex-
amples included 539 33 4 per 618 38 3 org and 458 28 4 gpe entity mentions of
the training examples 80 5 were found in the
kb matching 300 unique entities this set has a
higher number of nil entities than did bunescu
and pasca 2006 10 but lower than the tac-
kbp test set 43 all system development was done using a train 908 examples and development 707 examples split the tac-kbp and microsoft news data
sets were held out for final tests a model trained
on all 1615 examples was used for experiments 7 1 tac-kbp 2009 experiments
the kb is derived from english wikipedia pages
that contained an infobox entries contain basic
descriptions article text and attributes the tac-
kbp query set contains 3904 entity mentions for
560 distinct entities entity type was only provided
for evaluation the majority of queries were for
organizations 69 most queries were missing
from the kb 57 77 of the distinct gpes
in the queries were present in the kb but for
7data available from www dredze com
8http en wikipedia org wiki help infobox
pers and orgs these percentages were signifi-
cantly lower 19 and 30 respectively table 1 shows results on tac-kbp data us-
ing all of our features as well a subset of features
based on feature selection experiments on devel-
opment data we include scores for both micro-
averaged accuracy averaged over all queries and macro-averaged accuracy averaged over
each unique entity as well as the best and me-
dian reported results for these data mcnamee
and dang 2009 we obtained the best reported
results for macro-averaged accuracy as well as
the best results for nil detection with micro-
averaged accuracy which shows the advantage of
our approach to learning nil see mcnamee et
al 2009 for additional experiments the candidate selection phase obtained a re-
call of 98 6 similar to that of development data missed candidates included iron lady which
refers metaphorically to yulia tymoshenko pcc the spanish-origin acronym for the cuban com-
munist party and queen city a former nickname
for the city of seattle washington the system re-
turned a mean of 76 candidates per query but the
median was 15 and the maximum 2772 texas in
about 10 of cases there were four or fewer can-
didates and in 10 of cases there were more than
100 candidate kb nodes we observed that orgs
were more difficult due to the greater variation
and complexity in their naming and that they can
be named after persons or locations 7 2 feature effectiveness
we performed two feature analyses on the tac-
kbp data an additive study starting from a
small baseline feature set used in candidate selec-
tion we add feature groups and measure perfor-
mance changes omitting feature combinations and an ablative study starting from all features remove a feature group and measure performance class all non-nil nil
baseline 0 7264 0 4621 0 9251
acronyms 0 7316 0 4860 0 9161
ne analysis 0 7661 0 7181 0 8022
google 0 7597 0 7421 0 7730
doc kb text similarity 0 7313 0 6699 0 7775
wikitology 0 7318 0 4549 0 9399
all 0 7984 0 7063 0 8677
table 2 additive analysis micro-averaged accuracy table 2 shows the most significant features in
the feature addition experiments the baseline
includes only features based on string similarity
or aliases and is not effective at finding correct
entries and strongly favors nil predictions in-
clusion of features based on analysis of named-
entities popularity measures e g google rank-
ings and text comparisons provided the largest
gains the overall changes are fairly small roughly ±1 however changes in non-nil pre-
cision are larger the ablation study showed considerable redun-
dancy across feature groupings in several cases performance could have been slightly improved
by removing features removing all feature com-
binations would have improved overall perfor-
mance to 81 05 by gaining on non-nil for a
small decline on nil detection 7 3 experiments on microsoft news data
we downloaded the evaluation data used in
cucerzan 2007 9 20 news stories from msnbc
with 642 entity mentions manually linked to
wikipedia and another 113 mentions not having
any corresponding link to wikipedia 10 a sig-
nificant percentage of queries were not of type
per org or gpe e g christmas” serif
assigned entity types and we removed 297 queries
not recognized as entities counts in table 3 we learned a new model on the training data
above using a reduced feature set to increase
speed 11 using our fast candidate selection sys-
tem we resolved each query in 1 98 seconds me-
dian query processing time was proportional to
9
http research microsoft com en-us um people silviu webassistant testdata 10one of the msnbc news articles is no longer available
so we used 759 total entities 11we removed google fst and conjunction features
which reduced system accuracy but increased performance num queries accuracy
total nil all non-nil nil
nil 452 187 0 4137 0 0 1 0
gpe 132 20 0 9696 1 00 0 8000
org 115 45 0 8348 0 7286 1 00
per 205 122 0 9951 0 9880 1 00
all 452 187 0 9469 0 9245 0 9786
cucerzan 2007 0 914 table 3 micro-average results for microsoft data the number of candidates considered we selected
a median of 13 candidates for per 12 for org
and 102 for gpe accuracy results are in table
3 the high results reported for this dataset over
tac-kbp is primarily because we perform very
well in predicting popular and rare entries both
of which are common in newswire text one issue with our kb was that it was derived
from infoboxes in wikipedia’s oct 2008 version
which has both new entities 12 and is missing en-
tities 13 therefore we manually confirmed nil
answers and new answers for queries marked as
nil in the data while an exact comparison is not
possible as described above our results 94 7 appear to be at least on par with cucerzan’s sys-
tem 91 4 overall accuracy with the strong re-
sults on tac-kbp we believe that this is strong
confirmation of the effectiveness of our approach 8 conclusion
we presented a state of the art system to disam-
biguate entity mentions in text and link them to
a knowledge base unlike previous approaches our approach readily ports to kbs other than
wikipedia we described several important chal-
lenges in the entity linking task including han-
dling variations in entity names ambiguity in en-
tity mentions and missing entities in the kb and
we showed how to each of these can be addressed we described a comprehensive feature set to ac-
complish this task in a supervised setting impor-
tantly our method discriminately learns when not
to link with high accuracy to spur further re-
search in these areas we are releasing our entity
linking system 122008 vs 2006 version used in cucerzan 2007 we
could not get the 2006 version from the author or the internet 13since our kb was derived from infoboxes entities not
having an infobox were left out references
javier artiles satoshi sekine and julio gonzalo 2008 web people search results of the first evalu-
ation and the plan for the second in www amit bagga and breck baldwin 1998 entity-
based cross-document coreferencing using the vec-
tor space model in conference on computational
linguistics coling michele banko and oren etzioni 2008 the tradeoffs
between open and traditional relation extraction in
association for computational linguistics k bollacker c evans p paritosh t sturge and
j taylor 2008 freebase a collaboratively cre-
ated graph database for structuring human knowl-
edge in sigmod management of data e boschee r weischedel and a zamanian 2005 automatic information extraction in conference
on intelligence analysis razvan c bunescu and marius pasca 2006 using
encyclopedic knowledge for named entity disam-
biguation in european chapter of the assocation
for computational linguistics eacl peter christen 2006 a comparison of personal name
matching techniques and practical issues techni-
cal report tr-cs-06-02 australian national uni-
versity silviu cucerzan 2007 large-scale named entity
disambiguation based on wikipedia data in em-
pirical methods in natural language processing emnlp markus dreyer jason smith and jason eisner 2008 latent-variable modeling of string transductions
with finite-state methods in empirical methods in
natural language processing emnlp anthony fader stephen soderland and oren etzioni 2009 scaling wikipedia-based named entity dis-
ambiguation to arbitrary web text in wikiai09
workshop at ijcai 2009 thorsten joachims 2002 optimizing search engines
using clickthrough data in knowledge discovery
and data mining kdd martin klein and michael l nelson 2008 a com-
parison of techniques for estimating idf values to
generate lexical signatures for the web in work-
shop on web information and data management widm fangtao li zhicheng zhang fan bu yang tang xiaoyan zhu and minlie huang 2009 thu
quanta at tac 2009 kbp and rte track in text
analysis conference tac gideon s mann and david yarowsky 2003 unsuper-
vised personal name disambiguation in conference
on natural language learning conll andrew mccallum kamal nigam and lyle ungar 2000 efficient clustering of high-dimensional data
sets with application to reference matching in
knowledge discovery and data mining kdd paul mcnamee and hoa trang dang 2009 overview
of the tac 2009 knowledge base population track in text analysis conference tac paul mcnamee mark dredze adam gerber nikesh
garera tim finin james mayfield christine pi-
atko delip rao david yarowsky and markus
dreyer 2009 hltcoe approaches to knowledge
base population at tac 2009 in text analysis con-
ference tac marius pasca 2008 turning web text and search
queries into factual knowledge hierarchical class
attribute extraction in national conference on ar-
tificial intelligence aaai massimo poesio david day ron artstein jason dun-
can vladimir eidelman claudio giuliano rob
hall janet hitzeman alan jern mijail kabadjov stanley yong wai keong gideon mann alessan-
dro moschitti simone ponzetto jason smith josef
steinberger michael strube jian su yannick ver-
sley xiaofeng yang and michael wick 2008 ex-
ploiting lexical and encyclopedic resources for en-
tity disambiguation final report technical report jhu clsp 2007 summer workshop gerard salton and michael mcgill 1983 introduc-
tion to modern information retrieval mcgraw-
hill book company erik tjong kim sang and fien de meulder 2003 in-
troduction to the conll-2003 shared task language-
independent named entity recognition in confer-
ence on natural language learning conll zareen syed tim finin and anupam joshi 2008 wikipedia as an ontology for describing documents in proceedings of the second international confer-
ence on weblogs and social media aaai press