Abstract:
In this paper we concentrate on the resolution of the lexical ambiguity that arises when a given word has several different meanings. This specific task is commonly referred to as word sense disambiguation (WSD). The task of WSD consists of assigning the correct sense to words using an electronic dictionary as the source of word definitions. We present two WSD methods based on two main methodological approaches in this research area: a knowledge-based method and a corpus-based method. Our hypothesis is that word-sense disambiguation requires several knowledge sources in order to solve the semantic ambiguity of the words. These sources can be of different kinds--- for example, syntagmatic, paradigmatic or statistical information. Our approach combines various sources of knowledge, through combinations of the two WSD methods mentioned above. Mainly, the paper concentrates on how to combine these methods and sources of information in order to achieve good results in the disambiguation. Finally, this paper presents a comprehensive study and experimental work on evaluation of the methods and their combinations.

---

Extracted Text

 


 
Journal of Artificial Intelligence Research 23 (2005) 299-330
Submitted 07/04; published 03/05
Combining Knowledge- and Corpus-based
Word-Sense-Disambiguation Methods
Andres Montoyo
montoyo@dlsi.ua.es
Dept. of Software and Computing SystemsUniversity of Alicante, Spain
Armando Su´
arez
armando@dlsi.ua.es
Dept. of Software and Computing SystemsUniversity of Alicante, Spain
German Rigau
rigau@si.ehu.es
IXA Research GroupComputer Science DepartmentBasque Country University, Donostia
Manuel Palomar
mpalomar@dlsi.ua.es
Dept. of Software and Computing SystemsUniversity of Alicante, Spain
Abstract
In this paper we concentrate on the resolution of the lexical ambiguity that arises when
a given word has several different meanings. This specific task is commonly referred toas word sense disambiguation (WSD). The task of WSD consists of assigning the correctsense to words using an electronic dictionary as the source of word definitions. We presenttwo WSD methods based on two main methodological approaches in this research area: aknowledge-based method and a corpus-based method. Our hypothesis is that word-sensedisambiguation requires several knowledge sources in order to solve the semantic ambiguityof the words. These sources can be of different kinds— for example, syntagmatic, paradig-matic or statistical information. Our approach combines various sources of knowledge,through combinations of the two WSD methods mentioned above. Mainly, the paper con-centrates on how to combine these methods and sources of information in order to achievegood results in the disambiguation. Finally, this paper presents a comprehensive study andexperimental work on evaluation of the methods and their combinations.
1. Introduction
Knowledge technologies aim to provide meaning to the petabytes of information contentthat our multilingual societies will generate in the near future. Specifically, a wide range ofadvanced techniques are required to progressively automate the knowledge lifecycle. Theseinclude analyzing, and then automatically representing and managing, high-level meaningsfrom large collections of content data. However, to be able to build the next generationof intelligent open-domain knowledge application systems, we need to deal with conceptsrather than words.
c 2005 AI Access Foundation. All rights reserved.

 
Montoyo, Su´
arez, Rigau, & Palomar
1.1 Dealing with Word Senses
In natural language processing (NLP), word sense disambiguation (WSD) is defined as thetask of assigning the appropriate meaning (sense) to a given word in a text or discourse.As an example, consider the following three sentences:
1. Many cruise missiles have fallen on Baghdad.
2. Music sales will fall by up to 15% this year.
3. U.S. officials expected Basra to fall early.
Any system that tries to determine the meanings of the three sentences will need to
represent somehow three different senses for the verb fall. In the first sentence, the missileshave been launched on Baghdad. In the second sentence, sales will decrease, and in thethird the city will surrender early. WordNet 2.0 (Miller, 1995; Fellbaum, 1998)1 containsthirty-two different senses for the verb fall as well as twelve different senses for the nounfall. Note also that the first and third sentence belong to the same, military domain, butuse the verb fall with two different meanings.
Thus, a WSD system must be able to assign the correct sense of a given word, in
these examples, fall, depending on the context in which the word occurs. In the examplesentences, these are, respectively, senses 1, 2 and 9, as listed below.
• 1. fall—“descend in free fall under the influence of gravity” (“The branch fell from
the tree”; “The unfortunate hiker fell into a crevasse”).
• 2. descend, fall, go down, come down—“move downward but not necessarily all the
way” (“The temperature is going down”; “The barometer is falling”; “Real estateprices are coming down”).
• 9. fall—“be captured” (“The cities fell to the enemy”).
Providing innovative technology to solve this problem will be one of the main challenges
in language engineering to access advanced knowledge technology systems.
1.2 Word-Sense Disambiguation
Word sense ambiguity is a central problem for many established Human Language Tech-nology applications (e.g., machine translation, information extraction, question answering,information retrieval, text classification, and text summarization) (Ide & V´eronis, 1998).This is also the case for associated subtasks (e.g., reference resolution, acquisition of sub-categorization patterns, parsing, and, obviously, semantic interpretation). For this reason,many international research groups are working on WSD, using a wide range of approaches.However, to date, no large-scale, broad-coverage, accurate WSD system has been built (Sny-der & Palmer, 2004). With current state-of-the-art accuracy in the range 60–70%, WSD isone of the most important open problems in NLP.
1. http://www.cogsci.princeton.edu/˜wn/
300

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
Even though most of the techniques for WSD usually are presented as stand-alone
techniques, it is our belief, following McRoy (1992), that full-fledged lexical ambiguityresolution will require to integrate several information sources and techniques.
In this paper, we present two complementary WSD methods based on two different
methodological approaches, a knowledge-based and a corpus-based methods, as well as severalmethods that combine both into hybrid approaches.
The knowledge-based method disambiguates nouns by matching context with informa-
tion from a prescribed knowledge source. WordNet is used because it combines the char-acteristics of both a dictionary and a structured semantic network, providing definitionsfor the different senses of the English words and defining groups of synonymous words bymeans of synsets, which represent distinct lexical concepts. WordNet also organizes wordsinto a conceptual structure by representing a number of semantic relationships (hyponymy,hypernymy, meronymy, etc.) among synsets.
The corpus-based method implements a supervised machine-learning (ML) algorithm
that learns from annotated sense examples. The corpus-based system usually representslinguistic information for the context of each sentence (e.g., usage of an ambiguous word) inthe form of feature vectors. These features may be of a distinct nature: word collocations,part-of-speech labels, keywords, topic and domain information, grammatical relationships,etc. Based on these two approaches, the main objectives of the work presented in this paperare:
• To study the performance of different mechanisms of combining information sources
by using knowledge-based and corpus-based WSD methods together.
• To show that a knowledge-based method can help a corpus-based method to better
perform the disambiguation process and vice versa.
• To show that the combination of both approaches outperforms each of the methods
taken individually, demonstrating that the two approaches can play complementaryroles.
• Finally, to show that both approaches can be applied in several languages. In partic-
ular, we will perform several experiments in Spanish and English.
In the following section a summary of the background of word sense disambiguation is
presented. Sections 2.1 and 2.2 describe the knowledge-based and corpus-based systemsused in this work. Section 3 describes two WSD methods: the specification marks methodand the maximum entropy-based method. Section 4 presents an evaluation of our resultsusing different system combinations. Finally, some conclusions are presented, along with abrief discussion of work in progress.
2. Some Background on WSD
Since the 1950s, many approaches have been proposed for assigning senses to words incontext, although early attempts only served as models for toy systems. Currently, thereare two main methodological approaches in this area: knowledge-based and corpus-basedmethods. Knowledge-based methods use external knowledge resources, which define explicit
301

 
Montoyo, Su´
arez, Rigau, & Palomar
sense distinctions for assigning the correct sense of a word in context. Corpus-based methodsuse machine-learning techniques to induce models of word usages from large collections oftext examples. Both knowledge-based and corpus-based methods present different benefitsand drawbacks.
2.1 Knowledge-based WSD
Work on WSD reached a turning point in the 1980s and 1990s when large-scale lexicalresources such as dictionaries, thesauri, and corpora became widely available. The workdone earlier on WSD was theoretically interesting but practical only in extremely limiteddomains. Since Lesk (1986), many researchers have used machine-readable dictionaries(MRDs) as a structured source of lexical knowledge to deal with WSD. These approaches,by exploiting the knowledge contained in the dictionaries, mainly seek to avoid the need forlarge amounts of training material. Agirre and Martinez (2001b) distinguish ten differenttypes of information that can be useful for WSD. Most of them can be located in MRDs, andinclude part of speech, semantic word associations, syntactic cues, selectional preferences,and frequency of senses, among others.
In general, WSD techniques using pre-existing structured lexical knowledge resources
differ in:
• the lexical resource used (monolingual and/or bilingual MRDs, thesauri, lexical knowl-
edge base, etc.);
• the information contained in this resource, exploited by the method; and• the property used to relate words and senses.
Lesk (1986) proposes a method for guessing the correct word sense by counting word
overlaps between dictionary definitions of the words in the context of the ambiguous word.Cowie et al. (1992) uses the simulated annealing technique for overcoming the combinatorialexplosion of the Lesk method. Wilks et al. (1993) use co-occurrence data extracted from anMRD to construct word-context vectors, and thus word-sense vectors, to perform a large setof experiments to test relatedness functions between words and vector-similarity functions.
Other approaches measure the relatedness between words, taking as a reference a struc-
tured semantic net. Thus, Sussna (1993) employs the notion of conceptual distance betweennetwork nodes in order to improve precision during document indexing. Agirre and Rigau(1996) present a method for the resolution of the lexical ambiguity of nouns using the Word-Net noun taxonomy and the notion of conceptual density. Rigau et al. (1997) combine aset of knowledge-based algorithms to accurately disambiguate definitions of MRDs. Mihal-cea and Moldovan (1999) suggest a method that attempts to disambiguate all the nouns,verbs, adverbs, and adjectives in a given text by referring to the senses provided by Word-Net. Magnini et al. (2002) explore the role of domain information in WSD using WordNetdomains (Magnini & Strapparava, 2000); in this case, the underlying hypothesis is thatinformation provided by domain labels offers a natural way to establish semantic relationsamong word senses, which can be profitably used during the disambiguation process.
Although knowledge-based systems have been proven to be ready-to-use and scalable
tools for all-words WSD because they do not require sense-annotated data (Montoyo et al.,
302

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
2001), in general, supervised, corpus-based algorithms have obtained better precision thanknowledge-based ones.
2.2 Corpus-based WSD
In the last fifteen years, empirical and statistical approaches have had a significantly in-creased impact on NLP. Of increasing interest are algorithms and techniques that come fromthe machine-learning (ML) community since these have been applied to a large variety ofNLP tasks with remarkable success. The reader can find an excellent introduction to ML,and its relation to NLP, in the articles by Mitchell (1997), Manning and Sch¨
utze (1999), and
Cardie and Mooney (1999), respectively. The types of NLP problems initially addressed bystatistical and machine-learning techniques are those of language- ambiguity resolution, inwhich the correct interpretation should be selected from among a set of alternatives in aparticular context (e.g., word-choice selection in speech recognition or machine translation,part-of-speech tagging, word-sense disambiguation, co-reference resolution, etc.). Thesetechniques are particularly adequate for NLP because they can be regarded as classificationproblems, which have been studied extensively in the ML community. Regarding automaticWSD, one of the most successful approaches in the last ten years is supervised learningfrom examples, in which statistical or ML classification models are induced from semanti-cally annotated corpora. Generally, supervised systems have obtained better results thanunsupervised ones, a conclusion that is based on experimental work and international com-petitions2. This approach uses semantically annotated corpora to train machine–learning(ML) algorithms to decide which word sense to choose in which contexts. The words insuch annotated corpora are tagged manually using semantic classes taken from a particularlexical semantic resource (most commonly WordNet). Many standard ML techniques havebeen tried, including Bayesian learning (Bruce & Wiebe, 1994), Maximum Entropy (Su´arez& Palomar, 2002a), exemplar-based learning (Ng, 1997; Hoste et al., 2002), decision lists(Yarowsky, 1994; Agirre & Martinez, 2001a), neural networks (Towell & Voorhees, 1998),and, recently, margin-based classifiers like boosting (Escudero et al., 2000) and supportvector machines (Cabezas et al., 2001).
Corpus-based methods are called “supervised” when they learn from previously sense-
annotated data, and therefore they usually require a large amount of human interventionto annotate the training data (Ng, 1997). Although several attempts have been made (e.g.,Leackock et al., 1998; Mihalcea & Moldovan, 1999; Cuadros et al., 2004), the knowledgeacquisition bottleneck (too many languages, too many words, too many senses, too manyexamples per sense) is still an open problem that poses serious challenges to the supervisedlearning approach for WSD.
3. WSD Methods
In this section we present two WSD methods based, respectively, on the two main method-ological approaches outlined above: a specification marks method (SM) (Montoyo & Palo-mar, 2001) as a knowledge-based method, and a maximum entropy-based method (ME)(Su´
arez & Palomar, 2002b) as a corpus-based method. The selected methods can be seen
2. http://www.senseval.org
303

 
Montoyo, Su´
arez, Rigau, & Palomar
as representatives of both methodological approaches. The specification marks methodis inspired by the conceptual density method (Agirre & Rigau, 1996) and the maximumentropy method has been also used in other WSD systems (Dang et al., 2002).
3.1 Specification Marks Method
The underlying hypothesis of this knowledge base method is that the higher the similaritybetween two words, the larger the amount of information shared by two of its concepts. Inthis case, the information commonly shared by several concepts is indicated by the mostspecific concept that subsumes them in the taxonomy.
The input for this WSD module is a group of nouns W = w1,w2,...,wn in a con-
text. Each word wi is sought in WordNet, each having an associated set of possible sensesSi = Si1,Si2,...,Sin, and each sense having a set of concepts in the IS-A taxonomy (hy-
pernymy/hyponymy relations). First, this method obtains the common concept to all thesenses of the words that form the context. This concept is marked by the initial specifi-cation mark (ISM). If this initial specification mark does not resolve the ambiguity of theword, we then descend through the WordNet hierarchy, from one level to another, assigningnew specification marks. For each specification mark, the number of concepts containedwithin the subhierarchy is then counted. The sense that corresponds to the specificationmark with the highest number of words is the one chosen to be sense disambiguated withinthe given context. Figure 1 illustrates graphically how the word plant, having four differentsenses, is disambiguated in a context that also has the words tree, perennial, and leaf. Itcan be seen that the initial specification mark does not resolve the lexical ambiguity, sincethe word plant appears in two subhierarchies with different senses. The specification markidentified by plant2, flora2, however, contains the highest number of words (three)
from the context and will therefore be the one chosen to resolve the sense two of the wordplant. The words tree and perennial are also disambiguated, choosing for both the senseone. The word leaf does not appear in the subhierarchy of the specification mark plant2,
flora2, and therefore this word has not been disambiguated. These words are beyond
the scope of the disambiguation algorithm. They will be left aside to be processed by acomplementary set of heuristics (see section 3.1.2).
3.1.1 Disambiguation AlgorithmIn this section, we formally describe the SM algorithm which consists of the following fivesteps:
Step 1:All nouns are extracted from a given context. These nouns constitute the input context,Context = w1,w2,...,wn. For example, Context = plant,tree,perennial,leaf.Step 2:For each noun wi in the context, all its possible senses Si = Si1,Si2,...,Sin are
obtained from WordNet. For each sense Sij, the hypernym chain is obtained and storedin order into stacks. For example, Table 1 shows all the hypernyms synsets for eachsense of the word Plant.
Step 3:To each sense appearing in the stacks, the method associates the list of subsumed senses
304

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
 
Inicial 
 entity1 
Specification 
Mark (ISM) 
object1 
SM 
life form1 
SM 
natural object1 
substance1 
part4 
plant2, flora2  (*) 
artifact1 
person1 
material1 
section4 
plant part1 
perennial1 
vascular plant1 
Structure1 
paper1 
entertainer1 
plant organ1 
leaf3 
woody plant1 
building complex1 
sheet2 
performer1 
leaf1 
leaf2 
tree1 
plant1 
actor1 
plant4 
Figure 1: Specification Marks
plant1
plant2
plant3
plant4
building complex1
life form1
contrivance3
actor1
structure1
entity1
scheme1
performer1
artifact1
plan of action1
entertainer1
object1
plan1
person1
entity1
idea1
life form1
content5
entity1
cognition1psychological feature1
Table 1: Hypernyms synsets of plant
from the context (see Figure 2, which illustrates the list of subsumed senses for plant1and plant2 ).
Step 4:Beginning from the initial specification marks (the top synsets), the program descendsrecursively through the hierarchy, from one level to another, assigning to each specifi-cation mark the number of context words subsumed.
Figure 3 shows the word counts for plant1 through plant4 located within the speci-fication mark entity1, ..., life form1, flora2. For the entity1 specification mark,senses 1, 2, and 4 have the same maximal word counts (4). Therefore, it is notpossible to disambiguate the word plant using the entity1 specification mark, andit will be necessary to go down one level of the hyponym hierarchy by changing thespecification mark. Choosing the specification mark life form1, senses 2 and 4 ofplant have the same maximal word counts (3). Finally, it is possible to disambiguatethe word plant with the sense 2 using the plant2, flora2 specification mark,
because of this sense has the higher word density (in this case, 3).
305

 
Montoyo, Su´
arez, Rigau, & Palomar
For PLANT: 
For PLANT1: 
 
plant1 
 plant1 
 
building complex1 
 plant1 
 
structure1 
 plant1 
 
artifact1 
 plant1 
 
object1 
 plant1, leaf1, leaf2, leaf3 
entity1 
 plant1, plant2, plant4, tree1, perennial1, leaf1, leaf2, leaf3 
For PLANT2: 
 
plant2 
 plant2, tree1, perennial1 
 
life form1 
 plant2, plant4, tree1, perennial1 
entity1 
 plant1, plant2, plant4, tree1, perennial1, leaf1, leaf2, leaf3 
Figure 2: Data Structure for Senses of the Word Plant
For PLANT  
located within the specification mark entity1 
 
       For PLANT1 : 4 (plant, tree, perennial, leaf) 
 
       For PLANT2 : 4 (plant, tree, perennial, leaf) 
 
       For PLANT3 : 1 (plant) 
 
       For PLANT4 : 4 (plant, tree, perennial, leaf)                               …………… located within the specification mark life form1 
 
       For PLANT1 : 1 (plant) 
 
       For PLANT2 : 3 (plant, tree, perennial) 
 
       For PLANT3 : 1 (plant) 
 
       For PLANT4 : 3 (plant, tree, perennial) located within the specification mark plant 2, flora2 
 
       For PLANT1 : 1 (plant) 
 
       For PLANT2 : 3 (plant, tree, perennial) 
 
       For PLANT3 : 1 (plant) 
 
       For PLANT4 : 1 (plant) 
 
Figure 3: Word Counts for Four Senses of the Word Plant
Step 5:In this step, the method selects the word sense(s) having the greatest number of wordscounted in Step 4. If there is only one sense, then that is the one that is obviously chosen.If there is more than one sense, we repeat Step 4, moving down each level within thetaxonomy until a single sense is obtained or the program reach a leaf specificationmark. Figure 3 shows the word counts for each sense of plant (1 through 4) locatedwithin the specification mark entity1, ..., life form1, flora2. If the word cannotbe disambiguated in this way, then it will be necessary to continue the disambiguationprocess applying a complementary set of heuristics.
3.1.2 Heuristics
The specification marks method is combined with a set of five knowledge-based heuristics:hypernym/hyponym, definition, gloss hypernym/hyponym, common specification mark,and domain heuristics. A short description of each of these methods is provided below.
306

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
3.1.3 Hypernym/Hyponym Heuristic
This heuristic solves the ambiguity of those words that are not explicitly related in WordNet(i.e., leaf is not directly related to plant, but rather follows a hypernym chain plus a PART-OF relation). All the hypernyms/hyponyms of the ambiguous word are checked, lookingfor synsets that have compounds that match with some word from the context. Eachsynset in the hypernym/hyponym chain is weighted in accordance with its depth within thesubhierarchy. The sense then having the greatest weight is chosen. Figure 4 shows that,leaf1 being a hyponym of plant organ1 is disambiguated (obtain the greatest weight,weight(leaf 1) =
depthi=1 (
level
) = ( 4 ) + ( 5 ) = 1.5) because plant is contained within
total levels
6
6
the context of leaf.
Context: plant, tree, leaf, perennial Word non disambiguated: leaf. Senses: leaf1, leaf2, leaf3. 
 
For leaf1  
 
 
 
 
=> entity, something   
 
 
Level 1 
   => object, physical object 
 
Level 2 
 
            => natural object 
 
 
Level 3 
         => plant part 
 
 
Level 4 
            => plant organ 
 
 
Level 5 
               => leaf1, leafage, foliage 
Level 6 
 
Figure 4: Application of the Hypernym Heuristic
3.1.4 Definition Heuristic
In this case, all the glosses from the synsets of an ambiguous word are checked looking forthose that contain words from the context. Each match increases the synset count by one.The sense having the greatest count is then chosen. Figure 5 shows an example of thisheuristic. The sense sister1 is chosen, because it has the greatest weight.
Context: person, sister, musician. Words non disambiguated: sister,musician. Senses: sister1, sister2, sister3 sister4. 
 
For sister1   Weight = 2   1.  sister,  sis  -  (a  female  person  who  has  the  same  parents  as  another  person;  "my  sister  married  a musician") 
 
For sister3   Weight = 1 
 
3.  sister  -  (a  female  person  who  is  a  fellow  member  (of  a  sorority  or  labor  union  or  other  group); "none of her sisters would betray her") 
 
Figure 5: Application of the Definition Heuristic
307

 
Montoyo, Su´
arez, Rigau, & Palomar
3.1.5 Gloss Hypernym/Hyponym Heuristic
This method extends the previously defined hypernym/hyponym heuristic by using glossesof the hypernym/hyponym synsets of the ambiguous word. To disambiguate a given word,all the glosses of the hypernym/hyponym synsets are checked looking for words occurringin the context. Coincidences are counted. As before, the synset having the greatest count ischosen. Figure 6 shows a example of this heuristic. The sense plane1 is chosen, becauseit has the greatest weight.
Context: plane, air Words non disambiguated: plane Senses: plane1, plane2, plane3, plane4, plane5.  For Plane1: 
 Weight = 1 
 airplane, aeroplane, plane - (an aircraft that has fixed a wing and is powered by propellers or jets; "the flight was delayed due to trouble with the airplane")        => aircraft - (a vehicle that can fly)            => craft - (a vehicle designed for navigation in or on water or air or through outer space)                => vehicle - (a conveyance that transports people or objects)                    => conveyance, transport - (something that serves as a means of transportation)                        =>  instrumentality,  instrumentation  -  (an  artifact  (or  system  of  artifacts)  that  is 
instrumental in accomplishing some end) 
                           => artifact, artefact - (a man-made object)                                =>  object,  physical  object  -  (a  physical  (tangible  and  visible)  entity;  "it  was  full  of 
rackets, balls and other objects") 
                                   => entity, something - (anything having existence (living or nonliving))  
Figure 6: Application of the Gloss Hypernym Heuristic
3.1.6 Common Specification Mark Heuristic
In most cases, the senses of the words to be disambiguated are very close to each other andonly differ in subtle differences in nuances. The Common Specification Mark heuristic reducethe ambiguity of a word without trying to provide a full disambiguation. Thus, we selectthe specification mark that is common to all senses of the context words, reporting all sensesinstead of choosing a single sense from among them. To illustrate this heuristic, considerFigure 7. In this example, the word month is not able to discriminate completely amongfour senses of the word year. However, in this case, the presence of the word month canhelp to select two possible senses of the word year when selecting the time period, period asa common specification mark. This specification mark represents the most specific commonsynset of a particular set of words. Therefore, this heuristic selects the sense month1 andsenses year1 and year2 instead of attempting to choose a single sense or leaving themcompletely ambiguous.
3.1.7 Domain WSD Heuristic
This heuristic uses a derived resource, “relevant domains” (Montoyo et al., 2003), which isobtained combining both the WordNet glosses and WordNet Domains (Magnini & Strap-
308

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
Context: year, month. Words non disambiguated: year. Senses: year1, year2, year3, year4. 
 
For year1:  
 
 
For year2: 
 
 
For month1: 
 => abstraction 
 
 
=> abstraction 
 
            => abstraction 
     => measure, quantity   
     => measure, quantity   
  => measure, quantity 
          => time period, period 
         => time period, period 
    => time period, period 
=> year1, twelvemonth              => year2 
 
        => month1 
Figure 7: Example of Common Specification Mark Heuristic
parava, 2000)3. WordNet Domains establish a semantic relation between word senses bygrouping them into the same semantic domain (Sports, Medicine, etc.). The word bank,for example, has ten senses in WordNet 2.0, but three of them, “bank1”, “bank3”and “bank6” are grouped into the same domain label, Economy, whereas “bank2” and“bank7” are grouped into the domain labels Geography and Geology. These domain la-bels are selected from a set of 165 labels hierarchically organized. In that way, a domainconnects words that belong to different subhierarchies and part–of-speech.
“Relevant domains” is a lexicon derived from the WordNet glosses using WordNet Do-
mains. In fact, we use WordNet as a corpus categorized with domain labels. For eachEnglish word appearing in the gloses of WordNet, we obtain a list of their most repre-sentative domain labels. The relevance is obtained weighting each possible label with the”Association Ratio” formula (AR), where w is a word and D is a domain.
P (w
AR(w|D) = P(w|D) ∗ log
|D)
(1)
P (w)
This list can also be considered as a weighted vector (or point in a multidimensional
space). Using such word vectors of “Relevant domains”, we can derive new vectors torepresent sets of words—for instance, for contexts or glosses. We can then compare thesimilarity between a given context and each of the possible senses of a polysemous word—by using for instance the cosine function.
Figure 8 shows an example for disambiguating the word genotype in the following text:
There are a number of ways in which the chromosome structure can change, which willdetrimentally change the genotype and phenotype of the organism. First, the glosses of theword to be disambiguated and the context are pos–tagged and analyzed morphologically.Second, we build the context vector (CV) which combines in one structure the most rele-vant and representative domains related to the words from the text to be disambiguated.Third, in the same way, we build the sense vectors (SV) which group the most relevant andrepresentative domains of the gloss that is associated with each one of the word senses. Inthis example, genotype1 – (a group of organisms sharing a specific genetic constitution)and genotype2 – (the particular alleles at specified loci present in an organism). Finally,in order to select the appropriate sense, we made a comparison between all sense vectorsand the context vector, and we select the senses more approximate to the context vector. In
3. http://wndomains.itc.it/
309

 
Montoyo, Su´
arez, Rigau, & Palomar
this example, we show the sense vector for sense genotype1 and we select the genotype1
 
sense, b
  ecause its cosine is higher. 
 
 
       

AR
      



 Bio log y
  
0.00102837 

AR
    




Ecology 
0.00402855 


 Ecology 
0.084778

Botany
3.20408e - 06

Biology
0.047627




Zoology
1.77959e - 05
Bowling
0.019687






CV =  Anatomy
1.29592e - 05                         SV =
y
Archaeolog
0.016451  




 Physiology
1
0.00022653 
 Sociology
0.014251



Chemistry
0.000179857 
on
Alimentati
0.006510




 Geology
1.66327e - 05

s
Linguistic
0.005297




y
Meteorolog
0.00371308

...
...




...
...

     Sense vector genotype1.  
        Context vector 
 
  
  
 
 
 
 
       Selected sense 
 
genotype1 = 0.00804111 
genotype2 = 0.00340548 
 
 
Figure 8: Example of Domain WSD Heuristic
Defining this heuristic as “knowledge-based” or “corpus-based” can be seen controver-
sial because this heuristic uses WordNet gloses (and WordNet Domains) as a corpus toderive the ”relevant domains”. That is, using corpus techniques on WordNet. However,WordNet Domains was constructed semi-automatically (prescribed) following the hierarchyof WordNet.
3.1.8 Evaluation of Specification Marks Method
Obviously, we can also use different strategies to combine a set of knowledge-based heuris-tics. For instance, all the heuristics described in the previous section can be applied inorder passing to the next heuristic only the remaining ambiguity that previous heuristicswere not able to solve.
In order to evaluate the performance of the knowledge-based heuristics previously de-
fined, we used the SemCor collection (Miller et al., 1993), in which all content words areannotated with the most appropriate WordNet sense.In this case, we used a window offifteen nouns (seven context nouns before and after the target noun).
The results obtained for the specification marks method using the heuristics when ap-
plied one by one are shown in Table 2. This table shows the results for polysemous nounsonly, and for polysemous and monosemous nouns combined.
310

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
Heuristics
Precision Recall Coverage
Polysemic and monosemic nouns
0.553
0.522
0.943
Only polysemic nouns
0.377
0.311
0.943
Table 2: Results Using Heuristics Applied in Order on SemCor
The results obtained for the heuristics applied independently are shown in Table 3. As
shown, all the heuristics perform differently, providing different precision/recall figures.
Heuristics
Precision
Recall
Coverage
Mono+Poly Polysemic Mono+Poly Polysemic Mono+Poly Polysemic
Spec. Mark Method
0.383
0.300
0.341
0.292
0.975
0.948
Hypernym
0.563
0.420
0.447
0.313
0.795
0.745
Definition
0.480
0.300
0.363
0.209
0.758
0.699
Hyponym
0.556
0.393
0.436
0.285
0.784
0.726
Gloss hypernym
0.555
0.412
0.450
0.316
0.811
0.764
Gloss hyponym
0.617
0.481
0.494
0.358
0.798
0.745
Common specification
0.565
0.423
0.443
0.310
0.784
0.732
Domain WSD
0.585
0.453
0.483
0.330
0.894
0.832
Table 3: Results Using Heuristics Applied Independently
Another possibility is to combine all the heuristics using a majority voting schema (Rigau
et al., 1997). In this simple schema, each heuristic provides a vote, and the method selectsthe synset that obtains more votes. The results shown in Table 4 illustrate that when theheuristics are working independently, the method achieves a 39.1% recall for polysemousnouns (with full coverage), which represents an improvement of 8 percentual points overthe method in which heuristics are applied in order (one by one).
Precision
Recall
Mono+Poly Polysemic Mono+Poly Polysemic
Voting heuristics
0.567
0.436
0.546
0.391
Table 4: Results using majority voting on SemCor
We also show in Table 5 the results of our domain heuristic when applied on the English
all-words task from Senseval-2. In the table, the polysemy reduction caused by domainclustering can profitably help WSD. Since domains are coarser than synsets, word domaindisambiguation (WDD) (Magnini & Strapparava, 2000) can obtain better results than WSD.Our goal is to perform a preliminary domain disambiguation in order to provide an informedsearch–space reduction.
3.1.9 Comparison with Knowledge-based Methods
In this section we compare three different knowledge-based methods: conceptual density(Agirre & Rigau, 1996), a variant of the conceptual density algorithm (Fern´
andez-Amor´
os
et al., 2001); the Lesk method (Lesk, 1986) ; and the specification marks method.
311

 
Montoyo, Su´
arez, Rigau, & Palomar
Level WSD Precision Recall
Sense
0.44
0.32
Domain
0.54
0.43
Table 5: Results of Use of Domain WSD Heuristic
Table 6 shows recall results for the three methods when applied to the entire SemCor
collection. Our best result achieved 39.1% recall. This is an important improvement withrespect to other methods, but the results are still far below the most frequent sense heuris-tic. Obviously, none of the knowledge-based techniques and heuristics presented above aresufficient, in isolation, to perform accurate WSD. However, we have empirically demon-strated that a simple combination of knowledge-based heuristics can lead to improvementsin the WSD process.
WSD Method
Recall
SM and Voting Heuristics
0.391
UNED Method
0.313
SM with Cascade Heuristics
0.311
Lesk
0.274
Conceptual Density
0.220
Table 6: Recall results using three different knowledge–based WSD methods
3.2 Maximum Entropy-based Method
Maximum Entropy modeling provides a framework for integrating information for classi-fication from many heterogeneous information sources (Manning & Sch¨
utze, 1999; Berger
et al., 1996). ME probability models have been successfully applied to some NLP tasks,such as POS tagging or sentence-boundary detection (Ratnaparkhi, 1998).
The WSD method used in this work is based on conditional ME models. It has been
implemented using a supervised learning method that consists of building word-sense clas-sifiers using a semantically annotated corpus. A classifier obtained by means of an MEtechnique consists of a set of parameters or coefficients which are estimated using an opti-mization procedure. Each coefficient is associated with one feature observed in the trainingdata. The goal is to obtain the probability distribution that maximizes the entropy—thatis, maximum ignorance is assumed and nothing apart from the training data is considered.One advantage of using the ME framework is that even knowledge-poor features may be ap-plied accurately; the ME framework thus allows a virtually unrestricted ability to representproblem-specific knowledge in the form of features (Ratnaparkhi, 1998).
Let us assume a set of contexts X and a set of classes C. The function cl : X → C chooses
the class c with the highest conditional probability in the context x: cl(x) = arg maxc p(c|x).
Each feature is calculated by a function that is associated with a specific class c , and ittakes the form of equation (2), where cp(x) represents some observable characteristic in
312

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
the context4. The conditional probability p(c|x) is defined by equation (3), where αi is
the parameter or weight of the feature i, K is the number of features defined, and Z(x)is a normalization factor that ensures that the sum of all conditional probabilities for thiscontext is equal to 1.
1 if c = c and cp(x) = true
f (x, c) =
(2)
0 otherwise
1
K
p(c|x) =
αfi(x,c)
Z(x)
i
(3)
i=1
The learning module produces the classifiers for each word using a corpus that is syn-
tactically and semantically annotated. The module processes the learning corpus in orderto define the functions that will apprise the linguistic features of each context.
For example, consider that we want to build a classifier for the noun interest using
the POS label of the previous word as a feature and we also have the the following threeexamples from the training corpus:
... the widespread interest1 in the ...... the best interest5 of both ...... persons expressing interest1 in the ...
The learning module performs a sequential processing of this corpus, looking for the
pairs <POS-label, sense>. Then, the following pairs are used to define three functions(each context has a vector composed of three features).
<adjective,1><adjective,5><verb,1>
We can define another type of feature by merging the POS occurrences by sense:
< adjective,verb,1>
<adjective,5>
This form of defining the pairs means a reduction of feature space because all information
(of some kind of linguistic data, e.g., POS label at position -1) about a sense is contained injust one feature. Obviously, the form of the feature function 2 must be adapted to Equation4. Thus,
W(c ) = data of sense c 
(4)
1 if c = c and CP (x)
f
∈ W(c )
(c ,i)(x, c) =
0 otherwise
4. The ME approach is not limited to binary features, but the optimization procedure used for the estimation
of the parameters, the Generalized Iterative Scaling procedure, uses this kind of features.
313

 
Montoyo, Su´
arez, Rigau, & Palomar
We will refer to the feature function expressed by Equation 4 as “collapsed features”.
The previous Equation 2 we call “non-collapsed features”. These two feature definitions arecomplementary and can be used together in the learning phase.
Due to the nature of the disambiguation task, the number of times that a feature
generated by the first type of function (“non-collapsed”) is activated is very low, and thefeature vectors have a large number of null values. The new function drastically reducesthe number of features, with a minimal degradation in the evaluation results. In this way,more and new features can be incorporated into the learning process, compensating the lossof accuracy.
Therefore, the classification module carries out the disambiguation of new contexts using
the previously stored classification functions. When ME does not have enough informationabout a specific context, several senses may achieve the same maximum probability andthus the classification cannot be done properly. In these cases, the most frequent sense inthe corpus is assigned. However, this heuristic is only necessary for a minimum number ofcontexts or when the set of linguistic attributes processed is very small.
3.2.1 Description of Features
The set of features defined for the training of the system is described in Figure 9 and isbased on the features described by Ng and Lee (1996) and Escudero et al. (2000). Thesefeatures represent words, collocations, and POS tags in the local context. Both “collapsed”and “non-collapsed” functions are used.
• 0: word form of the target word
• s: words at positions ±1, ±2, ±3
• p: POS-tags of words at positions ±1, ±2, ±3
• b: lemmas of collocations at positions (−2,−1), (−1,+1), (+1,+2)
• c: collocations at positions (−2,−1), (−1,+1), (+1,+2)
• km: lemmas of nouns at any position in context, occurring at least m% times with a sense
• r: grammatical relation to the ambiguous word
• d: the word that the ambiguous word depends on
• m: multi-word if identified by the parser
• L: lemmas of content-words at positions ±1, ±2, ±3 (“collapsed” definition)
• W : content-words at positions ±1, ±2, ±3 (“collapsed” definition)
• S, B, C, P, D and M : “collapsed” versions (see Equation 4)
Figure 9: Features Used for the Training of the System
Actually, each item in Figure 9 groups several sets of features. The majority of them
depend on the nearest words (e.g., s comprises all possible features defined by the wordsoccurring in each sample at positions w−3, w−2, w−1, w+1, w+2, w+3 related to the am-
biguous word). Types nominated with capital letters are based on the “collapsed” functionform; that is, these features simply recognize an attribute belonging to the training data.
Keyword features (km) are inspired by Ng and Lee work. Noun filtering is done using
frequency information for nouns co-occurring with a particular sense. For example, let us
314

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
suppose m = 10 for a set of 100 examples of interest4 : if the noun bank is found 10 timesor more at any position, then a feature is defined.
Moreover, new features have also been defined using other grammatical properties: re-
lationship features (r) that refer to the grammatical relationship of the ambiguous word(subject, object, complement, ...) and dependency features (d and D) that extract the wordrelated to the ambiguous one through the dependency parse tree.
3.2.2 Evaluation of the Maximum Entropy Method
In this subsection we present the results of our evaluation over the training and testing dataof the Senseval-2 Spanish lexical–sample task. This corpus was parsed using ConexorFunctional Dependency Grammar parser for Spanish (Tapanainen & J¨arvinen, 1997).
The classifiers were built from the training data and evaluated over the test data. Table
7 shows which combination of groups of features works better for every POS and whichwork better for all words together.
Accuracy Feature selection
Nouns
0.683
LWSBCk5
Verbs
0.595
sk5
Adjectives
0.783
LWsBCp
ALL
0.671
0LWSBCk5
Table 7: Baseline: Accuracy Results of Applying ME on Senseval-2 Spanish Data
This work entails an exhaustive search looking for the most accurate combination of
features. The values presented here are merely informative and indicate the maximumaccuracy that the system can achieve with a particular set of features.
3.3 Improving ME accuracy
Our main goal is to find a method that will automatically obtain the best feature selection(Veenstra et al., 2000; Mihalcea, 2002; Su´arez & Palomar, 2002b) from the training data.We performed an 3-fold cross-validation process. Data is divided in 3 folds; then, 3 testsare done, each one with 2 folds as training data and the remaining one as testing data. Thefinal result is the average accuracy. We decided on just three tests because of the smallsize of the training data. Then, we tested several combinations of features over the trainingdata of the Senseval-2 Spanish lexical–sample and analyzed the results obtained for eachword.
In order to perform the 3-fold cross-validation process on each word, some preprocessing
of the corpus was done. For each word, all senses were uniformly distributed into the threefolds (each fold contains one-third of the examples of each sense). Those senses that hadfewer than three examples in the original corpus file were rejected and not processed.
Table 8 shows the best results obtained using three-fold cross-validation on the training
data. Several feature combinations were tested in order to find the best set for each selectedword. The purpose was to obtain the most relevant information for each word from thecorpus rather than applying the same combination of features to all of them. Therefore,the information in the column Features lists only the feature selection with the best result.
315

 
Montoyo, Su´
arez, Rigau, & Palomar
Word Features
Accur MFS
Word Features
Accur MFS
autoridad,N sbcp
0.589 0.503
clavar,V sbcprdk3
0.561 0.449
bomba,N 0LWSBCk5
0.762 0.707 conducir,V LWsBCPD 0.534 0.358
canal,N sbcprdk3
0.579 0.307
copiar,V 0sbcprdk3 0.457 0.338
circuito,N 0LWSBCk5
0.536 0.392
coronar,V sk5
0.698 0.327
coraz´on,N 0Sbcpk5
0.781 0.607 explotar,V 0LWSBCk5 0.593 0.318
corona,N sbcp
0.722 0.489
saltar,V LWsBC
0.403 0.132
gracia,N 0sk5
0.634 0.295
tocar,V 0sbcprdk3 0.583 0.313
grano,N 0LWSBCr
0.681 0.483
tratar,V sbcpk5
0.527 0.208
hermano,N 0Sprd
0.731 0.602
usar,V 0Sprd
0.732 0.669
masa,N LWSBCk5
0.756 0.455
vencer,V sbcprdk3
0.696 0.618
naturaleza,N sbcprdk3
0.527 0.424 brillante,A sbcprdk3
0.756 0.512
operaci´on,N 0LWSBCk5
0.543 0.377
ciego,A 0spdk5
0.812 0.565
´organo,N 0LWSBCPDk5 0.715 0.515
claro,A 0Sprd
0.919 0.854
partido,N 0LWSBCk5
0.839 0.524
local,A 0LWSBCr 0.798 0.750
pasaje,N sk5
0.685 0.451
natural,A sbcprdk10 0.471 0.267
programa,N 0LWSBCr
0.587 0.486 popular,A sbcprdk10 0.865 0.632
tabla,N sk5
0.663 0.488
simple,A LWsBCPD 0.776 0.621
actuar,V sk5
0.514 0.293
verde,A LWSBCk5 0.601 0.317
apoyar,V 0sbcprdk3
0.730 0.635
vital,A Sbcp
0.774 0.441
apuntar,V 0LWsBCPDk5 0.661 0.478
Table 8: Three-fold Cross-Validation Results on Senseval-2 Spanish Training Data: Best Aver-
aged Accuracies per Word
Strings in each row represent the entire set of features used when training each classifier. Forexample, autoridad obtains its best result using nearest words, collocations of two lemmas,collocations of two words, and POS information that is, s, b, c, and p features, respectively(see Figure 9). The column Accur (for “accuracy”) shows the number of correctly classifiedcontexts divided by the total number of contexts (because ME always classifies precision asequal to recall). Column MFS shows the accuracy obtained when the most frequent senseis selected.
The data summarized in Table 8 reveal that using “collapsed” features in the ME method
is useful; both “collapsed” and “non-collapsed” functions are used, even for the same word.For example, the adjective vital obtains the best result with “Sbcp” (the “collapsed” versionof words in a window (−3.. + 3), collocations of two lemmas and two words in a window
(−2.. + 2), and POS labels, in a window (−3.. + 3) too); we can here infer that single-word
information is less important than collocations in order to disambiguate vital correctly.
The target word (feature 0) is useful for nouns, verbs, and adjectives, but many of the
words do not use it for their best feature selection. In general, these words do not have arelevant relationship between shape and senses. On the other hand, POS information (pand P features) is selected less often. When comparing lemma features with word features(e.g., L versus W , and B versus C), they are complementary in the majority of cases.Grammatical relationships (r features) and word–word dependencies (d and D features)seem very useful, too, if combined with other types of attributes. Moreover, keywords (km
316

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
features) are used very often, possibly due to the source and size of contexts of Senseval-2Spanish lexical–sample data.
Table 9 shows the best feature selections for each part-of-speech and for all words. The
data presented in Tables 8 and 9 were used to build four different sets of classifiers in orderto compare their accuracy: MEfix uses the overall best feature selection for all words;MEbfs trains each word with its best selection of features (in Table 8); MEbfs.pos usesthe best selection per POS for all nouns, verbs and adjectives, respectively (in Table 9); and,finally, vME is a majority voting system that has as input the answers of the precedingsystems.
POS
Acc Features
System
Nouns
0.620 LWSBCk5
Verbs
0.559 sbcprdk3
MEbfs.pos
Adjectives 0.726 0spdk5ALL
0.615 sbcprdk3
MEfix
Table 9: Three-fold Cross-Validation Results on Senseval-2 Spanish Training Data: Best Aver-
aged Accuracies per POS
Table 10 shows a comparison of the four systems. MEfix has the lower results. This
classifier applies the same set of types of features to all words. However, the best featureselection per word (MEbfs) is not the best, probably because more training examples arenecessary. The best choice seems to select a fixed set of types of features for each POS(MEbfs.pos).
ALL
Nouns
0.677 MEbfs.pos 0.683 MEbfs.pos0.676 vME
0.678 vME
0.667 MEbfs
0.661 MEbfs
0.658 MEfix
0.646 MEfix
Verbs
Adjectives
0.583 vME
0.774 vME
0.583 MEbfs.pos 0.772 MEbfs.pos0.583 MEfix
0.771 MEbfs
0.580 MEbfs
0.756 MEfix
MEfix: sbcprdk3 for all words
MEbfs: each word with its
best feature selection
MEbfs.pos: LWSBCk5 for nouns,
sbcprdk3 for verbs,and 0spdk5 for adjectives
vME: majority voting between MEfix,
MEbfs.pos, and MEbfs
Table 10: Evaluation of ME Systems
317

 
Montoyo, Su´
arez, Rigau, & Palomar
While MEbfs predicts, for each word over the training data, which individually selected
features could be the best ones when evaluated on the testing data, MEbfs.pos is anaveraged prediction, a selection of features that, over the training data, performed a “goodenough” disambiguation of the majority of words belonging to a particular POS. When thisaveraged prediction is applied to the real testing data, MEbfs.pos performs better thanMEbfs.
Another important issue is that MEbfs.pos obtains an accuracy slightly better than
the best possible evaluation result achieved with ME (see Table 7)—that is, a best-feature-selection per POS strategy from training data guarantees an improvement on ME-basedWSD.
In general, verbs are difficult to learn and the accuracy of the method for them is lower
than for other POS; in our opinion, more information (knowledge-based, perhaps) is neededto build their classifiers. In this case, the voting system (vME) based on the agreementbetween the other three systems, does not improve accuracy.
Finally in Table 11, the results of the ME method are compared with those systems that
competed at Senseval-2 in the Spanish lexical–sample task5. The results obtained by MEsystems are excellent for nouns and adjectives, but not for verbs. However, when comparingALL POS, the ME systems seem to perform comparable to the best Senseval-2 systems.
ALL
Nouns
Verbs
Adjectives
0.713 jhu(R)
0.702 jhu(R)
0.643 jhu(R)
0.802 jhu(R)
0.682 jhu
0.683 MEbfs.pos 0.609 jhu
0.774 vME
0.677 MEbfs.pos 0.681 jhu
0.595 css244
0.772 MEbfs.pos
0.676 vME
0.678 vME
0.584 umd-sst
0.772 css244
0.670 css244
0.661 MEbfs
0.583 vME
0.771 MEbfs
0.667 MEbfs
0.652 css244
0.583 MEbfs.pos 0.764 jhu
0.658 MEfix
0.646 MEfix
0.583 MEfix
0.756 MEfix
0.627 umd-sst
0.621 duluth 8
0.580 MEbfs
0.725 duluth 8
0.617 duluth 8
0.612 duluth Z
0.515 duluth 10
0.712 duluth 10
0.610 duluth 10
0.611 duluth 10
0.513 duluth 8
0.706 duluth 7
0.595 duluth Z
0.603 umd-sst
0.511 ua
0.703 umd-sst
0.595 duluth 7
0.592 duluth 6
0.498 duluth 7
0.689 duluth 6
0.582 duluth 6
0.590 duluth 7
0.490 duluth Z
0.689 duluth Z
0.578 duluth X
0.586 duluth X
0.478 duluth X
0.687 ua
0.560 duluth 9
0.557 duluth 9
0.477 duluth 9
0.678 duluth X
0.548 ua
0.514 duluth Y
0.474 duluth 6
0.655 duluth 9
0.524 duluth Y
0.464 ua
0.431 duluth Y
0.637 duluth Y
Table 11: Comparison with the Spanish Senseval-2 systems
5. JHU(R) by Johns Hopkins University; CSS244 by Stanford University; UMD-SST by the University of
Maryland; Duluth systems by the University of Minnesota - Duluth; UA by the University of Alicante.
318

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
3.4 Comparing Specification Marks Method with Maximum Entropy-based
Method
The main goal of this section is to evaluate the Specification Marks Method (Montoyo &Su´
arez, 2001) and the Maximum Entropy-based Method (in particular, MEfix System)
on a common data set, to allow for direct comparisons. The individual evaluation of eachmethod has been carried out on the noun set (17 nouns) of the Spanish lexical-sample task(Rigau et al., 2001) from Senseval-26. Table 12 shows precision, recall and coverage ofboth methods.
Precision Recall Coverage
SM
0.464
0.464
0.941
ME
0.646
0.646
1
Table 12: Comparison of ME and SM for nouns in Senseval-2 Spanish lexical sample
In order to study a possible cooperation between both methods, we count those cases
that: the two methods return the correct sense for the same occurrence, at least one of themethods provides the correct sense and finally, none of both provides the correct sense. Asummary of the obtained results is shown in the Table 13. These results clearly show thatthere is a large room for improvement when combining both system outcomes. In fact, theyprovide also a possible upper bound precision for this technology, which can be set to 0.798(more than 15 percentual points higher than the current best system). Table 14 presentsa complementary view: wins, ties and loses between ME and SM when each context isexamined. Although ME performs better than SM, there are 122 cases (15 %) solved onlyby the SM method.
Contexts Percentage
Both OK
240
0.300
One OK
398
0.498
Zero OK
161
0.202
Table 13: Correct classifications of ME and SM for nouns in Senseval-2 Spanish lexical sample
Wins Ties Loses
ME – SM
267
240
122
Table 14: Wins, ties and loses of ME and SM systems for nouns in Senseval-2 Spanish lexical
sample
6. For both Spanish and English Senseval-2 corpora, when applying the Specification Marks method we
used the whole example as the context window for the target noun
319

 
Montoyo, Su´
arez, Rigau, & Palomar
4. Experimental Work
In this section we attempt to confirm our hypothesis that both corpus-based and knowledge-based methods can improve the accuracy of each other. The first subsection shows theresults of preprocessing the test data with the maximum entropy method (ME) in order tohelp the specification marks method (SM). Next, we test the opposite, if preprocessing thetest data with the domain heuristic can help the maximum entropy method to disambiguatemore accurately.
The last experiment combines the vME system (the majority voting system) and SM
method. Actually, it relies on simply adding the SM as one more heuristic to the votingscheme.
4.1 Integrating a Corpus-based WSD System into a Knowledge-based WSD
System
This experiment was designed to study and evaluate whether the integration of corpus-basedsystem within a knowledge-based helps improve word-sense disambiguation of nouns.
Therefore, ME can help to SM by labelling some nouns of the context of the target word.
That is, reducing the number of possible senses of some nouns of the context. In fact, wereduce the search space of the SM method. This ensures that the sense of the target wordwill be the one more related to the noun senses labelled by ME.
In this case, we used the noun words from the English lexical-sample task from Senseval-
2. ME helps SM by labelling some words from the context of the target word. These wordswere sense tagged using the SemCor collection as a learning corpus. We performed a three–fold cross-validation for all nouns having 10 or more occurrences. We selected those nounsthat were disambiguated by ME with high precision, that is, nouns that had percentagerates of accuracy of 90% or more. The classifiers for these nouns were used to disambiguatethe testing data. The total number of different noun classifiers (noun) activated for eachtarget word across the testing corpus is shown in Table 15.
Next, SM was applied, using all the heuristics for disambiguating the target words of
the testing data, but with the advantage of knowing the senses of some nouns that formedthe context of these targets words.
Table 15 shows the results of precision and recall when SM is applied with and without
first applying ME, that is, with and without fixing the sense of the nouns that form thecontext. A very small but consistent improvement was obtained through the completetest set (3.56% precision and 3.61% recall). Although the improvement is very low, thisexperiment empirically demonstrates that a corpus-based method such as maximum entropycan be integrated to help a knowledge-based system such as the specification marks method.
4.2 Integrating a Knowledge-based WSD system into a Corpus-based WSD
system
In this case, we used only the domain heuristic to improve ME because this information canbe added directly as domain features. The problem of data sparseness from which a WSDsystem based on features suffers could be increased by the fine-grained sense distinctionsprovided by WordNet. On the contrary, the domain information significantly reduces the
320

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
Without fixed senses With fixed senses
Target words
noun classifiers Precision
Recall
Precision Recall
art
63
0.475
0.475
0.524
0.524
authority
80
0.137
0.123
0.144
0.135
bar
104
0.222
0.203
0.232
0.220
bum
37
0.421
0.216
0.421
0.216
chair
59
0.206
0.190
0.316
0.301
channel
32
0.500
0.343
0.521
0.375
child
59
0.500
0.200
0.518
0.233
church
50
0.509
0.509
0.540
0.529
circuit
49
0.356
0.346
0.369
0.360
day
136
0.038
0.035
0.054
0.049
detention
22
0.454
0.454
0.476
0.454
dyke
15
0.933
0.933
0.933
0.933
facility
14
0.875
0.875
1
1
fatigue
38
0.236
0.230
0.297
0.282
feeling
48
0.306
0.300
0.346
0.340
grip
38
0.184
0.179
0.216
0.205
hearth
29
0.321
0.310
0.321
0.310
holiday
23
0.818
0.346
0.833
0.384
lady
40
0.375
0.136
0.615
0.363
material
58
0.343
0.338
0.359
0.353
mouth
51
0.094
0.094
0.132
0.132
nation
25
0.269
0.269
0.307
0.307
nature
37
0.263
0.263
0.289
0.289
post
41
0.312
0.306
0.354
0.346
restraint
31
0.200
0.193
0.206
0.193
sense
37
0.260
0.240
0.282
0.260
spade
17
0.823
0.823
0.941
0.941
stress
37
0.228
0.216
0.257
0.243
yew
24
0.480
0.480
0.541
0.520
Total
1294
0.300
0.267
0.336
0.303
Table 15: Precision and Recall Results Using SM to Disambiguate Words, With and With-
out Fixing of Noun Sense
word polysemy (i.e., the number of categories for a word is generally lower than the numberof senses for the word) and the results obtained by this heuristic have better precision thanthose obtained by the whole SM method, which in turn obtain better recall.
As shown in subsection 3.1.7, the domain heuristic can annotate word senses character-
ized by their domains. Thus, these domains will be used as an additional type of featuresfor ME in a context window of ±1, ±2, and ±3 from the target word. In addition, the
three more relevant domains were calculated also for each context and incorporated to thetraining in the form of features.
This experiment was also carried out on the English lexical-sample task data from
Senseval-2, and ME was used to generate two groups of classifiers from the training data.
The first group of classifiers used the corpus without information of domains; the second,
having previously been domain disambiguated with SM, incorporating the domain label ofadjacent nouns, and the three more relevant domains to the context. That is, providing tothe classifier a richer set of features (adding the domain features). However, in this case,we did not perform any feature selection.
The test data was disambiguated by ME twice, with and without SM domain labelling,
using 0lW sbcpdm (see Figure 9) as the common set of features in order to perform thecomparison. The results of the experiment are shown in Table 16.
The table shows that 7 out of 29 nouns obtained worse results when using the domains,
whereas 13 obtained better results. Although, in this case, we only obtained a very smallimprovement in terms of precision (2%)7.
7. This difference proves to be statistically significant when applying the test of the corrected difference of
two proportions (Dietterich, 1998; Snedecor & Cochran, 1989)
321

 
Montoyo, Su´
arez, Rigau, & Palomar
Target words
Without domains With domains Improvement
art
0.667
0.778
0.111
authority
0.600
0.700
0.100
bar
0.625
0.615
-0.010
bum
0.865
0.919
0.054
chair
0.898
0.898
channel
0.567
0.597
0.030
child
0.661
0.695
0.034
church
0.560
0.600
0.040
circuit
0.408
0.388
-0.020
day
0.676
0.669
-0.007
detention
0.909
0.909
dyke
0.800
0.800
facility
0.429
0.500
0.071
fatigue
0.850
0.850
feeling
0.708
0.688
-0.021
grip
0.540
0.620
0.080
hearth
0.759
0.793
0.034
holiday
1.000
0.957
-0.043
lady
0.900
0.900
material
0.534
0.552
0.017
mouth
0.569
0.588
0.020
nation
0.720
0.720
nature
0.459
0.459
post
0.463
0.512
0.049
restraint
0.516
0.452
-0.065
sense
0.676
0.622
-0.054
spade
0.765
0.882
0.118
stress
0.378
0.378
yew
0.792
0.792
All
0.649
0.669
0.020
Table 16: Precision Results Using ME to Disambiguate Words, With and Without Domains
(recall and precision values are equal)
We obtained important conclusions about the relevance of domain information for each
word. In general, the larger improvements appear for those words having well-differentiateddomains (spade, authority). Conversely, the word stress with most senses belonging to theFACTOTUM domain do not improves at all. For example, for spade, art and authority(with an accuracy improvement over 10%) domain data seems to be an important source ofknowledge with information that is not captured by other types of features. For those wordsfor which precision decrease up to 6.5%, domain information is confusing. Three reasonscan be exposed in order to explain this behavior: there is not a clear domain in the examplesor they do not represent correctly the context, domains do not differentiate appropriatelythe senses, or the number of training examples is too low to perform a valid assessment. Across-validation testing, if more examples were available, could be appropriate to performa domain tuning for each word in order to determine which words must use this preprocessand which not.
Nevertheless, the experiment empirically demonstrates that a knowledge-based method,
such as the domain heuristic, can be integrated successfully into a corpus-based system,such as maximum entropy, to obtain a small improvement.
4.3 Combining Results with(in) a Voting System
In previous sections, we have demonstrated that it is possible to integrate two different WSDapproaches. In this section we evaluate the performance when combining a knowledge-basedsystem, such as specification marks, and a corpus-based system, such as maximum entropy,in a simple voting schema.
322

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
In the two previous experiments we attempted to provide more information by pre-
disambiguating the data. Here, we use both methods in parallel and then we combine theirclassifications in a voting system, both for Senseval-2 Spanish and English lexical–sampletasks.
4.3.1 Senseval-2 Spanish lexical–sample task
vME+SM is an enrichment of vME: we added the SM classifier to the combination of thethree ME systems in vME (see Section 3.3). The results on the Spanish lexical–sample taskfrom Senseval-2 are shown in Table 17. Because it only works with nouns, vME+SMimproves accuracy for them only, but obtains the same score as JHU(R) while the overallscore reaches the second place.
ALL
Nouns
0.713 jhu(R)
0.702 jhu(R)
0.684 vME+SM
0.702 vME+SM
0.682 jhu
0.683 MEbfs.pos
0.677 MEbfs.pos 0.681 jhu0.676 vME
0.678 vME
0.670 css244
0.661 MEbfs
0.667 MEbfs
0.652 css244
0.658 MEfix
0.646 MEfix
0.627 umd-sst
0.621 duluth 8
0.617 duluth 8
0.612 duluth Z
0.610 duluth 10
0.611 duluth 10
0.595 duluth Z
0.603 umd-sst
0.595 duluth 7
0.592 duluth 6
0.582 duluth 6
0.590 duluth 7
0.578 duluth X
0.586 duluth X
0.560 duluth 9
0.557 duluth 9
0.548 ua
0.514 duluth Y
0.524 duluth Y
0.464 ua
Table 17: vME+SM in the Spanish lexical–sample task of Senseval-2
These results show that methods like SM and ME can be combined in order to achieve
good disambiguation results. Our results are in line with those of Pedersen (2002), whichalso presents a comparative evaluation between the systems that participated in the Spanishand English lexical-sample tasks of Senseval-2. Their focus is on pair comparisons betweensystems to assess the degree to which they agree, and on measuring the difficulty of the testinstances included in these tasks. If several systems are largely in agreement, then thereis little benefit in combining them since they are redundant and they will simply reinforceeach other. However, if some systems disambiguate instances that others do not, then thesystems are complementary and it may be possible to combine them to take advantage ofthe different strengths of each system to improve overall accuracy.
The results for nouns (only applying SM), shown in Table 18, indicate that SM has
a low level of agreement with all the other methods. However, the measure of optimalcombination is quite high, reaching 89% (1.00–0.11) for the pairing of SM and JHU. In
323

 
Montoyo, Su´
arez, Rigau, & Palomar
fact, all seven of the other methods achieved their highest optimal combination value whenpaired with the SM method.
System pair for nouns
Both OKa One OK b Zero OK c Kappa d
SM and JHU
0.29
0.32
0.11
0.06
SM and Duluth7
0.27
0.34
0.12
0.03
SM and DuluthY
0.25
0.35
0.12
0.01
SM and Duluth8
0.28
0.32
0.13
0.08
SM and Cs224
0.28
0.32
0.13
0.09
SM and Umcp
0.26
0.33
0.14
0.06
SM and Duluth9
0.26
0.31
0.16
0.14
Table 18: Optimal combination between the systems that participated in the Spanish
lexical–sample tasks of Senseval-2
a. Percentage of instances where both systems answers were correct.
b. Percentage of instances where only one answer is correct.c. Percentage of instances where none of both answers is correct.
d. The kappa statistic (Cohen, 1960) is a measure of agreement between multiple systems (or judges) that
is scaled by the agreement that would be expected just by chance. A value of 1.00 suggests completeagreement, while 0.00 indicates pure chance agreement.
This combination of circumstances suggests that SM, being a knowledge-based method,
is fundamentally different from the others (i.e., corpus-based) methods, and is able todisambiguate a certain set of instances where the other methods fail. In fact, SM is differentin that it is the only method that uses the structure of WordNet.
4.3.2 Senseval-2 English lexical–sample task
The same experiment was done on Senseval-2 English lexical–sample task data and theresults are shown in Table 19. The details of how the different systems were built can beconsulted in Section 3.2
Again, we can see in Table 19 that BFS per POS is better than per word, mainly because
the same reasons explained in Section 3.3.
Nevertheless, the improvement on nouns by using the vME+SM system is not as high
as for the Spanish data. The differences between both corpora have a significant relevanceabout the precision values that can be obtained. For example, the English data includesmulti-words and the sense inventory is extracted from WordNet, while the Spanish datais smaller and a dictionary was built for the task specifically, having a smaller polysemydegree.
The results of vME+SM are comparable to the systems presented at Senseval-2
where the best system (Johns Hopkins University) reported 64.2% precision (68.2%, 58.5%and 73.9% for nouns, verbs and adjectives, respectively).
Comparing these results with those obtained in section 4.2, we also see that using a
voting system with the best feature selection for ME and Specification Marks vME+SM,and using a non–optimized ME with the relevant domain heuristic, we obtain very similarperformance. That is, it seems that we obtain comparable performance combining different
324

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
All
Nouns Verbs Adjectives
MEfix
0.601 0.656 0.519
0.696
MEbfs
0.606 0.658 0.519
0.696
MEbfs.pos 0.609 0.664 0.519
0.696
vME+SM 0.619 0.667 0.535
0.707
MEfix: 0mcbWsdrvK3 for all words
MEbfs: each word with its
best feature selection
MEbfs.pos: 0Wsrdm for nouns,
0sbcprdmK10 for verbs,and 0mcbWsdrvK3 for adjectives
vME+SM: majority voting between MEfix,
MEbfs.pos, MEbfs, and Specification Marks
Table 19: Precision Results Using Best Feature Selection for ME and Specification Marks on
Senseval-2 English lexical–sample task data
classifiers resulting from a feature selection process or using a richer set of features (addingthe domain features) with much less computational overhead.
This analysis of the results from the Senseval-2 English and Spanish lexical–sample
tasks demonstrates that knowledge-based and corpus-based WSD systems can cooperateand can be combined to obtain improved WSD systems. The results empirically demon-strate that the combination of both approaches outperforms each of them individually,demonstrating that both approaches could be considered complementary.
5. Conclusions
The main hypothesis of this work is that WSD requires different kinds of knowledge sources(linguistic information, statistical information, structural information, etc.) and techniques.The aim of this paper was to explore some methods of collaboration between complementaryknowledge-based and corpus-based WSD methods. Two complementary methods have beenpresented: specification marks (SM) and maximum entropy (ME). Individually, both havebenefits and drawbacks. We have shown that both methods can collaborate to obtain betterresults on WSD.
In order to demonstrate our hypothesis, three different schemes for combining both
approaches have been presented. We have presented different mechanisms of combininginformation sources around knowledge-based and corpus-based WSD methods. We havealso shown that the combination of both approaches outperforms each of the methods indi-vidually, demonstrating that both approaches could be considered complementary. Finally,we have shown that a knowledge-based method can help a corpus-based method to betterperform the disambiguation process, and vice versa.
In order to help the specification marks method, ME disambiguates some nouns in the
context of the target word. ME selects these nouns by means of a previous analysis oftraining data in order to identify which ones seem to be highly accurately disambiguated.
325

 
Montoyo, Su´
arez, Rigau, & Palomar
This preprocess fixes some nouns reducing the search space of the knowledge-based method.In turn, ME is helped by SM by providing domain information of nouns in the contexts.This information is incorporated into the learning process in the form of features.
By comparing the accuracy of both methods, with and without the contribution of the
other, it was demonstrated that such combining schemes of WSD methods are possible andsuccessful.
Finally, we presented a voting system for nouns that included four classifiers, three of
them based on ME, and one of them based on SM. This cooperation scheme obtained thebest score for nouns when compared with the systems submitted to the Senseval-2 Spanishlexical–sample task and comparable results to those submitted to the Senseval-2 Englishlexical–sample task.
We are presently studying possible improvements in the collaboration between these
methods, both by extending the information that the two methods provide to each otherand by taking advantage of the merits of each one.
Acknowledgments
The authors wish to thank the anonymous reviewers of the Journal of Artificial Intelli-gence Research and COLING 2002, the 19th International Conference on ComputationalLinguistics, for helpful comments on earlier drafts of the paper. An earlier paper (Su´arez &Palomar, 2002b) about the corpus-based method (subsection 3.2) was presented at COLING2002.
This research has been partially funded by the Spanish Government under project CI-
CyT number TIC2000-0664-C02-02 and PROFIT number FIT-340100-2004-14 and the Va-lencia Government under project number GV04B-276 and the EU funded project MEAN-ING (IST-2001-34460).
References
Agirre, E., & Martinez, D. (2001a). Decision lists for english and basque. In Proceedings of
the SENSEVAL-2 Workshop. In conjunction with ACL’2001/EACL’2001 Toulouse,France.
Agirre, E., & Martinez, D. (2001b). Knowledge sources for word sense disambiguation. In
Proceedings of International Conference on Text, Speech and Dialogue (TSD’2001)Selezna Ruda, Czech Republic.
Agirre, E., & Rigau, G. (1996). Word Sense Disambiguation using Conceptual Density. In
Proceedings of the 16th International Conference on Computational Linguistic (COL-ING´96 Copenhagen, Denmark.
Berger, A. L., Pietra, S. A. D., & Pietra, V. J. D. (1996). A maximum entropy approach
to natural language processing. Computational Linguistics, 22 (1), 39–71.
326

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
Bruce, R., & Wiebe, J. (1994). Word sense disambiguation using decomposable models. In
Proceedings of the 32nd Annual Meeting of the Association for Computational Lin-guistics (ACL’1994), pp. 139–145 Las Cruces, US.
Cabezas, C., Resnik, P., & Stevens, J. (2001). Supervised Sense Tagging using Support
Vector Machines. In Proceedings of the Second International Workshop on EvaluatingWord Sense Disambiguation Systems (SENSEVAL-2) Toulouse, France.
Cardie, C., & Mooney, R. J. (1999). Guest editors’ introduction: Machine learning and
natural language. Machine Learning, 34 (1-3), 5–9.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educ. Psychol. Meas., 20,
37–46.
Cowie, J., Guthrie, J., & Guthrie, L. (1992). Lexical disambiguation using simulated anneal-
ing. In Proceedings of the 14th International Conference on Computational Linguistic,COLING´92, pp. 359–365 Nantes, France.
Cuadros, M., Atserias, J., Castillo, M., & Rigau, G. (2004). Automatic acquisition of sense
examples using exretriever. In IBERAMIA Workshop on Lexical Resources and TheWeb for Word Sense Disambiguation. Puebla, Mexico.
Dang, H. T., yi Chia, C., Palmer, M., & Chiou, F.-D. (2002). Simple features for chinese
word sense disambiguation. In Chen, H.-H., & Lin, C.-Y. (Eds.), Proceedings of the19th International Conference on Computational Linguistics (COLING´2002).
Dietterich, T. G. (1998). Approximate statistical test for comparing supervised classification
learning algorithms. Neural Computation, 10 (7), 1895–1923.
Escudero, G., M`arquez, L., & Rigau, G. (2000). Boosting applied to word sense disam-
biguation. In Proceedings of the 12th Conference on Machine Learning ECML2000Barcelona, Spain.
Fellbaum, C. (Ed.). (1998). WordNet. An Electronic Lexical Database. The MIT Press.
Fern´
andez-Amor´
os, D., Gonzalo, J., & Verdejo, F. (2001). The Role of Conceptual Rela-
tions in Word Sense Disambiguation. In Proceedings 6th International Conference onApplication of Natural Language to Information Systems (NLDB´2001)., pp. 87–98Madrid, Spain.
Hoste, V., Daelemans, W., Hendrickx, I., & van den Bosch, A. (2002). Evaluating the results
of a memory-based word-expert approach to unrestricted word sense disambiguation.In Proceedings of the ACL’2002 Workshop on Word Sense Disambiguation: RecentSuccesses and Future Directions, pp. 95–101 PA, USA.
Ide, N., & V´eronis, J. (1998). Introduction to the Special Issue on Word Sense Disambigua-
tion: The State of the Art. Computational Linguistics, 24 (1), 1–40.
Leackock, C., Chodorow, M., & Miller, G. (1998). Using corpus statistics and wordnet
relations for sense identification. Computational Linguistics. Special Issue on WSD,24 (1).
327

 
Montoyo, Su´
arez, Rigau, & Palomar
Lesk, M. (1986). Automated sense disambiguation using machine-readable dictionaries:
How to tell a pine cone from an ice cream cone. In Proceedings of the 1986 SIGDOCConference, Association for Computing Machinery, pp. 24–26 Toronto, Canada.
Magnini, B., & Strapparava, C. (2000). Experiments in Word Domain Disambiguation for
Parallel Texts. In Proceedings of the ACL Workshop on Word Senses and Multilin-guality Hong Kong, China.
Magnini, B., Strapparava, C., Pezzulo, G., & Gliozzo, A. (2002). The Role of Domain
Information in Word Sense Disambiguation. Natural Language Engineering, 8 (4),359–373.
Manning, C., & Sch¨
utze, H. (Eds.). (1999). Foundations of Statistical Natural Language
Processing. The MIT Press.
Manning, C. D., & Sch¨
utze, H. (1999). Foundations of Statistical Natural Language Pro-
cessing. The MIT Press, Cambridge, Massachusetts.
McRoy, S. W. (1992). Using multiple knowledge sources for word sense discrimination.
Computational Linguistics, 18 (1), 1–30.
Mihalcea, R. (2002). Instance based learning with automatic feature selection applied to
word sense disambiguation. In Chen, H.-H., & Lin, C.-Y. (Eds.), Proceedings of the19th International Conference on Computational Linguistics (COLING´2002).
Mihalcea, R., & Moldovan, D. (1999). A Method for word sense disambiguation of un-
restricted text. In Proceedings of the 37th Annual Meeting of the Association forComputational Linguistic, ACL’99, pp. 152–158 Maryland, Usa.
Miller, G. A. (1995). Wordnet: a lexical database for english. Communications of the ACM,
38 (11), 39–41.
Miller, G. A., Leacock, C., Tengi, R., & Bunker, T. (1993). A Semantic Concordance.
In Proceedings of ARPA Workshop on Human Language Technology, pp. 303–308Plainsboro, New Jersey.
Mitchell, T. M. (Ed.). (1997). Machine Learning. McGraw Hill.
Montoyo, A., & Palomar, M. (2001). Specification Marks for Word Sense Disambiguation:
New Development. In Gelbukh, A. F. (Ed.), CICLing, Vol. 2004 of Lecture Notes inComputer Science, pp. 182–191. Springer.
Montoyo, A., & Su´arez, A. (2001). The University of Alicante word sense disambiguation
system.. In Preiss, & Yarowsky (Preiss & Yarowsky, 2001), pp. 131–134.
Montoyo, A., Palomar, M., & Rigau, G. (2001). WordNet Enrichment with Classification
Systems.. In Proceedings of WordNet and Other Lexical Resources: Applications,Extensions and Customisations Workshop. (NAACL-01) The Second Meeting of theNorth American Chapter of the Association for Computational Linguistics, pp. 101–106 Carnegie Mellon University. Pittsburgh, PA, USA.
328

 
Combining Knowledge- and Corpus-based Word-Sense-Disambiguation Methods
Montoyo, A., Vazquez, S., & Rigau, G. (2003). M´etodo de desambiguaci´on l´exica basada
en el recurso l´exico Dominios Relevantes. Procesamiento del Lenguaje Natural, 31,141–148.
Ng, H. (1997). Exemplar-Base Word Sense Disambiguation: Some Recent Improvements.
In Proceedings of the 2nd Conference on Empirical Methods in Natural LanguageProcessing, EMNLP.
Ng, H. T., & Lee, H. B. (1996). Integrating multiple knowledge sources to disambiguate word
senses: An exemplar-based approach. In Joshi, A., & Palmer, M. (Eds.), Proceedingsof the 34th Annual Meeting of the Association for Computational Linguistics SanFrancisco. Morgan Kaufmann Publishers.
Pedersen, T. (2002). Assessing System Agreement and Instance Difficulty in the Lexical
Sample Tasks of Senseval-2. In Proceedings of the Workshop on Word Sense Disam-biguation: Recent Successes and Future Directions, ACL´2002 Philadelphia, USA.
Preiss, J., & Yarowsky, D. (Eds.). (2001). Proceedings of SENSEVAL-2, Toulouse, France.
ACL-SIGLEX.
Ratnaparkhi, A. (1998). Maximum Entropy Models for Natural Language Ambiguity Reso-
lution. Ph.D. thesis, University of Pennsylvania.
Rigau, G., Agirre, E., & Atserias, J. (1997). Combining unsupervised lexical knowledge
methods for word sense disambiguation. In Proceedings of joint 35th Annual Meetingof the Association for Computational Linguistics and 8th Conference of the EuropeanChapter of the Association for Computational Linguistics ACL/EACL’97 Madrid,Spain.
Rigau, G., Taul´e, M., Fern´
andez, A., & Gonzalo, J. (2001). Framework and results for the
spanish senseval.. In Preiss, & Yarowsky (Preiss & Yarowsky, 2001), pp. 41–44.
Snedecor, G. W., & Cochran, W. G. (1989). Statistical Methods (8 edition). Iowa State
University Press, Ames, IA.
Snyder, B., & Palmer, M. (2004).
The english all-words task.
In Proceedings of the
3rd ACL workshop on the Evaluation of Systems for the Semantic Analysis of Text(SENSEVAL-3). Barcelona, Spain.
Su´
arez, A., & Palomar, M. (2002a). Feature selection analysis for maximum entropy-based
wsd. In Gelbukh, A. F. (Ed.), CICLing, Vol. 2276 of Lecture Notes in ComputerScience, pp. 146–155. Springer.
Su´arez, A., & Palomar, M. (2002b). A maximum entropy-based word sense disambiguation
system. In Chen, H.-H., & Lin, C.-Y. (Eds.), Proceedings of the 19th InternationalConference on Computational Linguistics (COLING´2002), pp. 960–966.
Sussna, M. (1993). Word sense disamiguation for free-text indexing using a massive semantic
network. . In Proceedings of the Second International Conference on Information andKnowledge Base Management, CIKM´93, pp. 67–74 Arlington, VA.
329

 
Montoyo, Su´
arez, Rigau, & Palomar
Tapanainen, P., & J¨arvinen, T. (1997). A non-projective dependency parser. In Proceedings
of the Fifth Conference on Applied Natural Language Processing, pp. 64–71.
Towell, G. G., & Voorhees, E. M. (1998). Disambiguating highly ambiguous words. Com-
putational Linguistics, 24 (1), 125–145.
Veenstra, J., den Bosch, A. V., Buchholz, S., Daelemans, W., & Zavrel, J. (2000). Memory-
based word sense disambiguation. Computers and the Humanities, Special Issue onSENSEVAL, 34 (1–2), 171–177.
Wilks, Y., Fass, D., Guo, C.-M., McDonald, J., Plate, T., & Slator, B. (1993). Provid-
ing machine tractable dictionary tools. In Pustejovsky, J. (Ed.), Semantics and thelexicon, pp. 341–401. Kluwer Academic Publishers.
Yarowsky, D. (1994). Decision lists for lexical ambiguity resolution: Application to accent
restoration in spanish and french.. In In Proceedings of the 32nd Annual Meeting ofthe Association for Computational Linguistics (ACL’1994) Las Cruces, NM,.
330