Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
Mime-Version: 1.0 (Apple Message framework v752.2)
In-Reply-To: 
 <OF2117E6CB.1D30BAE5-ON882572D7.0081B55D-882572D7.00821BD0@il.ibm.com>
References: 
 <OF2117E6CB.1D30BAE5-ON882572D7.0081B55D-882572D7.00821BD0@il.ibm.com>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <894CA52B-4E42-4524-B7A7-0AA4C23E2095@apache.org>
Content-Transfer-Encoding: 7bit
From: Grant Ingersoll <gsingers@apache.org>
Subject: Re: Stop words (how to create ideal set of stop words?)
Date: Thu, 10 May 2007 20:34:33 -0400
To: java-user@lucene.apache.org

Also, from the empirical side, have a look at Luke (after indexing w/ 
o any stopwords, or just the standard ones) and see what the most  
common terms are and see if they are meaningful or not in the context  
of your application.

-Grant


On May 10, 2007, at 7:41 PM, Doron Cohen wrote:

> See also  en.wikipedia.org/wiki/Stop_words  and
> www.ranks.nl/tools/stopwords.html
>
> karl wettin <karl.wettin@gmail.com> wrote on 10/05/2007 13:57:33:
>
>>
>> 10 maj 2007 kl. 20.39 skrev Lukas Vlcek:
>>
>>> Can anybody point me to some references how to create an ideal set
>>> of stop
>>> words? I konw that this is more like a theoretical question but  
>>> how do
>>> Luceners determine which words shuold be excluded when creating
>>> Analyzers
>>> for a new languages?
>>
>> The idea with stop words is to keep the index as small as possible
>> without major loss of features, thus they ought to be frequently
>> occuring words with little or no semantic meaning. What these words
>> are really depends on language, corpus, et c.
>>
>>> And which technique was used for validation of stop
>>> word lists in current Analyzers?
>>
>> My guess is that they are manually choosen from a corpus term
>> frequency vector.
>>
>>> More specificaly I am interested in situations when there is a need
>>> to build
>>> a search engine around specific corpus (for example when we need to
>>> search
>>> set of articles related to programming languages only). Given a
>>> specific
>>> corpus is there any recommended technique of stop words derivation?
>>
>> If you have no knowledge of the language for wich you wish to produce
>> stop words, then it will be fairly hard to know what to consider a
>> stop word. You might be able to consider it as a text classification
>> problem. Feature/attribute selection for classifiers is a well
>> researched subject. Weka, Yale, R, et c are all tools that might help
>> you. But I honestly think no matter how you turn and twist the data,
>> manually choosing the stop words is the way to go.
>>
>>
>> --
>> karl
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org/tech/lucene.asp

Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/ 
LuceneFAQ


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org