lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doron Cohen <DOR...@il.ibm.com>
Subject Re: Stop words (how to create ideal set of stop words?)
Date Thu, 10 May 2007 23:41:08 GMT
See also  en.wikipedia.org/wiki/Stop_words  and
www.ranks.nl/tools/stopwords.html

karl wettin <karl.wettin@gmail.com> wrote on 10/05/2007 13:57:33:

>
> 10 maj 2007 kl. 20.39 skrev Lukas Vlcek:
>
> > Can anybody point me to some references how to create an ideal set
> > of stop
> > words? I konw that this is more like a theoretical question but how do
> > Luceners determine which words shuold be excluded when creating
> > Analyzers
> > for a new languages?
>
> The idea with stop words is to keep the index as small as possible
> without major loss of features, thus they ought to be frequently
> occuring words with little or no semantic meaning. What these words
> are really depends on language, corpus, et c.
>
> > And which technique was used for validation of stop
> > word lists in current Analyzers?
>
> My guess is that they are manually choosen from a corpus term
> frequency vector.
>
> > More specificaly I am interested in situations when there is a need
> > to build
> > a search engine around specific corpus (for example when we need to
> > search
> > set of articles related to programming languages only). Given a
> > specific
> > corpus is there any recommended technique of stop words derivation?
>
> If you have no knowledge of the language for wich you wish to produce
> stop words, then it will be fairly hard to know what to consider a
> stop word. You might be able to consider it as a text classification
> problem. Feature/attribute selection for classifiers is a well
> researched subject. Weka, Yale, R, et c are all tools that might help
> you. But I honestly think no matter how you turn and twist the data,
> manually choosing the stop words is the way to go.
>
>
> --
> karl


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message