Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.vatkov@gmail.com
 designates 209.85.219.225 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=WG+vJ9VASS1ubFXKbGzsOOXl2xif5Hl2vBm8HWuf/bXC/pUzFJ+jeoIft9JqNh6HoV
         YsrlatntM+nCO834SUBtROUhHlm7dGkZLPZ46VNI1jgh7q6uA8nwlNbBlCeT2ze5MFMM
         BPtJpWqlEjmpnGxhnAAMepuNkNGc1JfQzP1bg=
MIME-Version: 1.0
In-Reply-To: <3E300DFB-FED0-4752-98A8-33B0EC3D21B2@apache.org>
References: <e9c993f11001021720r2402c513m39f02a715742e405@mail.gmail.com>
	 <87c998321001021751g275d5c0axa366e84116535849@mail.gmail.com>
	 <e9c993f11001021831s269ecc4id3f51026bbc3031e@mail.gmail.com>
	 <c7d45fc71001022356u437c2788nbdb7a8680e6b091b@mail.gmail.com>
	 <e9c993f11001030457t29423bc8s140e8de9e0a8edd9@mail.gmail.com>
	 <0280C025-D0F1-4AC1-9330-3A38C7573796@apache.org>
	 <e9c993f11001030558g4c1bfa80qe259c0b0f4c89bf0@mail.gmail.com>
	 <3E300DFB-FED0-4752-98A8-33B0EC3D21B2@apache.org>
Date: Sun, 3 Jan 2010 16:08:20 +0200
Message-ID: <e9c993f11001030608p9888417p77cb309c6dd26f24@mail.gmail.com>
Subject: Re: Stopwords not working as expected
From: Bogdan Vatkov <bogdan.vatkov@gmail.com>
To: mahout-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00504502cd6515069c047c4323ae

--00504502cd6515069c047c4323ae
Content-Type: text/plain; charset=ISO-8859-1

Yesterday I had issues with mapping cluster results to dictionary entries -
it happened that I was using different dictionary - therefore the result
clusters shown really strange results.
But once I fixed all the commands, input/output files, etc. I got very good
result from clusterization POV (I mean clusters are quite correct having in
mind the input documents) but unfortunately the clusters contained mostly
words which I would like to stop - and which words I placed in the
stopwords.txt in Solr (re-indexed, restarted Solr, etc.).

Where do you suggest I debug the vector creation? Seems Solr respects the
stopwords but not the vector creation (then clustering).

On Sun, Jan 3, 2010 at 4:02 PM, Grant Ingersoll <gsingers@apache.org> wrote:

>
> On Jan 3, 2010, at 8:58 AM, Bogdan Vatkov wrote:
>
> > I have stopwords.txt file with 1200+ words, i did not understand this
> with
> > the stemming - you mean my stopwords are somehow ignored due to some
> > stemming or ?
>
> No, stopword removal happens before stemming so it is possible that a word
> that was not stopped was then stemmed to a stopword.
>
> I thought you said yesterday you got it straightened out.
>
> >
> > On Sun, Jan 3, 2010 at 3:53 PM, Grant Ingersoll <gsingers@apache.org>
> wrote:
> >
> >> Are you sure you have stopwords and it is not the result of stemming
> some
> >> other word?
> >>
> >> On Jan 3, 2010, at 7:57 AM, Bogdan Vatkov wrote:
> >>
> >>> my Solr config is like the default one:
> >>>
> >>>  <field name="msg_body" type="text" termVectors="true" indexed="true"
> >>> stored="true"/>
> >>>
> >>>  <fieldType name="text" class="solr.TextField"
> >> positionIncrementGap="100">
> >>>     <analyzer type="index">
> >>>       <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >>>       <filter class="solr.StopFilterFactory"
> >>>               ignoreCase="true"
> >>>               words="stopwords.txt"
> >>>               enablePositionIncrements="true"
> >>>               />
> >>>       <filter class="solr.WordDelimiterFilterFactory"
> >>> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> >>> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
> >>>       <filter class="solr.LowerCaseFilterFactory"/>
> >>>       <filter class="solr.SnowballPorterFilterFactory"
> >> language="English"
> >>> protected="protwords.txt"/>
> >>>     </analyzer>
> >>>     <analyzer type="query">
> >>>       <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> >>>       <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> >>> ignoreCase="true" expand="true"/>
> >>>       <filter class="solr.StopFilterFactory"
> >>>               ignoreCase="true"
> >>>               words="stopwords.txt"
> >>>               enablePositionIncrements="true"
> >>>               />
> >>>       <filter class="solr.WordDelimiterFilterFactory"
> >>> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> >>> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
> >>>       <filter class="solr.LowerCaseFilterFactory"/>
> >>>       <filter class="solr.SnowballPorterFilterFactory"
> >> language="English"
> >>> protected="protwords.txt"/>
> >>>     </analyzer>
> >>>   </fieldType>
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Bogdan
>
>


-- 
Best regards,
Bogdan

--00504502cd6515069c047c4323ae--