Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <19210665.post@talk.nabble.com>
Date: Thu, 28 Aug 2008 15:50:46 -0700 (PDT)
From: gaz77 <gareth.cole@bit10.net>
To: java-user@lucene.apache.org
Subject: Re: Confused with NGRAM results
In-Reply-To: <19202310.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <19202310.post@talk.nabble.com>


Thanks for the pointer.

I've gone into this in some depth, using the AnalyzerUtils class from the
lucene in action book.

It seems that the NGramTokenFilter is only processing part of the string
that goes in. It stops tokenising the words part way through. That's why the
documents weren't found in results.

I've had a look at the source code, and I think it's because the next()
function returns null when it hits a token smaller than the min ngram size.
For example, if I set the minimum to 3, then a 2-character token will cause
it to return null.

I'm not sure if this is by design or a bug. either way, at least I know
what's causing it now.

Cheers


-- 
View this message in context: http://www.nabble.com/Confused-with-NGRAM-results-tp19202310p19210665.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org