Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of koji@r.email.ne.jp designates
 202.224.39.198 as permitted sender)
Message-ID: <49412D0A.60500@r.email.ne.jp>
Date: Fri, 12 Dec 2008 00:08:58 +0900
From: Koji Sekiguchi <koji@r.email.ne.jp>
User-Agent: Thunderbird 2.0.0.18 (Macintosh/20081105)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: problem on NGram+Highlighter
Content-Type: text/plain; charset=ISO-2022-JP
Content-Transfer-Encoding: 7bit

Hello,

I have a problem when using n-gram and highlighter.
I thought it had been solved on the ticket:

http://issues.apache.org/jira/browse/LUCENE-627

Actually, I found this problem when I was using CJKTokenizer
on Solr, though, here is lucene program to reproduce it
using NGramTokenizer(min=2,max=2) instead of CJKTokenizer:

public class TestNGramHighlighter {
public static void main(String[] args) throws Exception {
Analyzer analyzer = new NGramAnalyzer();
final String TEXT = "ABCDEFGHIJKLMNABCDEFGHIJKLMN";
final String QUERY = "GHI";
QueryParser parser = new QueryParser("f",analyzer);
Query query = parser.parse(QUERY);
QueryScorer scorer = new QueryScorer(query,"f");
Highlighter h = new Highlighter( scorer );
System.out.println( h.getBestFragment(analyzer, "f", TEXT) );
}
static class NGramAnalyzer extends Analyzer {
public TokenStream tokenStream(String field, Reader input) {
return new NGramTokenizer(input,2,2);
}
}
}

expected output is:
ABCDEF<B>GHI</B>JKLMNABCDEF<B>GHI</B>JKLMN

but the actual output is:
ABCDEF<B>GHIJKLMNABCDEFGHI</B>JKLMN

Am I missing something?

Thank you,

Koji


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org