lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Sekiguchi <k...@r.email.ne.jp>
Subject problem on NGram+Highlighter
Date Thu, 11 Dec 2008 15:08:58 GMT
Hello,

I have a problem when using n-gram and highlighter.
I thought it had been solved on the ticket:

http://issues.apache.org/jira/browse/LUCENE-627

Actually, I found this problem when I was using CJKTokenizer
on Solr, though, here is lucene program to reproduce it
using NGramTokenizer(min=2,max=2) instead of CJKTokenizer:

public class TestNGramHighlighter {
public static void main(String[] args) throws Exception {
Analyzer analyzer = new NGramAnalyzer();
final String TEXT = "ABCDEFGHIJKLMNABCDEFGHIJKLMN";
final String QUERY = "GHI";
QueryParser parser = new QueryParser("f",analyzer);
Query query = parser.parse(QUERY);
QueryScorer scorer = new QueryScorer(query,"f");
Highlighter h = new Highlighter( scorer );
System.out.println( h.getBestFragment(analyzer, "f", TEXT) );
}
static class NGramAnalyzer extends Analyzer {
public TokenStream tokenStream(String field, Reader input) {
return new NGramTokenizer(input,2,2);
}
}
}

expected output is:
ABCDEF<B>GHI</B>JKLMNABCDEF<B>GHI</B>JKLMN

but the actual output is:
ABCDEF<B>GHIJKLMNABCDEFGHI</B>JKLMN

Am I missing something?

Thank you,

Koji


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message