lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From benafia salem <salem.ben.a...@gmail.com>
Subject Weighed Synonyms - Lucene 7.1
Date Thu, 16 Nov 2017 16:35:48 GMT
Hello,

I'm trying to implement a weighed synonyms analyzer using
SynonymGraphFilter and a custom payload analyzer.

My synonyms file looks like:
obama,president^0.9,barry^0.5
speaks,greets^0.8,congratulates^0.9,informs
media,press,journal,tv^0.8

I want user searching for "obama greets press" to find documents containing
"president speaks to media".

I want to keep the original "phrase" at the heighest score, and synonyms to
score less.

The problem is that my Synonyms Filter Could not undestand weights although
I passed the matching analyzer in the SynonymGraphFilter constructor.
Here is my createComponents method:

protected TokenStreamComponents createComponents(String fieldName) {
final KeywordTokenizer src = new KeywordTokenizer();
TokenStream tok = new LowerCaseFilter(src);
try {
tok = new SynonymGraphFilter(tok,
loadSynonyms(new FilesystemResourceLoader(Paths.get("/wsliat/lucene-res/")),
SolrSynonymParser.class.getName(), false, new PayloadAwareAnalyzer()),
true);
} catch (Exception e) {
e.printStackTrace();
}
tok = new DelimitedPayloadTokenFilter(tok, '^', this.weightEncoder);
return new TokenStreamComponents(src, tok) {
@Override
protected void setReader(final Reader reader) {
super.setReader(reader);
}
};
}
My PayloadAwareAnalyzer looks like this:

public class PayloadAwareAnalyzer extends Analyzer {

private PayloadEncoder weightEncoder;

public PayloadAwareAnalyzer(PayloadEncoder weightEncoder) {
this.weightEncoder = weightEncoder;
}
public  PayloadAwareAnalyzer() {
super();
this.weightEncoder =  new FloatEncoder();
}

@Override
protected TokenStreamComponents createComponents(String fieldName) {
final KeywordTokenizer src = new KeywordTokenizer();
TokenStream tok = new DelimitedPayloadTokenFilter(src, '^',
this.weightEncoder);
return new TokenStreamComponents(src, tok) {
@Override
protected void setReader(final Reader reader) {
super.setReader(reader);
}
};
}

}

I also added the according custom similarity for the searcher.

Any help would be much appreciated.
Thanks.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message