Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAESefzyo=h8mQ=v2yrkcxA4CNkh-HipOWqi2578LCs4vBB1ReA@mail.gmail.com>
References: 
 <CAESefzyo=h8mQ=v2yrkcxA4CNkh-HipOWqi2578LCs4vBB1ReA@mail.gmail.com>
Date: Tue, 21 Jul 2015 10:10:52 +0100
Message-ID: 
 <CAB-fSbx3FvQJ3=yYNNP+XU0rvNwDnQJF1TH5gMG9kA1df2XhZg@mail.gmail.com>
Subject: Re: Analyzer for supporting hyphenated words
From: Alessandro Benedetti <benedetti.alex85@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=f46d0435c012f1f0fc051b5f074c

--f46d0435c012f1f0fc051b5f074c
Content-Type: text/plain; charset=UTF-8

Hi Diego,
let me try to help :

I find this a little bit confused :

"For our customer it is important to find the word
- *wi-fi* by wi, *fi*, wifi, wi-fi
- jean-pierre by jean, pierre, jean-pierre, jean-*"

But :
"
The (exact) query "*FD-A320-REC-SIM-1*" returns
FD-A320-REC-SIM-1
MIA-*FD-A320-REC-SIM-1*
SIN-FD-A320-REC-SIM-1

for our customer this is wrong because this exact phrase match
query should only return the single entry FD-A320-REC-SIM-1
"

If you noticed the suffix "fi" in the first example can be compared to the
suffix "FD-A320-REC-SIM-1" in the second.
To qualify your requirement :

Do you want the user to be able to surround the query with "" to run the
phrase query with a NOT tokenized phrase ?
Because by default , a phrase query is tokenized like the others, but term
positions affect the matching !
In the case I identified your requirement, we can have a think to a
solution!


Cheers


2015-07-17 9:41 GMT+01:00 Diego Socaceti <socaceti@gmail.com>:

> Hi all,
>
> i'm new to lucene and tried to write my own analyzer to support
> hyphenated words like wi-fi, jean-pierre, etc.
> For our customer it is important to find the word
> - wi-fi by wi, fi, wifi, wi-fi
> - jean-pierre by jean, pierre, jean-pierre, jean-*
>
>
>
>
> The analyzer:
> public class SupportHyphenatedWordsAnalyzer extends Analyzer {
>
>   protected NormalizeCharMap charConvertMap;
>
>   public MinLuceneAnalyzer() {
>     initCharConvertMap();
>   }
>
>   protected void initCharConvertMap() {
>     NormalizeCharMap.Builder builder = new NormalizeCharMap.Builder();
>     builder.add("\"", "");
>     charConvertMap = builder.build();
>   }
>
>   @Override
>   protected TokenStreamComponents createComponents(final String fieldName)
> {
>
>     final Tokenizer src = new WhitespaceTokenizer();
>
>     TokenStream tok = new WordDelimiterFilter(src,
>         WordDelimiterFilter.PRESERVE_ORIGINAL
>             | WordDelimiterFilter.GENERATE_WORD_PARTS
>             | WordDelimiterFilter.GENERATE_NUMBER_PARTS
>             | WordDelimiterFilter.CATENATE_WORDS,
>         null);
>     tok = new LowerCaseFilter(tok);
>     tok = new LengthFilter(tok, 1, 255);
>     tok = new StopFilter(tok, StopAnalyzer.ENGLISH_STOP_WORDS_SET);
>
>     return new TokenStreamComponents(src, tok);
>   }
>
>   @Override
>   protected Reader initReader(String fieldName, Reader reader) {
>     return new MappingCharFilter(charConvertMap, reader);
>   }
> }
>
>
>
>
>
> The analyzer seems to work except for exact phrase match queries.
>
> e.g. the following words are indexed
>
> FD-A320-REC-SIM-1
> FD-A320-REC-SIM-10
> FD-A320-REC-SIM-11
> MIA-FD-A320-REC-SIM-1
> SIN-FD-A320-REC-SIM-1
>
>
> The (exact) query "FD-A320-REC-SIM-1" returns
> FD-A320-REC-SIM-1
> MIA-FD-A320-REC-SIM-1
> SIN-FD-A320-REC-SIM-1
>
> for our customer this is wrong because this exact phrase match
> query should only return the single entry FD-A320-REC-SIM-1
>
> Do you have any ideas or tips, how we have to change our current
> analyzer to support this requirement???
>
>
> Thanks and Kind regards
> Diego
>


-- 
--------------------------

Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

--f46d0435c012f1f0fc051b5f074c--