lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mauro Asprea (Commented) (JIRA)" <>
Subject [jira] [Commented] (SOLR-1279) ApostropheTokenizer
Date Thu, 16 Feb 2012 09:02:59 GMT


Mauro Asprea commented on SOLR-1279:

I confirm this is working using the WordDelimiterFilterFactory like Robert said:

<filter class="solr.WordDelimiterFilterFactory"

The using Solr Admin Analysis page I get the following:
Value: McDonal's

||Indexed Term|

One thing: You have to be sure that no previous filters remove the trailing "'s". In my case
I had the StandardFilterFactory which does remove tailing apostrophes.
> ApostropheTokenizer
> -------------------
>                 Key: SOLR-1279
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>          Components: Schema and Analysis
>            Reporter: Sergey Borisov
>            Priority: Minor
>             Fix For: 3.6, 4.0
>         Attachments:
> ApostropheTokenizer creates extra tokens during the analysis stage for the fields containing
apostrophes. The reason for adding this is to ensure that documents that differ only by apostrophe
have the same relevancy score. 
> For example, if the document contains string "McDonald's", it will be tokenized as "McDonald's
McDonalds". This way when the search is performed against "McDonald's" or "McDonalds" will
produce similar score.
> This code handles up to two apostrophes in a token.
> To use this tokenizer add the following line in schema.xml
> <analyzer type="index">
>       <filter class="org.apache.lucene.analysis.ApostropheTokenFactory"/>
> ...
> </analyzer>

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message