lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aigner, Thomas" <>
Subject RE: Indexing puncutation
Date Tue, 28 Jun 2005 18:55:52 GMT
Thanks for the info Chris.


I'd thought I'd provide some more infomation.  One problem is the
descriptions are not easily formatted. In other words, the description
doesn't follow a certain set of rules (num num - alpha alpha etc).  They
are literally anything a supplier has put in for them.  


The example below (21-MA-GAB) is stored differently by these analyzers:

WhitespaceAnalyzer:     [21-MA-GAB]

SimpleAnalyzer:         [ma] [gab]

StopAnalyzer:           [ma] [gab]

StandardAnalyzer:       [21-ma] [gab]

SynonymAnalyzer:        [21-ma] [gab]

      (One I created for synonyms.. much like the standard one)

SnowballAnalyzer:       [21-ma] [gab]


My problem is searching for 21magab returns nothing as well as 21ma*


This is just one of my punctuation problems.. there can be "" for inches
and 1/2 items etc..


I am currently using my SynonymnAnalyzer for some aliases to build the
index and the SnowballAnalyzer to query the index (nice stemming in it)




-----Original Message-----
From: Chris D [] 
Sent: Tuesday, June 28, 2005 2:41 PM
Subject: Re: Indexing puncutation


On 6/28/05, Aigner, Thomas <> wrote:

> Hello all,


>         I am VERY new to Lucene and we are trying out Lucene to see if

> it will accomplish the vast majority of our search functions.


>         I have a question about a good way to index some of our

> description codes.  We have description codes like 21-MA-GAB and other

> punctuation.  Our users need to be able to search for "21 MA GAB" or

> "21-MA_GAB" or "21MAGAB".  Is the best way to accomplish this by

> creating synonyms for the 3 different ways when punctuation is in

> to search for? I know I can stop punctuation in the index but what

> grouping the information together or with spaces?


> Thanks all in advance,

> Tom


There is a couple ways to do this, and I'm not sure which would be

best. (I'm also fairly new to lucene)


You can create a grammar that recognizes your product codes (see

StandardAnalyzer code for examples on how to do that) then use a

custom filter to normalize everything.


Forgive my poor lex but general idea


| <CODE: <NUM><NUM>  ("-"|"_"|""|" ") <ALPHA>+ ("-"|"_"|""|" ") <ALPHA>+


Then in the filter, normalize to strip out all of the punctuation.

This can be done with a regex or something faster but just for



   if (type == CODE_TYPE) {

      return new org.apache.lucene.analysis.Token(text.replaceAll("-",

""), t.startOffset(), t.endOffset(), type);

   } ... 


See StandardAnalyzer, it has a lot of code that would do what you need

and you can copy, paste and edit.


You could also do synonyms but that seems like it would be more


If you think of a better way, let me know, I have to do something






To unsubscribe, e-mail:

For additional commands, e-mail:


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message