lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aigner, Thomas" <TAig...@WescoDist.com>
Subject RE: Inconsistent tokenizing of words containing underscores.
Date Mon, 29 Aug 2005 22:12:29 GMT
What seems to be working for me is a punctuation filter that removes / -
_ etc and makes the token without them.  Then "most" of the time the
word XYZZZY_DE_SA0001 will be tokenized as XYZZZYDESA0001.  For this to
work, you will have to use the same punctuation filter on the strings
before you search for them.  

Tom

-----Original Message-----
From: Daniel Naber [mailto:lucenelist@danielnaber.de] 
Sent: Monday, August 29, 2005 3:15 PM
To: java-user@lucene.apache.org
Subject: Re: Inconsistent tokenizing of words containing underscores.

On Monday 29 August 2005 19:21, Jeremy Meyer wrote:

> The expected behavior is to sometimes treat a character as indicating
a
> new token and other times to ignore the same character?

It depends on whether there are digits in the token.  It's documented in

the javacc source for the tokenizer(?).

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message