lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Puffinburger" <>
Subject RE: 2.3.2 -> 2.4.0 StandardTokenizer issue
Date Thu, 19 Feb 2009 18:51:28 GMT
Actually, WhitespaceTokenizer won't work.   Too many person names and it
won't do anything with punctuation.   Something had to have changed in
StandardTokenizer, and we need some of the 2.4 fixes/features, so we are
kind of stuck.

-----Original Message-----
From: Philip Puffinburger [] 
Sent: Monday, February 16, 2009 7:19 PM
Subject: 2.3.2 -> 2.4.0 StandardTokenizer issue

We have our own Analyzer which has the following


Public final TokenStream tokenStream(String fieldname, Reader reader) {

  TokenStream result = new StandardTokenizer(reader);

  result = new StandardFilter(result);

  result = new MyAccentFilter(result);

  result = new LowerCaseFilter(result);

  result = new StopFilter(result);


  return result;



In 2.3.2 if the token ‘Cómo’ came through this it would get changed to
‘como’ by the time it made it through the filters.    In 2.4.0 this isn’t
the case.   It treats this one token as two so we get ‘co’ and ‘mo’.    So
instead of search ‘como’ or ‘Cómo’ to get all the hits we now have to do
them separately.


I switched to the WhitespaceTokenizer as a test and that is indexing and
searching the way we expect it, but I haven’t looked into what we lost by
using that tokenizer.


Were we relying on a bug to get what we wanted from StandardTokenizer or did
something break in 2.4.0?

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message