lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Isakson" <Eric.Isak...@sas.com>
Subject RE: Accentuated characters
Date Tue, 10 Dec 2002 21:22:35 GMT
If you really want to make your own TokenFilter, have a look at org.apache.lucene.analysis.LowerCaseFilter.next()

it does:
  public final Token next() throws java.io.IOException {
    Token t = input.next();

    if (t == null)
      return null;

    t.termText = t.termText.toLowerCase();

    return t;
  }

The termText member of the Token class is package scoped, so you will have to implement your
filter in the org.apache.lucene.analysis package. No worries about encoding as the termText
is already a java (unicode) string. You will just have to provide the mechanism to get the
accented characters converted to there non-accented equivalents. java.text.Collator has some
magic that does this for string comparisons but I couldn't find any public methods that give
you access to convert a string to its non-accented equivalent.

Eric
--
Eric D. Isakson        SAS Institute Inc.
Application Developer  SAS Campus Drive
XML Technologies       Cary, NC 27513
(919) 531-3639         http://www.sas.com



-----Original Message-----
From: stephane vaucher [mailto:vaucher@LUB.UMontreal.CA]
Sent: Tuesday, December 10, 2002 2:58 PM
To: lucene-user@jakarta.apache.org
Subject: Accentuated characters


Hello everyone,

I wish to implement a TokenFilter that will remove accentuated 
characters so for example 'é' will become 'e'. As I would rather not 
reinvent the wheel, I've tried to find something on the web and on the 
mailing lists. I saw a mention of a contrib that could do this (see 
http://www.mail-archive.com/lucene-user%40jakarta.apache.org/msg02146.html), 
but I don't see anything applicable.

Has anyone done this yet, if so I would much appreciate some pointers 
(or code), otherwise, I'll be happy to contribute whatever I produce 
(but it might be very simple since I'll only need to deal with french).

Cheers,
Stephane


--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message