lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmet Arslan <iori...@yahoo.com>
Subject Re: Strange behaviour of StandardTokenizer
Date Fri, 18 Jun 2010 07:17:41 GMT
> okay, so it is recognized as a number? 

Yes. You can see token type definitions in *.jflex file.

> Maybe I'll have to use another tokenizer.

MappingCharFilter with StandardTokenizer option exists.

NormalizeCharMap map = new NormalizeCharMap();
map.add("-", " "); 

TokenStream stream = new StandardTokenizer(
        new MappingCharFilter(map,
        new StringReader("nl-lt0"))); 




      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message