lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mordo, Aviran (EXP N-NANNATEK)" <>
Subject RE: Implement a tokenizer
Date Mon, 21 May 2007 21:11:38 GMT
What you need to do is to create your own tokenizer. Just copy the code
from the StandardTokenizer to your XYZTokenizer and make your changes.
Then you need to create your own Analyzer class (again copy the code
from the StandardAnalyzer) and user your XYZTokenizer in the new
XYZAnalyzer you created.



-----Original Message-----
From: bhecht [] 
Sent: Monday, May 21, 2007 2:59 PM
Subject: Implement a tokenizer

Hi there,

I was interested in changing the StandardTokenzier so it will not remove
the "+" (plus) sign from my stream.
Looking in the code and documentation, it reads: 

"If this tokenizer does not suit your application, please consider
copying this source code directory to your project and maintaining your
own grammar-based tokenizer."

I can't understand from this code where I should jump in, and do my
Can someone point me out to where I should look at in order perform my

Thanks in advanced
View this message in context:
Sent from the Lucene - Java Users mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message