lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wulf Berschin <>
Subject Re: Undo hyphenation when indexing
Date Mon, 04 Apr 2011 12:21:04 GMT
Thank you, Yonnik for this hint. (Again, I wasn't aware that obviousely 
Solr offers useful extensions for the Lucene indexing process and I 
wonder why they haven't been added to Lucene itself.)

Anyway, since the HyphenatedWordsFilter needs newlines in the input I 
will have to take another Tokenizer than StandardTokenizer. If I simply 
take the WhitespaceTokenizerFactory (as suggested by 
HyphenatedWordsFilterFactory) I will loose the punctuation handling done 
by StandardTokenizer, right? What will I have to borrow for that? Or do 
I have to extend StandardTokenizerImpl.jflex?


Am 01.04.2011 18:23, schrieb Yonik Seeley:
> Solr has a hyphenated word filter you could copy.
> On trunk, this has been folded into the analysis module.
> -Yonik
> -- Lucene/Solr User Conference, May
> 25-26, San Francisco
> On Fri, Apr 1, 2011 at 11:50 AM, Wulf Berschin<>  wrote:
>> Hi,
>> for indexing PDF files we have to undo word hyphenation. The basic idea is
>> simply to remove the hyphen when a new line and a small letter follows. Of
>> course this approach isnt 100%-foolproofed but checking against a dictionary
>> wouldnt be as well...
>> Since we face this problem too when highlighting using HTMLCharStripper
>> (yes, we do have hyphenation in our HTML docs...) it seems to me I have to
>> adjust the JFlex generated StandardTokenizerImpl.
>> Is this the right approach and hwo would I have to modify this script?
>> Thanks
>> Wulf
>> PS: I see that there are changes made in the brand new 3.1.0 version we are
>> using 3.0.3, but as far I understand no relevant changes in this respect.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message