lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shaya Potter <spot...@gmail.com>
Subject easy way to figure out most common tokens?
Date Wed, 15 Aug 2012 17:46:37 GMT
Is there an easy way to figure out the most common tokens and then 
remove those tokens from the documents.

use case: imagine one is indexing a mailing list (such as this 
java-user) and is extracting all e-mail addresses in the messages and 
adding them to a doc.

What that means is that one will be a lot of

java-user-unsubscribe@lucene.apache.org
java-user-help@lucene.apache.org

due to that being in the signature of each email.

while, the best approach might be to not put it in the index in the 
first place, I'm wondering if there's a good way to process the index 
after the fact to remove these type of entries.

thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message