lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: Language detection
Date Thu, 27 Jun 2013 17:11:15 GMT
You can use the LangDetectLanguageIdentifierUpdateProcessorFactory update 
processor to redirect languages to alternate fields, and then set the 
non-English fields to be "ignored". But, the document would still be 
indexed, just without the redirected text fields.

(Examples of using that update processor are in my book - but not the 
"ignored" step.)

There is also a Tika-specific processor as well:
TikaLanguageIdentifierUpdateProcessorFactory

If you really want to completely suppress the indexing of documents 
containing non-English text, you'll have to make an explicit check before 
sendting the document to Solr. Tika also has language detection, so you 
could call Tika from an external process before sending the document to 
Solr.

-- Jack Krupansky

-----Original Message----- 
From: Hang Mang
Sent: Thursday, June 27, 2013 11:45 AM
To: java-user@lucene.apache.org
Subject: Language detection

Hello,

is there some kind of a filter or component that I could use to filter
non-english text? I have a preprocessing step that I only want to index
English documents.

Best,
Gucko 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message