lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fuad Efendi" <f...@efendi.ca>
Subject RE: Can SOLR Index UTF-16 Text
Date Wed, 03 Oct 2012 17:29:33 GMT
Something is missing from the body of your Email... As I pointed in my
previous message, "in general" Solr can index _everything_ (provided that
you have Tokenizer for that); but, additionally to _indexing_ you need an
HTTP-based _search_ which must understand UTF-16 (for instance)

Easiest solution is to transfer files to UTF-8 before indexing and to use
UTF-8 as a as default Java character encoding ( java -Dfile.encoding=UTF-8
...; including even Tomcat HTTP settings). This is really the simplest...
and fastest by performance... and you should be able to use Highlighter
feature and etc...


-Fuad Efendi
http://www.tokenizer.ca





-----Original Message-----
From: vybe3142 [mailto:vybe3142@gmail.com] 
Sent: October-03-12 12:30 PM
To: solr-user@lucene.apache.org
Subject: Re: Can SOLR Index UTF-16 Text

Thanks for all the responses. Problem partially solved (see below)

1. In a sense, my question is theoretical since the input to out SOLR server
is (currently) UTF-8 files produced by a third party text extraction utility
(not Tika). On the server side, we read and index the text via a custom data
handler. Last week, I tried a UTF-16 file to see what would happen, and it
wasn't handled correctly, as explained in my original question.

2. The file is UTF 16


3. We can either (a)stream the data to SOLR in the call or (b)use the
stream.file parameter to provide the file path to the SOLR handler.

Assuming case (a)

Here's how the SOLRJ request is constructed (code edited for conciseness)



If I replace the last line with

things work !!!!

What would I need to do in case (b), . wherer the raw file is loaded
remotely  i.e. my handler reads the file directly



In this case, how can I control what the content type is ?

Thanks




--
View this message in context:
http://lucene.472066.n3.nabble.com/Can-SOLR-Index-UTF-16-Text-tp4010834p4011
634.html
Sent from the Solr - User mailing list archive at Nabble.com.



Mime
View raw message