lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tod <listac...@gmail.com>
Subject Re: Retrieving indexed content containing multiple languages
Date Tue, 16 Nov 2010 15:07:10 GMT
On 11/11/2010 3:24 PM, Dennis Gearon wrote:
> I look forward to the eanswers to this one.

Well, it seems it was as easy as adding the CJKTokenizerFactory:

<fieldtype name="text_cjk" class="solr.TextField" 
positionIncrementGap="100">
  <analyzer>
   <tokenizer class="solr.CJKTokenizerFactory"/>
  </analyzer>
</fieldtype>


Once I did that and reindexed I could search for both english and 
chinese using the default 'text' field.  The next hurdle was getting the 
javascript to cooperate.  The chinese characters were getting corrupted 
on the way to the AJAX call against the Solr server.

As it turned out I was performing a POST to Solr using the jQuery .ajax 
api call.  Apparently when executing a POST you need to make sure the 
characters entered into the input field of the form are converted to 
unicode (\u7968 for example) prior to the AJAX call to Solr. 
Conversely, if executing a GET you need to convert the characters to 
UTF8 (%E7%A5%A8).

So now my customers are happily finding the appropriate document using 
english and chinese.

If someone could check my math I would appreciate it.  If it looks 
reasonable and there is nothing else written about it on the wiki I'll 
create a tutorial to give everybody else a leg up.


- Tod



> ----- Original Message ----
> From: Tod<listacctc@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Thu, November 11, 2010 11:35:23 AM
> Subject: Retrieving indexed content containing multiple languages
>
> My Solr corpus is currently created by indexing metadata from a relational
> database as well as content pointed to by URLs from the database.  I'm using a
> pretty generic out of the box Solr schema.  The search results are presented via
> an AJAX enabled HTML page.
>
> When I perform a search the document title (for example) has a mix of english
> and chinese characters.  Everything there is fine - I can see the english and
> chinese returned from a facet query on title.  I can search against the title
> using english words it contains and I get back an expected result.  I asked a
> chinese friend to perform the same search using chinese and nothing is returned.
>
> How should I go about getting this search to work?  Chinese is just one
> language, I'll probably need to support more in the future.
>
> My thought is that the chinese characters are indexed as their unicode
> equivalent so all I'll need to do is make sure the query is encoded
> appropriately and just perform a regular search as I would if the terms were in
> english.  For some reason that sounds too easy.
>
> I see there is a CJK tokenizer that would help here.  Do I need that for my
> situation?  Is there a fairly detailed tutorial on how to handle these types of
> language challenges?
>
>
> Thanks in advance - Tod
>
>


Mime
View raw message