Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of benson@basistech.com
 designates 209.85.219.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <534164ED.6090702@orcatec.com>
References: <20140406105940290168.13b5@1c6f6534dcfc>
	<CALm0H54DQ0f_qRR9m5zRePXh8goCQFTY=ghQfHd_rRLd_YNg+A@mail.gmail.com>
	<534164ED.6090702@orcatec.com>
Date: Sun, 6 Apr 2014 10:37:27 -0400
Message-ID: 
 <CALm0H55bCEkq2zwRU3rP-2wPzp=uRVpByThBPvuR+xH41Uvg5Q@mail.gmail.com>
Subject: Re: Confuse with Kuromoji
From: Benson Margulies <benson@basistech.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=089e015387029d243904f660b0b0

--089e015387029d243904f660b0b0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Sun, Apr 6, 2014 at 10:30 AM, Herb Roitblat <herb.roitblat@orcatec.com>w=
rote:

> Just curious, what are some of the things that people do to properly
> tokenize the queries with mixed language collections?  What do you do wit=
h
> mixed language queries?
>

You can either force the user to tell you the language, or ...

   you can run a language detector. They are less accurate for short
strings, or ...

     you can process it in _all_ of the languages and OR up the results.


>
> On 4/6/2014 4:51 AM, Benson Margulies wrote:
>
>> You must know what language each text is in, and use an appropriate
>> analyzer. Some people do this by using a separate field (text_eng,
>> text_spa, text_jpn). Other people put some extra information at the
>> beginning of the field, and then make an analyzer that peeks in order to
>> dispatch to the correct tokenizer.
>>
>>
>> On Sat, Apr 5, 2014 at 9:59 PM, <j7a42e4fd7qux@softbank.ne.jp> wrote:
>>
>>  I am pretty new with Lucene, however I have not problem understanding
>>> what
>>> is about.
>>> My big problem is trying to understand how Kuromoji works. I need to
>>> implement a search functinality thats supports initially English, Spani=
sh
>>> and Japanese. I doesn't seem to be a deal with the two firsts, as I can
>>> just use the analyzers=E3=83=BCcommon to index both languages contents,=
 but when
>>> it
>>> comes to Japanese it has it's own analyzer. I could't find any clues
>>> about
>>> combining analyzers, so I still don't if I can combine all languages
>>> under
>>> the same index (which would be ideal, as I expect mix searches in the
>>> context of my project) or I have to detect the language first and then
>>> index Japanese texts separately (what it will be a big disadvantage whe=
n
>>> it
>>> comes to mixed searches and future localization expansion).
>>> I found out about Lucene throgh Kuromoji, it will be great to find out =
a
>>> solution to be able to use all the greatness that Lucene offers.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--089e015387029d243904f660b0b0--