Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of
 SRS0=tQ+EKB=QV=basetechnology.com=jack@yourhostingaccount.com designates
 65.254.253.110 as permitted sender)
Message-ID: <737CC217F0A849B7993C6E7B11714BF7@JackKrupansky>
From: "Jack Krupansky" <jack@basetechnology.com>
To: <solr-user@lucene.apache.org>
References: <1373218361758-4076057.post@n3.nabble.com>
In-Reply-To: <1373218361758-4076057.post@n3.nabble.com>
Subject: Re: Why shouldn't lang-id component work at query-time?
Date: Sun, 7 Jul 2013 13:47:35 -0400
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="iso-8859-1";
	reply-type=original
Content-Transfer-Encoding: 8bit
Importance: Normal
Sender: "Jack Krupansky" <jack@basetechnology.com>

The problem at query time is simple: a typical query has too few terms to 
reliably identify the language using statistical techniques, especially for 
a language like English which is famous for "borrowing" words from other 
languages. I mean, is "raison d'�tre" REALLY French anymore? Or, are 
"sombrero" or "poncho" or "ma�ana" really strictly Spanish anymore?

Multi-lingual support is an art/craft; don't expect cookbook answers that 
will apply to all apps in all environments.

That said, Edismax searching of multiple field, one for each language is 
probably the best you're going to do without doing something 
super-sophisticated.

-- Jack Krupansky

-----Original Message----- 
From: adfel70
Sent: Sunday, July 07, 2013 1:32 PM
To: solr-user@lucene.apache.org
Subject: Why shouldn't lang-id component work at query-time?

Hi,
I'm trying to integrate solr's lang-id component in my solr environment.
In my scenario, I have documents in many different languages. I want to
index them in the same solr collection, to different fields and apply
language-specific analyzers on each field by its language.

So far lang-id component does exactly what I need.

The problem is that in all recepies that I've read, eventually at query-time
I have to indicate which language I'm querying.
Either by specifying the field I want to search:
/solr/collection/select?q=text_it:abc abc
Or by creating a language-specific request handler which I would have to use
like this:
/solr/collection/selectIT?q=text:abc abc

Either way, I must tell solr the language, which in my case - a web
client+many different languages, it's quite problematic.

I was wondering why shouldn't lang-id component provide a full ability to
index and query on multi-languages when both in indexing and in querying the
language is transparent to the client.
This could be achieved by applying the same language-detection tool at query
time.

Any insights?


--
View this message in context: 
http://lucene.472066.n3.nabble.com/Why-shouldn-t-lang-id-component-work-at-query-time-tp4076057.html
Sent from the Solr - User mailing list archive at Nabble.com.