lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elie Choueiri" <elie.choue...@elementn.com>
Subject RE: Multiple Languages with Lucene (Arabic & English)
Date Thu, 26 Jul 2007 07:06:39 GMT

Thanks for the clarification, I'll play around with it and head back if
things don't work according to plan.

Thanks again,
e

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Tuesday, July 24, 2007 5:53 PM
To: java-user@lucene.apache.org
Subject: Re: Multiple Languages with Lucene (Arabic & English)

You'll also find lots of discussion about indexing multiple
languages if you search the mail archive for things like multiple
language.

I think one thing you're missing is that Lucene indexes data however
you tell it to. You have both total control over and total responsibility
for how things are indexed. You could, say, determine that a document
was English, and index all of the fields like
title_english
text_english
etc.. and then write that document out to your index. Say the next
document was Arabic. You could index
title_arabic
text_arabic
and write that document to your index. Now if you're searching
in Arabic, you search on the title_arabic and text_arabic fields and
you'd never get any hits on English documents.

Not saying you want to do it this way, but you need to be clear that
Lucene well do what you tell it to. Nothing more and nothing less <G>.

Best
Erick


On 7/24/07, Grant Ingersoll <gsingers@apache.org> wrote:
>
>
> On Jul 24, 2007, at 3:21 AM, Elie Choueiri wrote:
>
> > Hi
> >
> >
> >
> > I'm new to searching and am trying to use Lucene to search English
> > & Arabic
> > documents.  I've got a bunch of questions (hopefully you'll find some
> > interesting!) and am hoping someone's gone through some of them and
> > has some
> > answers for me!
> >
> >
> >
> > First, do I have to worry about the Arabic Analyzer overwriting the
> > index
> > files of the English analyzer? (Or vice versa?)
> >
> > i.e. When I index documents a second time, will data be overwritten?
> >
>
> That depends whether you tell Lucene to create a new index or not.
> See the IndexWriter API for your options.
>
> >
> >
> > I could just store the index files for different languages in a
> > different
> > location, but it's good to know and I'd rather not if I don't have
> > to :)
> >
> >
> >
> > Also, on the same note, if I'm indexing documents that contain both
> > Arabic
> > and English, will the index files created by the English (or Arabic)
> > analyzer contain garbage or become corrupted because of the language
> > difference?
> >
>
> I don't know if it will be corrupted, but probably won't be all that
> useful, either. You may find the PerFieldAnalyzerWrapper to be helpful.
>
> >
> >
> > It is possible to index (using an English/Latin/Standard analyzer)
> > a file
> > that contains both english and arabic words, and expect the
> > searches in
> > English using the same analyzer to be valid, right?
>
> I should think so.  I don't recall running across this case too much,
> but do remember the reverse, Arabic files w/ some English and the
> Arabic analyzer usually just skipped over the English leaving it
> intact, thus searching those English terms in the Arabic index worked
> just fine.
>
> >
> >
> >
> > In an Arabic document with a single English word (the name of a
> > corporation,
> > for example) will the English word even be indexed and located by a
> > search?
> > I could test something like this with a small subset of documents,
> > but I
> > doubt the actual usefulness of a test with such a tiny (relatively
> > speaking!) amount of data.. I know we can tell Lucene to store the
> > full copy
> > of the document, but does that affect the index itself?
> >
> >
> >
> > Finally, and here's the tricky one, are searches that contain both
> > English
> > and Arabic words valid?  My limited understanding of the way search
> > engines
> > work tells me the search analyzes the context of words as well as
> > statistical data to decide the relevance of hits, is this still
> > valid for
> > multi-lingual searches?
> >
> >
>
> They are valid, just not sure how useful, but that is for your app to
> decide.  I guess if your users know both Arabic and English, it
> probably isn't a big deal.  Lucene just tries to match up what is in
> the query w/ what is in the index, so if you have validly analyzed
> tokens in both the query and the index then Lucene should find them.
>
> HTH,
> Grant
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org/tech/lucene.asp
>
> Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

-----Original Message-----
From: Grant Ingersoll [mailto:gsingers@apache.org] 
Sent: Tuesday, July 24, 2007 3:11 PM
To: java-user@lucene.apache.org
Subject: Re: Multiple Languages with Lucene (Arabic & English)


On Jul 24, 2007, at 3:21 AM, Elie Choueiri wrote:

> Hi
>
>
>
> I'm new to searching and am trying to use Lucene to search English  
> & Arabic
> documents.  I've got a bunch of questions (hopefully you'll find some
> interesting!) and am hoping someone's gone through some of them and  
> has some
> answers for me!
>
>
>
> First, do I have to worry about the Arabic Analyzer overwriting the  
> index
> files of the English analyzer? (Or vice versa?)
>
> i.e. When I index documents a second time, will data be overwritten?
>

That depends whether you tell Lucene to create a new index or not.   
See the IndexWriter API for your options.

>
>
> I could just store the index files for different languages in a  
> different
> location, but it's good to know and I'd rather not if I don't have  
> to :)
>
>
>
> Also, on the same note, if I'm indexing documents that contain both  
> Arabic
> and English, will the index files created by the English (or Arabic)
> analyzer contain garbage or become corrupted because of the language
> difference?
>

I don't know if it will be corrupted, but probably won't be all that  
useful, either. You may find the PerFieldAnalyzerWrapper to be helpful.

>
>
> It is possible to index (using an English/Latin/Standard analyzer)  
> a file
> that contains both english and arabic words, and expect the  
> searches in
> English using the same analyzer to be valid, right?

I should think so.  I don't recall running across this case too much,  
but do remember the reverse, Arabic files w/ some English and the  
Arabic analyzer usually just skipped over the English leaving it  
intact, thus searching those English terms in the Arabic index worked  
just fine.

>
>
>
> In an Arabic document with a single English word (the name of a  
> corporation,
> for example) will the English word even be indexed and located by a  
> search?
> I could test something like this with a small subset of documents,  
> but I
> doubt the actual usefulness of a test with such a tiny (relatively
> speaking!) amount of data.. I know we can tell Lucene to store the  
> full copy
> of the document, but does that affect the index itself?
>
>
>
> Finally, and here's the tricky one, are searches that contain both  
> English
> and Arabic words valid?  My limited understanding of the way search  
> engines
> work tells me the search analyzes the context of words as well as
> statistical data to decide the relevance of hits, is this still  
> valid for
> multi-lingual searches?
>
>

They are valid, just not sure how useful, but that is for your app to  
decide.  I guess if your users know both Arabic and English, it  
probably isn't a big deal.  Lucene just tries to match up what is in  
the query w/ what is in the index, so if you have validly analyzed  
tokens in both the query and the index then Lucene should find them.

HTH,
Grant

--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org/tech/lucene.asp

Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message