hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Bolotin" <ig...@collarity.com>
Subject RE: Hadoop + Lucene integration: possible? how?
Date Mon, 15 Jan 2007 20:29:19 GMT
Actually there is a patch available for creating Lucene indexes directly
on DFS.
See here: http://issues.apache.org/jira/browse/LUCENE-532

But as Andrzej mentioned - the performance of searching is not stellar.

Igor Bolotin

-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org] 
Sent: Monday, January 15, 2007 4:54 AM
To: hadoop-user@lucene.apache.org
Subject: Re: Hadoop + Lucene integration: possible? how?

maarten@sherpa-consulting.be wrote:
> I'm new to lucene and Hadoop but what I can't seem to find in the 
> docs, internet... is how (and if possible?) to use Hadoop as the 
> underlying FS for Lucene?
> Could anyone explain me how these can be tied together? Some small 
> code/configuration example would be nice :-)

It's possible to use Hadoop DFS to host a read-only Lucene index and use
it for searching (Nutch has an implementation of FSDirectory for this
purpose), but the performance is not stellar ... Currently it's not
(yet) possible to use HDFS for creating Lucene indexes, a minor change
to Lucene index format would be required.

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

View raw message