hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Kerzner <mark.kerz...@shmsoft.com>
Subject Re: Hadoop/Lucene + Solr architecture suggestions?
Date Thu, 11 Oct 2012 02:26:47 GMT
That is very interesting, Lance, thank you.

Mark

On Wed, Oct 10, 2012 at 9:15 PM, Lance Norskog <goksron@gmail.com> wrote:

> In the LucidWorks Big Data product, we handle this with a reducer that
> sends documents to a SolrCloud cluster. This way the index files are not
> managed by Hadoop.
>
> ----- Original Message -----
> | From: "Ted Dunning" <tdunning@maprtech.com>
> | To: user@hadoop.apache.org
> | Cc: "Hadoop User" <user@hadoop.apache.org>
> | Sent: Wednesday, October 10, 2012 7:58:57 AM
> | Subject: Re: Hadoop/Lucene + Solr architecture suggestions?
> |
> | I prefer to create indexes in the reducer personally.
> |
> | Also you can avoid the copies if you use an advanced hadoop-derived
> | distro. Email me off list for details.
> |
> | Sent from my iPhone
> |
> | On Oct 9, 2012, at 7:47 PM, Mark Kerzner <mark.kerzner@shmsoft.com>
> | wrote:
> |
> | > Hi,
> | >
> | > if I create a Lucene index in each mapper, locally, then copy them
> | > to under /jobid/mapid1, /jodid/mapid2, and then in the reducers
> | > copy them to some Solr machine (perhaps even merging), does such
> | > architecture makes sense, to create a searchable index with
> | > Hadoop?
> | >
> | > Are there links for similar architectures and questions?
> | >
> | > Thank you. Sincerely,
> | > Mark
> |
>

Mime
View raw message