lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Potter <thelabd...@gmail.com>
Subject Re: Solr Cloud and Hadoop
Date Fri, 12 Oct 2012 19:46:09 GMT
Hi Rui,

If you're going to shard and/or replicate your index, then be sure to take
a look at CloudSolrServer in the SolrJ client library. CloudSolrServer is
an extension to SolrServer that works with Zookeeper to understand the
shards and replicas in a Solr cluster. Using CloudSolrServer, there is no
single point-of-failure during distributed indexing.

At my company, we use Pig (on top of Hadoop) to "enrich" documents before
they are indexed so we developed a Pig StoreFunc that uses CloudSolrServer
under the covers. We achieve very high throughput rates with this
configuration. Also, you mentioned you are new to Hadoop so definitely take
a look at Pig vs. doing lower-level MapReduce tasks.

Cheers,
Tim

On Fri, Oct 12, 2012 at 1:41 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hello Rui,
>
> If your data to be indexed is in HDFS, using MapReduce to parallelize
> indexing is still a good idea.
>
> Otis
> --
> Search Analytics - http://sematext.com/search-analytics/index.html
> Performance Monitoring - http://sematext.com/spm/index.html
>
>
> On Fri, Oct 12, 2012 at 2:35 PM, Rui Vaz <rui.wax@gmail.com> wrote:
> > Hello,
> >
> > Solr Cloud and Hadoop are new to me. And I am figuring out an
> > architecture to do a
> > distributed indexing/searching system in a cluster. Integrating them is
> an
> > option.
> >
> > I would like to know if Hadoop + Solr is still a good option to build
> the a
> > big index in a cluster,
> > using HDFS and MapReduce, or if the new functionalities in Solr Cloud
> make
> > Hadoop unnecessary.
> >
> > I know I provided few insight about the number of shards, or if I have
> more
> > network throughput
> > or memory constraints. I want to launch the discussion and see diferent
> > points of view.
> >
> > Thank you very much,
> > --
> > Rui Vaz
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message