lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Heavy Multi-threaded indexing and SolrCloud 4.10.1 replicas out of synch.
Date Mon, 27 Oct 2014 13:26:22 GMT
Hi,

You may simply be overwhelming your cluster-nodes. Have you checked
various metrics to see if that is the case?

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



> On Oct 26, 2014, at 9:59 PM, S.L <simpleliving016@gmail.com> wrote:
>
> Folks,
>
> I have posted previously about this , I am using SolrCloud 4.10.1 and have
> a sharded collection with  6 nodes , 3 shards and a replication factor of 2.
>
> I am indexing Solr using a Hadoop job , I have 15 Map fetch tasks , that
> can each have upto 5 threds each , so the load on the indexing side can get
> to as high as 75 concurrent threads.
>
> I am facing an issue where the replicas of a particular shard(s) are
> consistently getting out of synch , initially I thought this was beccause I
> was using a custom component , but I did a fresh install and removed the
> custom component and reindexed using the Hadoop job , I still see the same
> behavior.
>
> I do not see any exceptions in my catalina.out , like OOM , or any other
> excepitions, I suspecting thi scould be because of the multi-threaded
> indexing nature of the Hadoop job . I use CloudSolrServer from my java code
> to index and initialize the CloudSolrServer using a 3 node ZK ensemble.
>
> Does any one know of any known issues with a highly multi-threaded indexing
> and SolrCloud ?
>
> Can someone help ? This issue has been slowing things down on my end for a
> while now.
>
> Thanks and much appreciated!

Mime
View raw message