Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of solr@elyograg.org designates
 166.70.79.219 as permitted sender)
Message-ID: <53448AEB.8090204@elyograg.org>
Date: Tue, 08 Apr 2014 17:48:59 -0600
From: Shawn Heisey <solr@elyograg.org>
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64;
 rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: Investigating performance issues in solr cloud
References: 
 <CADjjot9uw6UDhfS5cpjhOjA+VWM5FR31z33Ke+yoGdfo8HijXQ@mail.gmail.com>
In-Reply-To: 
 <CADjjot9uw6UDhfS5cpjhOjA+VWM5FR31z33Ke+yoGdfo8HijXQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 4/8/2014 5:30 PM, Utkarsh Sengar wrote:
> I see sudden drop in throughput once every 3-4 days. The "downtime" is for
> about 2-6minutes and things stabilize after that.
> 
> But I am not sure what is causing it the problem.
> 
> I have 3 shards with 20GB of data on each shard.
> Solr dashboard: http://i.imgur.com/6RWT2Dj.png
> Newrelic graphs when during the downtime of about 4hours:
> http://i.imgur.com/9vhKiB2.png
> JVM memory graph says its normal: http://i.imgur.com/pAycgdC.png
> 
> I thought it was GC pauses but it should be in the newrelic logs.
> 
> How can I go about investigating this problem? I am running solr 4.4.0, I
> don't see a strong reason to upgrade yet.

Lots of questions:

How many total machines?  What is your replicationFactor?  Does each
machine have one shard replica and therefore 20GB of total index data,
or if you add up all the index directories for the cores on each
machine, is there more than 20GB of data?

What options are you passing to your JVM when you start the servlet
container that runs Solr?

The dashboard says that this machine has 24GB of RAM and a 9GB heap.  Is
this the case for all machines?  Is there any software other than Solr
on the machine?

If it's a linux/unix machine, can you run top, press shift-M to sort by
memory, and grab a screenshot?  If it's a Windows machine, a similar
list should be available in the task manager, but it must include all
processes for all users on the whole machine, and it would be best if it
showed virtual memory as well as private.

Thanks,
Shawn