Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of apache@elyograg.org
 designates 166.70.79.219 as permitted sender)
Message-ID: <5509A209.5030001@elyograg.org>
Date: Wed, 18 Mar 2015 10:04:25 -0600
From: Shawn Heisey <apache@elyograg.org>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: Whole RAM consumed while Indexing.
References: 
 <CA+fnCvV0yuaYOdBEvBbz8ToPcPfEmPackjwrMq512v61FFrKmA@mail.gmail.com>
	<CAEFAe-FpgcGisa2=098eem24vHZOZHYqLmAoYJ9LjOP1QTMbmw@mail.gmail.com>
	<CA+fnCvV4X5D0u6UsZpOOsKFKUh72XArhnqEMG3g+hWxpx3GvYg@mail.gmail.com>
	<CAN4YXveNxrpoDYLcfeaUz4D8OCXrdtShbuMXZ_Lx3Jy54RYQug@mail.gmail.com>
	<CA+fnCvWvgSB34G3a-gYXH3Vr39T6Oigy-LsHwimrDmFb-nj0iQ@mail.gmail.com>
	<CAN4YXvdak-v+Fs2uy-y=PVOv-aTRbP7Fi+67j17QvLyx7ZF32g@mail.gmail.com>
	<CADMUOEXh0U=8mEU4RA5LM2Chcyz2d6np=rr1pR+WU+tJ0=KkaA@mail.gmail.com>
	<CA+fnCvUj+=JXrr+7oWGk0ENxGLi6voHSxvLufbo6oeYTdp8C7Q@mail.gmail.com>
	<CAN4YXvdiBu-MEuFSYjo=yO=gFd8RW06Dc9=nS3Uq_ZPaefdV1Q@mail.gmail.com>
 <CA+fnCvXO0mj0pGOrwU_JwQ2hXUmwYutx1NOZr0oaGemXkpOciw@mail.gmail.com>
In-Reply-To: 
 <CA+fnCvXO0mj0pGOrwU_JwQ2hXUmwYutx1NOZr0oaGemXkpOciw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

On 3/18/2015 9:44 AM, Nitin Solanki wrote:
>              I am just saying. I want to be sure on commits difference..
> What if I do frequent commits or not? And why I am saying that I need to
> commit things so very quickly because I have to index 28GB of data which
> takes 7-8 hours(frequent commits).
> As you said, do commits after 60000 seconds then it will be more expensive.
> If I don't encounter with **"overlapping searchers" warning messages** then
> I feel it seems to be okay. Is it?

Even if the commit only handles a single document and it's a soft
commit, it is an expensive operation in terms of CPU, and in a
garbage-collected environment like Java, memory churn as well.  A commit
also invalidates the Solr caches, so if you have autowarming turned on,
then you have the additional overhead of doing a bunch of queries to
warm the new cache - on every single soft commit.

Doing commits as often as three times a second (you did say the interval
was 300 milliseconds) is generally a bad idea.  Increasing the interval
to once a minute will take a huge amount of load off of your servers, so
indexing will happen faster.

Thanks,
Shawn