lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Whole RAM consumed while Indexing.
Date Wed, 18 Mar 2015 16:04:25 GMT
On 3/18/2015 9:44 AM, Nitin Solanki wrote:
>              I am just saying. I want to be sure on commits difference..
> What if I do frequent commits or not? And why I am saying that I need to
> commit things so very quickly because I have to index 28GB of data which
> takes 7-8 hours(frequent commits).
> As you said, do commits after 60000 seconds then it will be more expensive.
> If I don't encounter with **"overlapping searchers" warning messages** then
> I feel it seems to be okay. Is it?

Even if the commit only handles a single document and it's a soft
commit, it is an expensive operation in terms of CPU, and in a
garbage-collected environment like Java, memory churn as well.  A commit
also invalidates the Solr caches, so if you have autowarming turned on,
then you have the additional overhead of doing a bunch of queries to
warm the new cache - on every single soft commit.

Doing commits as often as three times a second (you did say the interval
was 300 milliseconds) is generally a bad idea.  Increasing the interval
to once a minute will take a huge amount of load off of your servers, so
indexing will happen faster.

Thanks,
Shawn


Mime
View raw message