lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: solr 3.5 taking long to index
Date Sun, 15 Apr 2012 06:14:42 GMT
You're doing more commits than you need. You may want to turn off
autocommit since you are running commit yourself. Every commit causes
segment activity, so if you want to minimize that, you don't need
autocommit.

About memory sizing: you should drop the memory assigned to Solr until
it slows down, then increase it a little. All of the rest should be
used by the OS for disk caching.

With this much ram, investigate "Large Pages" support. This is an
operating system hack to make large programs run faster in large ram
machines.

On Sat, Apr 14, 2012 at 7:33 PM, Rohit <rohit@in-rev.com> wrote:
> Hey Shawn,
>
> Solr is working better, though not out of the woods, freed up some memory is the system
and also increased the mergeFactor to 20.
>
> Has another question, we had autocommit ON all this while in our solrconfig.xml, but
since the upgrade we have been noticing keeping autocommit on is increasing the commit time,
though I cannot find a reason are they related in anyway?
>
>
> Regards,
> Rohit
> Mobile: +91-9901768202
> About Me: http://about.me/rohitg
>
>
> -----Original Message-----
> From: Shawn Heisey [mailto:solr@elyograg.org]
> Sent: 13 April 2012 11:01
> To: solr-user@lucene.apache.org
> Subject: Re: solr 3.5 taking long to index
>
> On 4/12/2012 8:42 PM, Rohit wrote:
>> The machine has a total ram of around 46GB. My Biggest concern is Solr index time
gradually increasing and then the commit stops because of timeouts, out commit rate is very
high, but I am not able to find the root cause of the issue.
>
> For good performance, Solr relies on the OS having enough free RAM to keep critical portions
of the index in the disk cache.  Some numbers that I have collected from your information
so far are listed below.
> Please let me know if I've got any of this wrong:
>
> 46GB total RAM
> 36GB RAM allocated to Solr
> 300GB total index size
>
> This leaves only 10GB of RAM free to cache 300GB of index, assuming that this server
is dedicated to Solr.  The critical portions of your index are very likely considerably larger
than 10GB, which causes constant reading from the disk for queries and updates.  With a high
commit rate and a relatively low mergeFactor of 10, your index will be doing a lot of merging
during updates, and some of those merges are likely to be quite large, further complicating
the I/O situation.
>
> Another thing that can lead to increasing index update times is cache warming, also greatly
affected by high I/O levels.  If you visit the /solr/corename/admin/stats.jsp#cache URL,
you can see the warmupTime for each cache in milliseconds.
>
> Adding more memory to the server would probably help things.  You'll want to carefully
check all the server and Solr statistics you can to make sure that memory is the root of problem,
before you actually spend the money.  At the server level, look for things like a high iowait
CPU percentage.  For Solr, you can turn the logging level up to INFO in the admin interface
as well as turn on the infostream in solrconfig.xml for extensive debugging.
>
> I hope this is helpful.  If not, I can try to come up with more specific things you
can look at.
>
> Thanks,
> Shawn
>
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message