lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Frequent deletions
Date Mon, 12 Jan 2015 20:02:31 GMT
On 1/10/2015 11:46 PM, ig01 wrote:
> Thank you all for your response,
> The thing is that we have 180G index while half of it are deleted documents.
> We  tried to run an optimization in order to shrink index size but it
> crashes on ‘out of memory’ when the process reaches 120G.   
> Is it possible to optimize parts of the index? 
> Please advise what can we do in this situation.

If you are getting "OutOfMemoryError" exceptions from Java, that means
your heap isn't large enough to accomplish what you have asked the
program to do (between the configuration and what you have actually
requested).  You'll either need to allocate more memory to the heap, or
you need to change your config so less memory is required.

I see from a later reply that the 120GB size you have mentioned is your
Java heap.  Unless you've got hundreds of millions of documents on one
Solr instance/server (which would not be a good idea) and/or a serious
misconfiguration, I cannot imagine needing a heap that big for Solr.

The largest index on my dev Solr server has 98 million documents in
seven shards, with a total index size a little over 120GB (six shards
each 20GB and a seventh shard that's less than 1GB), and my heap size is
7 gigabytes.  There is a smaller index as well with 17 million docs in
three shards, that one is about 10GB on disk.  Unlike the production
servers, the dev server has all the index data contained on one server.

Here's a wiki page that covers things which cause large heap
requirements.  A later section also describes steps you can take to
reduce memory usage.

https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

How many documents do you have on a single Solr server?  Can you use a
site like http://apaste.info to share your solrconfig.xml?  I don't know
if we'll need the schema, but it might be a good idea to share that as well.

Thanks,
Shawn


Mime
View raw message