lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Doupnik <...@netlab1.net>
Subject Re: Solr-8.1.0 uses much more memory
Date Sun, 26 May 2019 18:52:47 GMT
     I do queries while indexing, have done so for a long time, without 
difficulty nor memory usage spikes from dual use. The system has been 
designed to support that.
     Again, one may look at the numbers using "top" or similar. Try Solr 
v8.0 and 8.1 to see the difference which I experience here. For 
reference, the only memory adjustables set in my configuration is in the 
Solr startup script solr.in.sh saying add "-Xss1024k" in the SOLR_OPTS 
list and setting SOLR_HEAP="4024m".
     Thanks,
     Joe D.

On 26/05/2019 19:43, Jörn Franke wrote:
> I think this is also a very risky memory strategy. What happens if you Index and query
at the same time etc. maybe it is more worth to provide as much memory as for concurrent operations
are needed. This includes JVM memory but also the disk caches.
>
>> Am 26.05.2019 um 20:38 schrieb Joe Doupnik <jrd@netlab1.net>:
>>
>>> On 26/05/2019 19:15, Joe Doupnik wrote:
>>>> On 26/05/2019 19:08, Shawn Heisey wrote:
>>>>> On 5/25/2019 9:40 AM, Joe Doupnik wrote:
>>>>>       Comparing memory consumption (real, not virtual) of quiesent Solr
v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but
v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10.
This is a major memory consumption issue. I have seen no mention of it in the docs nor forums.
>>>> If Solr is using 4 to 5 GB of memory on your system, it is only doing that
because you told it that it was allowed to.
>>>>
>>>> If you run a Java program with a minimum heap that's smaller than the max
heap, which Solr does not do by default, then what you will find is that Java *might* stay
lower than the maximum for a while.  But eventually it WILL allocate the entire maximum heap
from the OS, plus some extra for Java itself to work with. Solr 8.0 and Solr 8.1 are not different
from each other in this regard.
>>>>
>>>> Thanks,
>>>> Shawn
>>> --------
>>>      Not to be argumentative, prior to Solr v8.1 quiesent resident memory remained
at about the 1.6GB level, and during active indexing it could exceed 3.5GB. With the same
configuration settings Solr v8.1 changes that to use _a lot_ more memory. Thus something significant
has changed with Solr v8.1 when compared to its predecessors. The question is what, and what
can we do about it.
>>>      I am not about to enter a guessing game with Solr and Java and its heap
usage. That is far to complex to hope to win.
>>>      Thus, something changed, for the worse here in the field, and I do not know
what.
>>>      Thanks,
>>>      Joe D.
>> ---------------
>>      If I were forced to guess about this situation it woud be to flag an item mentioned
vaguely in passing: the garbage collector. How to return it to status quo ante is not known
here. Presumably such a step would be covered in the yet to appear documentation for Solr
v8.1
>>      To add a little more to the story. Memory remained at the 1.6GB level except
when doing heavy indexing. To "adjust" Solr so that it always consumes too much, as at present,
is not acceptable, nor is acceptable risking trouble by setting an upper limit down to say
1.6GB and thence cause indexing to fail.
>>      We see the dilemna. Expert assistance is needed to resolve this.
>>      Thanks,
>>      Joe D.


Mime
View raw message