lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ankur.168" <ankur.15...@gmail.com>
Subject Lucene 6.5.1 memory consumption on 64 bit linux System
Date Mon, 16 Apr 2018 05:28:53 GMT
Hi,

I am facing GC issues while fetching search response from Lucene. Below are
the details of how I am using Lucene.

Server Details - Linux 64 bit, 12 cores 32 GB Ram 
Total number of applications running on box - 4(there are 2 other
application running on same box with lesser configurations)
*JVM 1(to index data)* Details - total - 8 GB, eden=5 GB, S0+S1 = 1 GB and
Old gen=2 GB
*JVM 2 (to retrieve data)* Details - total - 8 GB, eden=5 GB, S0+S1 = 1 GB
and Old gen=2 GB

I am using *Lucene 6.5.1 to index* my data from DB and then retrieve the
same. I have 2 different indexes(lets say index 1 and index 2). *Index 1
size is 1.49 GB* with docs count 436299 and terms count 5517926. *Index 2
size is 525 MB* with docs count 615357 and terms count 2909071. 
Both indexes are stored using *MMapDirectory *implementation with all fields
as indexed true. I reindex both indexes twice a day using cron jobs. 

While retrieving I initialize indexReader and IndexSearcher on server
startup, here again I am using *FSDirectory.open* which in turn create
MMapdirectory object since I am running this on a 64 bit linux box. There is
a job which runs every 5 mins to check if new index is created or not, in
case new index is created I will recreate indexReader and IndexSearcher and
close the old ones.

When I search for docs, I am using *SearcherManager.acquire() to obtain a
new Searcher *Instance and in finally block I am release the same using
SearcherManager.release(). So basically Searcher is every request based.

I have noticed that one server startup for *first 2-3 days there are too
many minor GC* happens but rare Major GC, but *after 2-3 days Major GC
starts* and sometimes it pauses my application for a minute too. 
*The Eden space is always filled even if I am not firing any requests to the
application.*
When I took a look at heap dump, its either ScoreDoc or SearcherManager
instances which is consuming most of the memory.

Can you guys help me here to understand below things-
1. why this behaviour is happening?
2. How lucene manages memory in JVM? 
3. How indexReader and Searcher manager instances consume the memory?



--
Sent from: http://lucene.472066.n3.nabble.com/Lucene-Java-Users-f532864.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message