lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth teotia <siddharthteo...@gmail.com>
Subject Memory usage
Date Thu, 07 Nov 2019 04:27:37 GMT
Hi All

I have some questions about the memory usage. I would really appreciate if
someone can help answer these.

I understand from the docs that during reading/querying, Lucene uses
MMapDirectory (assuming it is supported on the platform). So the Java heap
overhead in this case will purely come from the objects that are
allocated/instantiated on the query path to process the query and build
results etc.  But the whole index itself will not be loaded into memory
because we memory mapped the file. Is my understanding correct? In this
case, we are better off not increasing the Java heap and keep as much
as possible available for the file system cache for mmap to do its job
efficiently.

However, are there any portions of index structures that are completely
loaded in memory regardless of whether it is MMapDirectory or not? If so,
are they loaded in Java heap or do we use off-heap (direct buffers) in
such cases?

Secondly, on the write path I think even though the writer opens a
MMapDirectory, the writes are gathered/buffered in memory upto a flush
threshold controlled by IndexWriterConfig. Is this buffering done in Java
heap or direct memory?

Thanks a lot for help
Siddharth

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message