lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrien Grand <jpou...@gmail.com>
Subject Re: CompressingStoredFieldReader object taking lot of heap space
Date Thu, 22 Dec 2016 17:54:40 GMT
Hi Mukul,

Lucene maintains a number of threadlocals alongside each segment. While
these structures should remain small (a couple tens of KB at most), the
fact we have them per segment and in threadlocals means that overall memory
usage for these structures is linear with the number of threads that have
access to the index and to the number of segments that your index has.

So you plan on having access to an index from multiple thread, you should
make sure it is a fixed thread pool that has a reasonable number of threads
(in the order of the number of parallel threads that the host can run).

Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mranjan@egain.com> a écrit :

> Hi Adrien,
>
> Could you please explain the 2nd point in detail?
>
> Thanks,
> Mukul Ranjan
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Wednesday, December 21, 2016 6:43 PM
> To: java-user@lucene.apache.org
> Subject: Re: CompressingStoredFieldReader object taking lot of heap space
>
> This issue has been reported a couple times and is usually due to one of
> these issues:
>  - readers are not always closed
>  - the index is accessed from a non fixed thread pool
>  - there are too many indices open at the same time
>
> Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mranjan@egain.com> a écrit :
>
> > Hi,
> >
> > We are using lucene 5.5.2 for search.
> >
> > We ran load test for 8 hours on our setup and observed that
> > "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> > object is taking  210 MB out of 1.2 GB heap space. There are 8890
> > instances of this class present in the heap.
> >
> > Could you please confirm if this is the normal behaviour or there is
> > some issue with the code.
> >
> > Thanks,
> > Mukul Ranjan
> >
> > -----
> > Watch an eGain Try+Buy(tm) customer success video<
> > https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message