lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrien Grand <jpou...@gmail.com>
Subject Re: CompressingStoredFieldReader object taking lot of heap space
Date Fri, 23 Dec 2016 13:03:09 GMT
Hi,

SearcherManager only cares about managing readers/searchers in a
thread-safe way, it does not know about the thread pool it runs in.

How does your application handle threads? I'm thinking you might be running
in a servlet container, or a similar framework that handles it for you?

Le ven. 23 déc. 2016 à 13:57, Mukul Ranjan <mranjan@egain.com> a écrit :

> Hi Adrien,
>
>
>
> We are using searcher manager for acquiring  and releasing index searcher
> object. Is there any way we can control fixed thread pool using searcher
> Manager?
>
>
>
> Thanks,
> Mukul Ranjan
>
>
>
> *From:* Adrien Grand [mailto:jpountz@gmail.com]
> *Sent:* Thursday, December 22, 2016 11:25 PM
> *To:* Mukul Ranjan <mranjan@egain.com>
> *Cc:* java-user@lucene.apache.org
>
>
> *Subject:* Re: CompressingStoredFieldReader object taking lot of heap
> space
>
>
>
> Hi Mukul,
>
> Lucene maintains a number of threadlocals alongside each segment. While
> these structures should remain small (a couple tens of KB at most), the
> fact we have them per segment and in threadlocals means that overall memory
> usage for these structures is linear with the number of threads that have
> access to the index and to the number of segments that your index has.
>
> So you plan on having access to an index from multiple thread, you should
> make sure it is a fixed thread pool that has a reasonable number of threads
> (in the order of the number of parallel threads that the host can run).
>
>
>
> Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mranjan@egain.com> a écrit :
>
> Hi Adrien,
>
> Could you please explain the 2nd point in detail?
>
> Thanks,
> Mukul Ranjan
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Wednesday, December 21, 2016 6:43 PM
> To: java-user@lucene.apache.org
> Subject: Re: CompressingStoredFieldReader object taking lot of heap space
>
> This issue has been reported a couple times and is usually due to one of
> these issues:
>  - readers are not always closed
>  - the index is accessed from a non fixed thread pool
>  - there are too many indices open at the same time
>
> Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mranjan@egain.com> a écrit :
>
> > Hi,
> >
> > We are using lucene 5.5.2 for search.
> >
> > We ran load test for 8 hours on our setup and observed that
> > "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> > object is taking  210 MB out of 1.2 GB heap space. There are 8890
> > instances of this class present in the heap.
> >
> > Could you please confirm if this is the normal behaviour or there is
> > some issue with the code.
> >
> > Thanks,
> > Mukul Ranjan
> >
> > -----
> > Watch an eGain Try+Buy(tm) customer success video<
> > https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message