lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mukul Ranjan <mran...@egain.com>
Subject RE: CompressingStoredFieldReader object taking lot of heap space
Date Fri, 23 Dec 2016 13:14:50 GMT
Thanks Adrien for quick reply!

We use jboss app server to handle application level threads.

Thanks,
Mukul Ranjan

From: Adrien Grand [mailto:jpountz@gmail.com]
Sent: Friday, December 23, 2016 6:33 PM
To: Mukul Ranjan <mranjan@egain.com>; Lucene Users Mailing List <java-user@lucene.apache.org>
Subject: Re: CompressingStoredFieldReader object taking lot of heap space

Hi,
SearcherManager only cares about managing readers/searchers in a thread-safe way, it does
not know about the thread pool it runs in.
How does your application handle threads? I'm thinking you might be running in a servlet container,
or a similar framework that handles it for you?

Le ven. 23 déc. 2016 à 13:57, Mukul Ranjan <mranjan@egain.com<mailto:mranjan@egain.com>>
a écrit :
Hi Adrien,

We are using searcher manager for acquiring  and releasing index searcher object. Is there
any way we can control fixed thread pool using searcher Manager?

Thanks,
Mukul Ranjan

From: Adrien Grand [mailto:jpountz@gmail.com<mailto:jpountz@gmail.com>]
Sent: Thursday, December 22, 2016 11:25 PM
To: Mukul Ranjan <mranjan@egain.com<mailto:mranjan@egain.com>>
Cc: java-user@lucene.apache.org<mailto:java-user@lucene.apache.org>

Subject: Re: CompressingStoredFieldReader object taking lot of heap space

Hi Mukul,

Lucene maintains a number of threadlocals alongside each segment. While these structures should
remain small (a couple tens of KB at most), the fact we have them per segment and in threadlocals
means that overall memory usage for these structures is linear with the number of threads
that have access to the index and to the number of segments that your index has.
So you plan on having access to an index from multiple thread, you should make sure it is
a fixed thread pool that has a reasonable number of threads (in the order of the number of
parallel threads that the host can run).

Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mranjan@egain.com<mailto:mranjan@egain.com>>
a écrit :
Hi Adrien,

Could you please explain the 2nd point in detail?

Thanks,
Mukul Ranjan

-----Original Message-----
From: Adrien Grand [mailto:jpountz@gmail.com<mailto:jpountz@gmail.com>]
Sent: Wednesday, December 21, 2016 6:43 PM
To: java-user@lucene.apache.org<mailto:java-user@lucene.apache.org>
Subject: Re: CompressingStoredFieldReader object taking lot of heap space

This issue has been reported a couple times and is usually due to one of these issues:
 - readers are not always closed
 - the index is accessed from a non fixed thread pool
 - there are too many indices open at the same time

Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mranjan@egain.com<mailto:mranjan@egain.com>>
a écrit :

> Hi,
>
> We are using lucene 5.5.2 for search.
>
> We ran load test for 8 hours on our setup and observed that
> "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> object is taking  210 MB out of 1.2 GB heap space. There are 8890
> instances of this class present in the heap.
>
> Could you please confirm if this is the normal behaviour or there is
> some issue with the code.
>
> Thanks,
> Mukul Ranjan
>
> -----
> Watch an eGain Try+Buy(tm) customer success video<
> https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message