Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
MIME-Version: 1.0
References: <CY4PR08MB2551292911A5AD435DC07743A4930@CY4PR08MB2551.namprd08.prod.outlook.com>
 <CAPsWd+PFOcRgGGw=aNB694DFZ77rrxoZsJTm9X54FGx16Hq9rw@mail.gmail.com>
 <CY4PR08MB25517EABFF345F1B7871E033A4920@CY4PR08MB2551.namprd08.prod.outlook.com>
 <CAPsWd+O=zjPMm0tmwUX-jy+HdCAebTKRgyH-5B7z+KaZbEbPjg@mail.gmail.com> <CY4PR08MB25511B7D5B0AD135F4511AD5A4950@CY4PR08MB2551.namprd08.prod.outlook.com>
In-Reply-To: <CY4PR08MB25511B7D5B0AD135F4511AD5A4950@CY4PR08MB2551.namprd08.prod.outlook.com>
From: Adrien Grand <jpountz@gmail.com>
Date: Fri, 23 Dec 2016 13:03:09 +0000
Message-ID: <CAPsWd+P8EE0_OVmxc4d0uobfSiiSXG5-KmemhT8O9tj41f48eA@mail.gmail.com>
Subject: Re: CompressingStoredFieldReader object taking lot of heap space
To: Mukul Ranjan <mranjan@egain.com>,
	Lucene Users Mailing List <java-user@lucene.apache.org>
Content-Type: multipart/alternative; boundary=001a113f8d5696c6ad05445302b3
archived-at: Fri, 23 Dec 2016 13:03:27 -0000

--001a113f8d5696c6ad05445302b3
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

SearcherManager only cares about managing readers/searchers in a
thread-safe way, it does not know about the thread pool it runs in.

How does your application handle threads? I'm thinking you might be running
in a servlet container, or a similar framework that handles it for you?

Le ven. 23 d=C3=A9c. 2016 =C3=A0 13:57, Mukul Ranjan <mranjan@egain.com> a =
=C3=A9crit :

> Hi Adrien,
>
>
>
> We are using searcher manager for acquiring  and releasing index searcher
> object. Is there any way we can control fixed thread pool using searcher
> Manager?
>
>
>
> Thanks,
> Mukul Ranjan
>
>
>
> *From:* Adrien Grand [mailto:jpountz@gmail.com]
> *Sent:* Thursday, December 22, 2016 11:25 PM
> *To:* Mukul Ranjan <mranjan@egain.com>
> *Cc:* java-user@lucene.apache.org
>
>
> *Subject:* Re: CompressingStoredFieldReader object taking lot of heap
> space
>
>
>
> Hi Mukul,
>
> Lucene maintains a number of threadlocals alongside each segment. While
> these structures should remain small (a couple tens of KB at most), the
> fact we have them per segment and in threadlocals means that overall memo=
ry
> usage for these structures is linear with the number of threads that have
> access to the index and to the number of segments that your index has.
>
> So you plan on having access to an index from multiple thread, you should
> make sure it is a fixed thread pool that has a reasonable number of threa=
ds
> (in the order of the number of parallel threads that the host can run).
>
>
>
> Le jeu. 22 d=C3=A9c. 2016 =C3=A0 07:25, Mukul Ranjan <mranjan@egain.com> =
a =C3=A9crit :
>
> Hi Adrien,
>
> Could you please explain the 2nd point in detail?
>
> Thanks,
> Mukul Ranjan
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Wednesday, December 21, 2016 6:43 PM
> To: java-user@lucene.apache.org
> Subject: Re: CompressingStoredFieldReader object taking lot of heap space
>
> This issue has been reported a couple times and is usually due to one of
> these issues:
>  - readers are not always closed
>  - the index is accessed from a non fixed thread pool
>  - there are too many indices open at the same time
>
> Le mer. 21 d=C3=A9c. 2016 =C3=A0 13:37, Mukul Ranjan <mranjan@egain.com> =
a =C3=A9crit :
>
> > Hi,
> >
> > We are using lucene 5.5.2 for search.
> >
> > We ran load test for 8 hours on our setup and observed that
> > "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> > object is taking  210 MB out of 1.2 GB heap space. There are 8890
> > instances of this class present in the heap.
> >
> > Could you please confirm if this is the normal behaviour or there is
> > some issue with the code.
> >
> > Thanks,
> > Mukul Ranjan
> >
> > -----
> > Watch an eGain Try+Buy(tm) customer success video<
> > https://www.youtube.com/watch?v=3DewEeyeqJRCE&feature=3Dyoutu.be/>
> >
>
>

--001a113f8d5696c6ad05445302b3--