lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Right memory for search application
Date Tue, 27 Apr 2010 14:12:26 GMT
Sorting by score down to the second will use a lot of memory.  Can you
make it less granular?  And I think that switching that field to a
NumericField will give you some savings - this has come up before but
I can't remember the details.  I'm sure someone else will.


--
Ian.


On Tue, Apr 27, 2010 at 2:49 PM, Samarendra Pratap <samarzone@gmail.com> wrote:
> Hi Ian. Thanks for the points
>
> Here are my answers -
>
> 1. Our default option is sort by score, however almost 8% of searches use
> sorting on a field (yyyymmddHHMMSS). This field is indexed as string (not as
> NumericField or DateField).
>
> 2. We are opening readers at the time of starting the application, but
> Searchers are opened for every request choosing the right readers. A few
> days back I got an answer on this very forum that reopening a searcher is
> not as big issue as reopening a reader. However we are *not closing Searcher
> *s after request is served. Should that cause memory problem? Shouldn't GC
> handle that all as the reference of Searcher becomes inaccessible?
>
> 3. I can't do a profiling on production servers but running "top" shows that
> immediately starting search server resident memory (RES) starts from 500m
> and Virtual memory (VIRT) is around 5400m. It grows to RES => 1g in 5
> minutes and 4g within half an hour while VIRT remains same.
>
> 4. Right! I'll be doing a memory profiling. I hope I'll get some hints from
> there too.
>
> Do above points (specially # 2) indicate towards anything which I can do
> besides profiling?
>
>
>
> On Tue, Apr 27, 2010 at 6:53 PM, Ian Lea <ian.lea@gmail.com> wrote:
>
>> There is no simple answer.  However your app does sound to be using
>> rather a lot of memory for what you describe as simple searches.
>>
>> Are you using lucene sorting?  That can use lots of memory.  How are
>> you using/reusing searchers/readers?  Having multiple ones open, or
>> failing to close old ones, will use more memory.  Does memory usage
>> grow then stabilize or keep on growing?
>>
>> A memory profiler/heap dump could tell you what is really using all the
>> space.
>>
>>
>> --
>> Ian.
>>
>> On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap <samarzone@gmail.com>
>> wrote:
>> > Hi.
>> >  I am searching for some guidance on right memory options for my Search
>> > Server application. How much memory a lucene based application should be
>> > given?
>> >
>> >  Till a few days back I was running my search server on java 1.4 with
>> memory
>> > options "-Xmx3600m" which was running quite fine. After upgrading the JVM
>> to
>> > *java 6* we noticed a few times that our application fell idle (perhaps
>> > hung). It was not serving the requests although the port was open. I
>> changed
>> > the "-Xmx" from 3600m to 5000m.
>> >  Currently it is running OK but this created a curiosity in my mind that
>> how
>> > to find the right memory size for a lucene based application (or any java
>> > application). I believe it heavily depends on index size but how do I
>> > calculate it (at least approximately) without hit & trial?
>> >
>> > The details about my application and server are following
>> >
>> > *[ system configuration ]*
>> > # uname -a
>> > Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007
>> x86_64
>> > x86_64 x86_64 GNU/Linux
>> >
>> > *[ java version ]*
>> > # /usr/lib/jre1.6.0_20/bin/java -version
>> > java version "1.6.0_20"
>> > Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
>> >
>> > *[ application command line ]*
>> > # java -Xmx5000m -server -classpath
>> > lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
>> > ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer
>> conf/search.conf
>> >
>> > *[ memory information ]*
>> > *
>> > # cat /proc/meminfo
>> > MemTotal:      8156660 kB
>> > MemFree:         46924 kB
>> > Buffers:         53768 kB
>> > Cached:        3194224 kB
>> > SwapCached:         60 kB
>> > Active:        6086272 kB
>> > Inactive:       919276 kB
>> > SwapTotal:     2096472 kB
>> > SwapFree:      2095432 kB
>> > Dirty:            1148 kB
>> > Writeback:           0 kB
>> > AnonPages:     3756496 kB
>> > Mapped:          24800 kB
>> > Slab:          1058124 kB
>> > SReclaimable:    15144 kB
>> > SUnreclaim:    1042980 kB
>> > PageTables:      12232 kB
>> > NFS_Unstable:        0 kB
>> > Bounce:              0 kB
>> > CommitLimit:   6174800 kB
>> > Committed_AS:  4588900 kB
>> > VmallocTotal: 34359738367 kB
>> > VmallocUsed:    264904 kB
>> > VmallocChunk: 34359473387 kB
>> >
>> > More info*
>> > Search Server application is running on a few servers each of which
>> contains
>> > same of copy of index.
>> > Total size of index: 23 GB
>> > Total Documents in index: 18,000,000
>> > Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
>> > "contents" is the largest one, created from a text of as large as 5 - 8
>> KB)
>> > Query frequency: 1 query per second per server (in peak hours)
>> > Queries are taking around 800 - 1000 ms per query
>> >
>> > *Other memory related things in Search application*
>> > There is really nothing else which will consume noticeable memory. No
>> > database connection is made. The application simply returns the IDs from
>> the
>> > index based on the query.
>> >
>> > Any help on choosing right memory option is much appreciated.
>> >
>> > --
>> > Regards,
>> > Samar
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Regards,
> Samar
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message