lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Right memory for search application
Date Tue, 27 Apr 2010 13:23:25 GMT
There is no simple answer.  However your app does sound to be using
rather a lot of memory for what you describe as simple searches.

Are you using lucene sorting?  That can use lots of memory.  How are
you using/reusing searchers/readers?  Having multiple ones open, or
failing to close old ones, will use more memory.  Does memory usage
grow then stabilize or keep on growing?

A memory profiler/heap dump could tell you what is really using all the space.


--
Ian.

On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap <samarzone@gmail.com> wrote:
> Hi.
>  I am searching for some guidance on right memory options for my Search
> Server application. How much memory a lucene based application should be
> given?
>
>  Till a few days back I was running my search server on java 1.4 with memory
> options "-Xmx3600m" which was running quite fine. After upgrading the JVM to
> *java 6* we noticed a few times that our application fell idle (perhaps
> hung). It was not serving the requests although the port was open. I changed
> the "-Xmx" from 3600m to 5000m.
>  Currently it is running OK but this created a curiosity in my mind that how
> to find the right memory size for a lucene based application (or any java
> application). I believe it heavily depends on index size but how do I
> calculate it (at least approximately) without hit & trial?
>
> The details about my application and server are following
>
> *[ system configuration ]*
> # uname -a
> Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007 x86_64
> x86_64 x86_64 GNU/Linux
>
> *[ java version ]*
> # /usr/lib/jre1.6.0_20/bin/java -version
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
>
> *[ application command line ]*
> # java -Xmx5000m -server -classpath
> lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
> ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer conf/search.conf
>
> *[ memory information ]*
> *
> # cat /proc/meminfo
> MemTotal:      8156660 kB
> MemFree:         46924 kB
> Buffers:         53768 kB
> Cached:        3194224 kB
> SwapCached:         60 kB
> Active:        6086272 kB
> Inactive:       919276 kB
> SwapTotal:     2096472 kB
> SwapFree:      2095432 kB
> Dirty:            1148 kB
> Writeback:           0 kB
> AnonPages:     3756496 kB
> Mapped:          24800 kB
> Slab:          1058124 kB
> SReclaimable:    15144 kB
> SUnreclaim:    1042980 kB
> PageTables:      12232 kB
> NFS_Unstable:        0 kB
> Bounce:              0 kB
> CommitLimit:   6174800 kB
> Committed_AS:  4588900 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed:    264904 kB
> VmallocChunk: 34359473387 kB
>
> More info*
> Search Server application is running on a few servers each of which contains
> same of copy of index.
> Total size of index: 23 GB
> Total Documents in index: 18,000,000
> Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
> "contents" is the largest one, created from a text of as large as 5 - 8 KB)
> Query frequency: 1 query per second per server (in peak hours)
> Queries are taking around 800 - 1000 ms per query
>
> *Other memory related things in Search application*
> There is really nothing else which will consume noticeable memory. No
> database connection is made. The application simply returns the IDs from the
> index based on the query.
>
> Any help on choosing right memory option is much appreciated.
>
> --
> Regards,
> Samar
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message