lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samarendra Pratap <samarz...@gmail.com>
Subject Right memory for search application
Date Tue, 27 Apr 2010 12:51:37 GMT
Hi.
 I am searching for some guidance on right memory options for my Search
Server application. How much memory a lucene based application should be
given?

 Till a few days back I was running my search server on java 1.4 with memory
options "-Xmx3600m" which was running quite fine. After upgrading the JVM to
*java 6* we noticed a few times that our application fell idle (perhaps
hung). It was not serving the requests although the port was open. I changed
the "-Xmx" from 3600m to 5000m.
 Currently it is running OK but this created a curiosity in my mind that how
to find the right memory size for a lucene based application (or any java
application). I believe it heavily depends on index size but how do I
calculate it (at least approximately) without hit & trial?

The details about my application and server are following

*[ system configuration ]*
# uname -a
Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007 x86_64
x86_64 x86_64 GNU/Linux

*[ java version ]*
# /usr/lib/jre1.6.0_20/bin/java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

*[ application command line ]*
# java -Xmx5000m -server -classpath
lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer conf/search.conf

*[ memory information ]*
*
# cat /proc/meminfo
MemTotal:      8156660 kB
MemFree:         46924 kB
Buffers:         53768 kB
Cached:        3194224 kB
SwapCached:         60 kB
Active:        6086272 kB
Inactive:       919276 kB
SwapTotal:     2096472 kB
SwapFree:      2095432 kB
Dirty:            1148 kB
Writeback:           0 kB
AnonPages:     3756496 kB
Mapped:          24800 kB
Slab:          1058124 kB
SReclaimable:    15144 kB
SUnreclaim:    1042980 kB
PageTables:      12232 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   6174800 kB
Committed_AS:  4588900 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    264904 kB
VmallocChunk: 34359473387 kB

More info*
Search Server application is running on a few servers each of which contains
same of copy of index.
Total size of index: 23 GB
Total Documents in index: 18,000,000
Maximum fields per index: 56 (all analyzed, 4 small fields stored, field
"contents" is the largest one, created from a text of as large as 5 - 8 KB)
Query frequency: 1 query per second per server (in peak hours)
Queries are taking around 800 - 1000 ms per query

*Other memory related things in Search application*
There is really nothing else which will consume noticeable memory. No
database connection is made. The application simply returns the IDs from the
index based on the query.

Any help on choosing right memory option is much appreciated.

-- 
Regards,
Samar

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message