lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fornoville, Tom" <Tom.Fornovi...@truvo.com>
Subject RE: Right memory for search application
Date Tue, 27 Apr 2010 14:00:33 GMT
Samarendra,

In regard to point #2, the GC should indeed handle the clean-up and that
might explain the "idle" time with your original configuration during
major collections. 

Have you checked that your machine is correctly identified as a server
and has optimized GC settings?

More info in the excellent article here:
http://www.lucidimagination.com/blog/2009/09/19/java-garbage-collection-
boot-camp-draft/

Regards,
Tom

-----Original Message-----
From: Samarendra Pratap [mailto:samarzone@gmail.com] 
Sent: dinsdag 27 april 2010 15:50
To: java-user@lucene.apache.org
Subject: Re: Right memory for search application

Hi Ian. Thanks for the points

Here are my answers -

1. Our default option is sort by score, however almost 8% of searches
use
sorting on a field (yyyymmddHHMMSS). This field is indexed as string
(not as
NumericField or DateField).

2. We are opening readers at the time of starting the application, but
Searchers are opened for every request choosing the right readers. A few
days back I got an answer on this very forum that reopening a searcher
is
not as big issue as reopening a reader. However we are *not closing
Searcher
*s after request is served. Should that cause memory problem? Shouldn't
GC
handle that all as the reference of Searcher becomes inaccessible?

3. I can't do a profiling on production servers but running "top" shows
that
immediately starting search server resident memory (RES) starts from
500m
and Virtual memory (VIRT) is around 5400m. It grows to RES => 1g in 5
minutes and 4g within half an hour while VIRT remains same.

4. Right! I'll be doing a memory profiling. I hope I'll get some hints
from
there too.

Do above points (specially # 2) indicate towards anything which I can do
besides profiling?



On Tue, Apr 27, 2010 at 6:53 PM, Ian Lea <ian.lea@gmail.com> wrote:

> There is no simple answer.  However your app does sound to be using
> rather a lot of memory for what you describe as simple searches.
>
> Are you using lucene sorting?  That can use lots of memory.  How are
> you using/reusing searchers/readers?  Having multiple ones open, or
> failing to close old ones, will use more memory.  Does memory usage
> grow then stabilize or keep on growing?
>
> A memory profiler/heap dump could tell you what is really using all
the
> space.
>
>
> --
> Ian.
>
> On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap
<samarzone@gmail.com>
> wrote:
> > Hi.
> >  I am searching for some guidance on right memory options for my
Search
> > Server application. How much memory a lucene based application
should be
> > given?
> >
> >  Till a few days back I was running my search server on java 1.4
with
> memory
> > options "-Xmx3600m" which was running quite fine. After upgrading
the JVM
> to
> > *java 6* we noticed a few times that our application fell idle
(perhaps
> > hung). It was not serving the requests although the port was open. I
> changed
> > the "-Xmx" from 3600m to 5000m.
> >  Currently it is running OK but this created a curiosity in my mind
that
> how
> > to find the right memory size for a lucene based application (or any
java
> > application). I believe it heavily depends on index size but how do
I
> > calculate it (at least approximately) without hit & trial?
> >
> > The details about my application and server are following
> >
> > *[ system configuration ]*
> > # uname -a
> > Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST
2007
> x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > *[ java version ]*
> > # /usr/lib/jre1.6.0_20/bin/java -version
> > java version "1.6.0_20"
> > Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> >
> > *[ application command line ]*
> > # java -Xmx5000m -server -classpath
> > lib/lucene-core-2.9.2.jar:lib/mysql-connector-j
> > ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer
> conf/search.conf
> >
> > *[ memory information ]*
> > *
> > # cat /proc/meminfo
> > MemTotal:      8156660 kB
> > MemFree:         46924 kB
> > Buffers:         53768 kB
> > Cached:        3194224 kB
> > SwapCached:         60 kB
> > Active:        6086272 kB
> > Inactive:       919276 kB
> > SwapTotal:     2096472 kB
> > SwapFree:      2095432 kB
> > Dirty:            1148 kB
> > Writeback:           0 kB
> > AnonPages:     3756496 kB
> > Mapped:          24800 kB
> > Slab:          1058124 kB
> > SReclaimable:    15144 kB
> > SUnreclaim:    1042980 kB
> > PageTables:      12232 kB
> > NFS_Unstable:        0 kB
> > Bounce:              0 kB
> > CommitLimit:   6174800 kB
> > Committed_AS:  4588900 kB
> > VmallocTotal: 34359738367 kB
> > VmallocUsed:    264904 kB
> > VmallocChunk: 34359473387 kB
> >
> > More info*
> > Search Server application is running on a few servers each of which
> contains
> > same of copy of index.
> > Total size of index: 23 GB
> > Total Documents in index: 18,000,000
> > Maximum fields per index: 56 (all analyzed, 4 small fields stored,
field
> > "contents" is the largest one, created from a text of as large as 5
- 8
> KB)
> > Query frequency: 1 query per second per server (in peak hours)
> > Queries are taking around 800 - 1000 ms per query
> >
> > *Other memory related things in Search application*
> > There is really nothing else which will consume noticeable memory.
No
> > database connection is made. The application simply returns the IDs
from
> the
> > index based on the query.
> >
> > Any help on choosing right memory option is much appreciated.
> >
> > --
> > Regards,
> > Samar
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Regards,
Samar

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message