lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Scale up design
Date Wed, 15 Dec 2010 11:06:16 GMT
On Wed, 2010-12-15 at 09:42 +0100, Ganesh wrote:
> What is the advantage of going for 64 Bit.

Larger maximum heap, more memory in the machine.

> People claim performance and usage of more RAM.

Yes, pointers normally take up 64bit on a 64bit machine. Depending on
the application, the overhead can be anything from practically
non-existing to close to 100%. You can set an option for the JVM to try
and use smaller pointers on 64bit machines. This limits the maximum
memory allocation in the JVM to 32GB, which seems like a fair compromise
at this point in time.
http://wikis.sun.com/display/HotSpotInternals/CompressedOops

> In 32 Bit OS, JVM handles 1 to 1.5 GB of RAM then in case
> of 64 Bit, Single JVM cannot use more than 1.5 GB RAM?

Say what? When running on a 64bit, the JVM heap limit is normally the
system's per-process memory limit. For Linux this is generally well
above any real world hardware. For Windows it seems like you need to
enable something:
http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx
(note: I have no experience with 64bit Windows)

> Please help me with some more ideas. We need to design whether
> to scale out or scale up.

Maybe you could describe your vision in more detail? What scale are you
looking at? How large is your index in GB, how many documents, how fast
do you need the searcher to respond, are you doing any sorting or
faceting (and do you facet on a few unique values or things like title
or author)?

It makes little sense to try and get a single machine to handle billions
of documents with large faceting, but it seems silly to distribute 10GB
of index with 1 million documents. As a general rule of thumb. As always
your mileage might wary.

For the record, our current index is 40GB/9 million records. We're doing
sorting on title and faceting on 15 fields, out of which 2 has 4-6
million unique values. This runs on a single machine (okay, 2, but they
are mirrored) with 6GB of RAM and it works fine with sub-second response
times (normally <300ms AFAIR). Our experimental setup can get by with
1.2GB and would thus not require 64bit.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message