lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson" <erickerick...@gmail.com>
Subject Re: In memory MultiSearcher
Date Tue, 22 May 2007 23:34:21 GMT
You're right, I am suggesting that you use the Lucene
caching and see if it is adequate.

Mind you, I have no clue whether your application will be well
served by this or not, I've just seen too many examples of folks
(includeing me) jumping into a solution to a problem that doesn't
exist to be able to refrain from asking "Do you *know* you need
to get fancy?" <G>...

FWIW, a couple of things to watch out for. The searchers need to
process a few queries before the caches are built up, so if you
are doing load testing, be aware that the first few queries aren't
representative.

Second, you'll probably need to build at least a simple load tester
(actually, I'm probably sure some exist off the shelf, but we've
been doing things in-house so far) to fire off a bunch of threads
that make requests and measure response. I'd hate for my off-the-cuff
advice to cause your app to tip over the first time you opened it up
to the public <G>....

Best
Erick

On 5/22/07, Peter W. <peter@marketingbrokers.com> wrote:
>
> Erick,
>
> Thanks for the reply, this is a web application.
>
> If you want to serve image files in a scalable fashion
> on the Internet you make Apache serve them from
> memory, not the filesystem.
>
> For databases, some sites use a distributed object
> memory caching system such as memcached.
>
> I was hoping the idea translates to Lucene and
> was trying to overcome reading multiple indexes
> from attached disks into main memory on one
> machine first, then across a balanced farm.
>
> I think what you are saying is use FSDirectories and
> the regular built-in Lucene caching provided instead.
>
> Let's give that a try in the servlet init() method!
>
> Regards,
>
> Peter W.
>
>
> On May 21, 2007, at 2:46 PM, Erick Erickson wrote:
>
> > Why are you doing this in the first place? Do you actually have
> > evidence that the default Lucene behavior (caching, etc) is inadequate
> > for your needs?
> >
> > I'd *strongly* recommend, if you haven't, just using the regular
> > FSDirectories rather than RAMDirectories and only getting
> > complex if that's too slow...
> >
> > I ask because I am searching FS-based indexes that are 4G with
> > no problem. The index *was* 8G and had only a 10% performance hit.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message