lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian Lea" <ian....@gmail.com>
Subject Re: Improving Index Search Performance
Date Wed, 26 Mar 2008 14:51:54 GMT
Well, caching is designed to use memory.  If you are saying that you
haven't got enough memory to cache all your values then caching them
all isn't going to work, at any level. If you implemented your own
cache you could control memory usage with an LRU algorithm or whatever
made sense for your application.  We use an array as the cache with
the lucene document id as the index, so don't have to store document
vs field mapping.

I'm not aware of any way to tell lucene to cache documents, let alone
up to a user-supplied memory threshold.

If you are on a unix type OS and your lucene app is the only or main
thing on the machine, the OS is likely using the spare memory as cache
behind the scenes.  Don't know if the same applies to MS Win.


--
Ian.


On Wed, Mar 26, 2008 at 12:51 PM, Shailendra Mudgal
<mudgal.shailendra@gmail.com> wrote:
> > The bottom line is that reading fields from docs is expensive.
>  > FieldCache will, I believe, load fields for all documents but only
>  > once - so the second and subsequent times it will be fast.  Even
>  > without using a cache it is likely that things will speed up because
>  > of caching by the OS.
>
>
>  As i mentioned in my previous mail that the companyId is a multivalued
>  field, so caching it will consume a lot of memory. And this way we'll have
>  to keep the document vs field mapping also in the memory.
>
>
>
>  > If you've got plenty of memory vs index size you could look at
>  > RAMDirectory or MMapDirectory.  Or how about some solid state disks?
>  > Someone recently posted some very impressive performance stats.
>
>
>  The index size is around 20G and the available Memory is 4G so, keeping the
>  entire index into the memory  is not possible.   But as i mentioned earlier
>  that it is using only 1 G out of 4 G, so is their a way to specify the
>  lucene to cache more documents , say use 2G for caching the index ??
>
>  I'll appreciate more suggestions on the same problem.
>
>  Regards,
>  Vipin
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message