lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: fieldcache gives OOM. Deos a LRU-style fieldcache exist?
Date Mon, 12 Nov 2007 15:58:00 GMT
https://issues.apache.org/jira/browse/LUCENE-831

This patch should be a good start for you. Not sure how stale it is at 
this point though.

Britske wrote:
> First my question: 
>
> Is there an (experimental / patch-version) lucene-fieldcache available which
> uses some kind of eviction-strategy (LRU or whatever) so that OOM's would
> never happen in my case, but instead some sorts would simply get evicted?
>
> Now the background for those who are interested: 
>
> I'm having a usecase in which products can have a lot of variants. (Up to
> 10.000) 
> These variants can each have their own price (Which I need to sort and
> filter by ) and a bunch of stored values (which I just need to display to
> the client). At all times only 1 variant per product needs to be shown in a
> search-result. I've modeled this by letting each possible variant have 2
> fields: 1 pricefield and 1 storagefield. This means that every entry in the
> index is a product with possible more than 20.000 fields. Potentially:
> because the actual collection of available variants is sparse per product.
> This all sounds a bit exotic I know, but uptil I was pointed to this problem
> everything went fine. (index-time is a 'bit' slow but that's no real
> problem.)
>
> The problem is that potentially all pricefields of all variants can be used
> for sorting, meaning that the fieldcache would have to contain up to 10.000
> cached lists. This will give OutOfMemory-errors sooner or later, (products
> will grow to about 400.000, so the fieldcache will grow to about 16GB) which
> is not what my evironment will be able to handle any time soon.
>
> Thanks in advance, 
>
> Geert-Jan
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message