lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Stewart <Robert.Stew...@INFONGEN.COM>
Subject RE: Question regarding sorting and memory consumption in lucene
Date Fri, 10 Oct 2008 15:57:19 GMT
I have had a similar problem. What I do is load all the date field values at index startup,
convert dates (timestamps) to a Julian date (# of seconds since 1970/1/1). Then I pre-sort
that array using a very fast O(n) distribution sort, and then keep an array of integers which
is the pre-sorted permutation of all documents in the index. So that, for docid=N, perm[N]=sorted
order.  Then it just takes enumerating docids in results (from a bitarray) to get the sorted
order of results.  Our index is approx. 38million docs. Sorting by date is around 20ms.

-----Original Message-----
From: Aleksander M. Stensby []
Sent: Friday, October 10, 2008 11:45 AM
Subject: Re: Question regarding sorting and memory consumption in lucene

That's a really good idea Mark! :)
Thanks! Will try to see if can make a quick change with your suggestion.
(Too bad quick isn't really a word in my vocabulary when it's 6 o'clock on
a Friday :(
Guess it'll be a looong night.. :(


On Fri, 10 Oct 2008 17:07:31 +0200, mark harwood <>

> Update: The statement "...cost is field size (10 bytes ?) times number
> of documents" is wrong.
> What you actually have is the cost of the unique strings  (estimated at
> 10 * 1460  -effectively nothing) BUT you have to add the cost of the
> array of object references to those strings so
>        30m x 8 bytes on 64bit java = 240mb
> or
>     30m x 4 bytes on 32bit   = 120mb
> ....which is where the bulk of the cost comes in.
> How about using a field cache of "short" which is effectively:
>      new short[reader.maxDoc]
> or
>         2bytes * 30 million = 60 meg.
> Each short could represent up to 65536 values - capable of representing
> a date range of 179 years.
> ----- Original Message ----
> From: mark harwood <>
> To:
> Sent: Friday, 10 October, 2008 15:43:35
> Subject: Re: Question regarding sorting and memory consumption in lucene
> I think you have your memory cost calculation wrong.
> The cost is field size (10 bytes ?) times number of documents NOT number
> of unique terms.
> The cache is essentially an array of  size reader.maxDoc() which is
> indexed directly into on docId to retrieve field values.
> You are right in needing to factor in the cost of keeping one active
> cache while busy warming-up a new one so that effectively doubles the
> RAM requirements.
> ----- Original Message ----
> From: Aleksander M. Stensby <>
> To:
> Sent: Friday, 10 October, 2008 15:25:29
> Subject: Re: Question regarding sorting and memory consumption in lucene
> Unfortunately no, since the documents that are added may come form a new
> "source" containing old documents aswell..:/
> I tried deploying our webapplication without any searcher objects and it
> consumes basically ~200mb of memory in tomcat.
> With 6 searchers the same applications manages to consume over 2.5 GB of
> memory when warming... :(
> I might have done some super-idiotic logic in the way I handle searching,
> but I can seriously not see what that might be...
> But I assume that people deal with much larger indexes than this, right?
> cheers,
>   Aleksander
> On Fri, 10 Oct 2008 15:18:46 +0200, mark harwood
> <>
> wrote:
>> Assuming content is added in chronological order and with no updates to
>> existing docs couldn't you rely on internal Lucene document id to give a
>> chronological sort order?
>> That would require no memory cache at all when sorting.
>> Querying across multiple indexes simultaneously however may present an
>> added complication...
>> ----- Original Message ----
>> From: Aleksander M. Stensby <>
>> To:
>> Sent: Friday, 10 October, 2008 13:51:50
>> Subject: Re: Question regarding sorting and memory consumption in lucene
>> I'll follow up on my own question...
>> Let's say that we have 4 years of data, meaning that there will be
>> roughly
>> 4 * 365 = 1460 unique terms for our sort field.
>> For one index, lets say with 30 million docs, the cache should use
>> approx
>> 100mb, or am I wrong? and thus for 6 indexes we would need approx 600 mb
>> for the caches? (and an additional 100mb every time we warm a new
>> searcher
>> and swap it out...) As far as the string versus int or long goes, I
>> don't
>> really see any big gain in changig it since 1460 * 10  bytes extra
>> memory
>> doesnt really make much difference. Or?
>> I guess we should consider reducing the index size or at least only
>> allow
>> sorted search on a subset of the index (or a pruned version of the
>> index...) ? Would that be better for us?
>> But then again, I assume that there are much larger lucene-based indexes
>> out there than ours, and you guys must have some solution to this issue,
>> right? :)
>> best regards,
>>   Aleksander
>> On Fri, 10 Oct 2008 14:09:36 +0200, Aleksander M. Stensby
>> <> wrote:
>>> Hello, I've read a lot of threads now on memory consumption and
>>> sorting,
>>> and I think I have a pretty good understanding of how things work, but
>>> I
>>> could still need some input here..
>>> We currently have a system consisting of 6 different lucene indexes
>>> (all
>>> have the same structure, so you could say it is a form of sharding). We
>>> currently use this approach because we want to be able to give users
>>> access to different index (but not necessarily  all indexes) etc.
>>> (We are planning to move to a solr-based system, but for now we would
>>> like to solve this issue with our current lucene-based system.)
>>> The thing is, the indexes are rather big (ranging from 5G to 20G per
>>> index and 10 - 30 million entries per index.)
>>> We keep one searcher object open per index, and when the index is
>>> changed (new documents added in batches several times a day), we update
>>> the searcher objects.
>>> In the warmup procedure we did a couple of searches and things work
>>> fine, BUT i realized that in our application we return hits sorted by
>>> date by default, and our warmup procedure did non-sorted queries... so
>>> still the first searches done by the user after an update was slow
>>> (obviously).
>>> To cope, I changed the warmup procedure to include a sorted search, and
>>> now the user will not notice slow queries. Good!
>>> But, the problem at hand is that we are running into memory problems
>>> (and I understand that sorting does consume a lot of memory...) But is
>>> there any way that is "best practice" to deal with this? The field we
>>> sort on is an un_indexed text field representing the date. typically
>>> "2008-10-10". I am aware that string field sorting consumes a lot of
>>> memory, so should we change this field to something different? Would
>>> this help us with the memory problems?
>>> As a sidenote / couriosity question: Does it matter if we use the
>>> search
>>> method returning Hits versus the search method returning TopFieldDocs?
>>> (we are not iterating them in any way when this memory issue occurs)
>>> Thanks in advance for any guidance we may get.
>>> Best regards,
>>>   Aleksander M. Stensby

Aleksander M. Stensby
Senior Software Developer
Integrasco A/S
+47 41 22 82 72

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message