lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tony Schwartz" <t...@simpleobjects.com>
Subject RE: OutOfMemoryError on addIndexes()
Date Thu, 18 Aug 2005 13:56:11 GMT
I think you're jumping into the conversation too late.  What you have said here does not
address the problem at hand.  That is, in TermInfosReader, all terms in the segment get
loaded into three very large arrays.  If your index is massive and has many fields
indexed (dates for example), you need huge amounts of memory to accommodate for it.  The
only way around this is to change lucene in such a way that it does not store all these
terms in memory.  It is possible that an implementation can be made that will be fast
for regular fields, but go to disk for fields that are "special" (i.e. dates) or some
such thing.

Tony Schwartz
tony@simpleobjects.com
"What we need is more cowbell."




> You can still have the complete date as a separate field, and sort or
> filter by it, just don't use this field in your query.
>
> Aviran
> http://www.aviransplace.com
>
> -----Original Message-----
> From: Tony Schwartz [mailto:tony@simpleobjects.com]
> Sent: Thursday, August 18, 2005 8:32 AM
> To: java-user@lucene.apache.org
> Subject: Re: OutOfMemoryError on addIndexes()
>
> Is this a viable solution?
> Doesn't this make sorting and filtering much more complex and much more
> expensive as well?
>
> Tony Schwartz
> tony@simpleobjects.com
>
>
>> On Wednesday 17 August 2005 22:49, Paul Elschot wrote:
>>> > the index could potentially be huge.
>>> >
>>> > So if this is indeed the case, it is a potential scalability
>>> > bottleneck in lucene index size.
>>>
>>> Splitting the date field into century, year in century, month, day,
>>> hour, seconds, and milliseconds will reduce the total number of
> indexed terms to 2300 or so.
>>
>> 1300 is closer, I forgot that I had split off the centuries.
>>
>> Regards,
>> Paul Elschot
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message