lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: Lucene RAM buffer size limit
Date Mon, 26 Apr 2010 14:24:26 GMT
Good point. I'll set the heap size larger.

It will be interesting to measure if there is a noticeable improvement from
2GB RAM buffer size vs say 100MB or 512MB ... I'll report the results here
when I have them

But in the meantime, I think, unless it makes absolutely no sense and does
not improve anything, that we should not limit to 2GB just because we use an
int somewhere.

Shai

On Mon, Apr 26, 2010 at 5:18 PM, Mark Miller <markrmiller@gmail.com> wrote:

> On 4/26/10 10:06 AM, Shai Erera wrote:
>
>> Hi
>>
>> Today we limit the RAM buffer size to 2048. I was wondering why, until
>> Mark referenced me to that issue
>> (https://issues.apache.org/jira/browse/LUCENE-1995) where someone
>> complained about it as well, but I didn't find a reason to why it's
>> limited. I mean, I understand it's probably an int is used somewhere,
>> but all the issue says here
>> (
>> https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757
>> <
>> https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757
>> >)
>> is "Maybe someday Lucene will allow a larger RAM buffer than 2GB".
>>
>> Why "maybe"? :)
>>
>> I set up a benchmark run on a new machine I got, 4xQuad, 64GB RAM, and
>> turns out I can't really take advantage of all that RAM ... :).
>>
>> So I don't mind to look at the code, but I guess since Mike and a bunch
>> of other smarter people were on that issue, there is probably something
>> I need to know before I set to change it?
>>
>> Appreciate your thoughts on that. Today, more and more machines come w/
>> more than 2GB ...
>>
>> Shai
>>
>
> Keep in mind that you need a lot more than 2GB of ram to use a really large
> RAM buffer:
> http://search.lucidimagination.com/search/document/b18fbeaaf53ead6c/io_exception_during_merge_optimize
>
> 64 GB might cover it though ;)
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Mime
View raw message