Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <7394069.1164156663160.JavaMail.jira@brutus>
Date: Tue, 21 Nov 2006 16:51:03 -0800 (PST)
From: "Chuck Williams (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Updated: (LUCENE-709) [PATCH] Enable application-level
 management of IndexWriter.ramDirectory size
In-Reply-To: <26596647.1163138077092.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

     [ http://issues.apache.org/jira/browse/LUCENE-709?page=all ]

Chuck Williams updated LUCENE-709:
----------------------------------

    Attachment: ramDirSizeManagement.patch

This one should be golden as it addresses all the issues that have been raised and I believe the syncrhonization is fairly well optimized.

Size is now computed based on buffer size, and so is a more accurate accounting of actual memory usage.

I've added all the various checking and FileNotFoundExceptions that Doug suggested.

I've also changed RamFile.buffers to an ArrayList per Yonik's last suggestion.  This is probably better than cosmetic since it does allow some unnecessary syncrhonization to be eliminated.

Unfortunately, my local Lucene differs now fairly substantially from the head -- wish you guys would commit more of my patches so merging wasn't so difficult :-) -- so I'm not using the version submitted here, but I did merge it into the head carefully and all tests pass, including the new RAMDIrectory tests specifically for the functionality this patch provides.


> [PATCH] Enable application-level management of IndexWriter.ramDirectory size
> ----------------------------------------------------------------------------
>
>                 Key: LUCENE-709
>                 URL: http://issues.apache.org/jira/browse/LUCENE-709
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.0.1
>         Environment: All
>            Reporter: Chuck Williams
>         Attachments: ramdir.patch, ramdir.patch, ramDirSizeManagement.patch, ramDirSizeManagement.patch, ramDirSizeManagement.patch, ramDirSizeManagement.patch
>
>
> IndexWriter currently only supports bounding of in the in-memory index cache using maxBufferedDocs, which limits it to a fixed number of documents.  When document sizes vary substantially, especially when documents cannot be truncated, this leads either to inefficiencies from a too-small value or OutOfMemoryErrors from a too large value.
> This simple patch exposes IndexWriter.flushRamSegments(), and provides access to size information about IndexWriter.ramDirectory so that an application can manage this based on total number of bytes consumed by the in-memory cache, thereby allow a larger number of smaller documents or a smaller number of larger documents.  This can lead to much better performance while elimianting the possibility of OutOfMemoryErrors.
> The actual job of managing to a size constraint, or any other constraint, is left up the applicatation.
> The addition of synchronized to flushRamSegments() is only for safety of an external call.  It has no significant effect on internal calls since they all come from a sychronized caller.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org