Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Mon, 22 Aug 2011 16:23:29 +0000 (UTC)
From: "Simon Willnauer (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <749730.1561.1314030209152.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <109535752.19982.1308556427501.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (LUCENE-3218) Make CFS appendable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/LUCENE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simon Willnauer updated LUCENE-3218:
------------------------------------

    Attachment: LUCENE-3218.patch

here is a rough patch against trunk that detaches CFDirectory from other dir impls but still extends Directory. all optimizations still remain while always the top-level directory is used to create / open the CFS. 
I didn't follow uwes last approach since I could figure out when to close the RAF reference since like in the MMap case we want to forcefully unmap the file and therefor would also need to close the RAF reference in the base stream. I use a helper construct (IndexInputHandle - yes need to find a better name) that can only be pulled from another directory (protected). So this is private to the directory impls but solves our cases here nicely I think.

still rough but its a start.

> Make CFS appendable  
> ---------------------
>
>                 Key: LUCENE-3218
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3218
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.4, 4.0
>            Reporter: Simon Willnauer
>            Priority: Blocker
>             Fix For: 3.4, 4.0
>
>         Attachments: LUCENE-3218.patch, LUCENE-3218.patch, LUCENE-3218.patch, LUCENE-3218.patch, LUCENE-3218.patch, LUCENE-3218_3x.patch, LUCENE-3218_test_fix.patch, LUCENE-3218_tests.patch
>
>
> Currently CFS is created once all files are written during a flush / merge. Once on disk the files are copied into the CFS format which is basically a unnecessary for some of the files. We can at any time write at least one file directly into the CFS which can save a reasonable amount of IO. For instance stored fields could be written directly during indexing and during a Codec Flush one of the written files can be appended directly. This optimization is a nice sideeffect for lucene indexing itself but more important for DocValues and LUCENE-3216 we could transparently pack per field files into a single file only for docvalues without changing any code once LUCENE-3216 is resolved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org