lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-2373) Change StandardTermsDictWriter to work with streaming and append-only filesystems
Date Tue, 06 Apr 2010 22:57:34 GMT
Change StandardTermsDictWriter to work with streaming and append-only filesystems
---------------------------------------------------------------------------------

                 Key: LUCENE-2373
                 URL: https://issues.apache.org/jira/browse/LUCENE-2373
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
            Reporter: Andrzej Bialecki 
             Fix For: 3.1


Since early 2.x times Lucene used a skip/seek/write trick to patch the length of the terms
dict into a place near the start of the output data file. This however made it impossible
to use Lucene with append-only filesystems such as HDFS.

In the post-flex trunk the following code in StandardTermsDictWriter initiates this:
{code}
    // Count indexed fields up front
    CodecUtil.writeHeader(out, CODEC_NAME, VERSION_CURRENT); 

    out.writeLong(0);                             // leave space for end index pointer
{code}
and completes this in close():
{code}
      out.seek(CodecUtil.headerLength(CODEC_NAME));
      out.writeLong(dirStart);
{code}

I propose to change this layout so that this pointer is stored simply at the end of the file.
It's always 8 bytes long, and we known the final length of the file from Directory, so it's
a single additional seek(length - 8) to read it, which is not much considering the benefits.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message