incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marvin Humphrey (JIRA)" <>
Subject [jira] Updated: (LUCY-75) CompoundFileWriter
Date Wed, 25 Nov 2009 01:54:39 GMT


Marvin Humphrey updated LUCY-75:

    Attachment: cf_writer_support.diff

CompoundFileWriter consolidates files within a directory into a combinded
"cf.dat" file, storing their file metadata within a "cfmeta.json" file.
Nested subdirectories and files ending in ".json" are excluded.  

The ".json" exclusion policy is necessary because we want our "segmeta.json"
files to be visible when browsing the index; it causes no problems with file
descriptor limits because JSON files tend to be slurped rather than held open.

Each sub-file within the cf.dat file begins on an offset which is a multiple
of 8, allowing memory-mapped regions to be cast to any type of pointer without
memory alignment problems.

The Lucene version of CompoundFileWriter lives underneath
org.apache.lucene.index and operates on a logical index segment; in contrast,
this version lives under Lucy::Store and operates on any subdirectory,
regardless of what files the subdirectory contains.

> CompoundFileWriter
> ------------------
>                 Key: LUCY-75
>                 URL:
>             Project: Lucy
>          Issue Type: Sub-task
>          Components: Core
>            Reporter: Marvin Humphrey
>            Assignee: Marvin Humphrey
>         Attachments: cf_writer_support.diff, CompoundFileWriter.bp, CompoundFileWriter.c
> Consolidate files within a subdirectory to save file descriptors.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message