lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terry Steichen" <>
Subject Re: Lucene external field storage contribution.
Date Wed, 10 Nov 2004 01:48:30 GMT

Sorry for the delay in replying.  I think your idea for an external field storage mechanism
is excellent.  I'd love to see it, and if I can, will be willing to help make that happen.


  ----- Original Message ----- 
  From: Kevin A. Burton 
  To: Lucene Users List 
  Sent: Sunday, November 07, 2004 4:47 PM
  Subject: Lucene external field storage contribution.

  About 3 months ago I developed a external storage engine which ties into 

  I'd like to discuss making a contribution so that this is integrated 
  into a future version of Lucene.

  I'm going to paste my original PROPOSAL in this email. 

  There wasn't a ton of feedback first time around but I figure squeaky 
  wheel gets the grease...

  >> I created this proposal because we need this fixed at work. I want to 
  >> go ahead and work on a vertical fix for our version of lucene and then 
  >> submit this back to Jakarta.
  >> There seems to be a lot of interest here and I wanted to get feedback 
  >> from the list before moving forward ...
  >> Should I put this in the wiki?!
  >> Kevin
  >> ** OVERVIEW **
  >> Currently Lucene supports 'stored fields; where the content of these 
  >> fields are
  >> kept within the lucene index for use in the future.
  >> While acceptable for small indexes, larger amounts of stored fields 
  >> prevent:
  >> - Fast index merges since the full content has to be continually merged.
  >> - Storing the indexes in memory (since a LOT of memory would be 
  >> required and
  >> this is cost prohibitive)
  >> - Fast queries since block caching can't be used on the index data.
  >> For example in our current setup our index size is 20G.  Nearly 90% of 
  >> this is
  >> content.  If we could store the content outside of Lucene our merges and
  >> searches would be MUCH faster.  If we could store the index in MEMORY 
  >> this could
  >> be orders of magnitude faster.
  >> ** PROPOSAL **
  >> Provide an external field storage mechanism which supports legacy indexes
  >> without modification.  Content is stored in a "content segment". The only
  >> changes would be a field with 3(or 4 if checksum enabled) values.
  >>       Logical ID of the content segment.  This is an integer value.  
  >> There is
  >>       a global Lucene property named CS_ROOT which stores all the 
  >> content.
  >>       The segments are just flat files with pointers.  Segments are 
  >> broken
  >>       into logical pieces by time and size.  Usually 100M of content 
  >> would be
  >>       in one segment.
  >> - CS_OFFSET
  >>       The byte offset of the field.
  >> - CS_LENGTH
  >>       The length of the field.
  >>       Optional checksum to verify that the content is correct when 
  >> fetched
  >>       from the index.
  >> - The field value here would be exactly 'N:O:L' where N is the segment 
  >> number,
  >>   O is the offset, and L is the length.  O and L are 64bit values.  N 
  >> is a 32
  >>   bit value (though 64bit wouldn't really hurt).
  >> This mechanism allows for the external storage of any named field.
  >> CS_OFFSET, and CS_LENGTH allow use with RandomAccessFile and new NIO 
  >> code for
  >> efficient content lookup.  (Though filehandle caching should probably 
  >> be used).
  >> Since content is broken into logical 100M segments the underlying 
  >> filesystem can
  >> orgnize the file into contiguous blocks for efficient non-fragmented 
  >> lookup.
  >> File manipulation is easy and indexes can be merged by simply 
  >> concatenating the
  >> second file to the end of the first.  (Though the segment, offset, and 
  >> length
  >> need to be updated).  (FIXME: I think I need to think about this more 
  >> since I
  >> will have < 100M per syncs)
  >> Supporting full unicode is important.  Full java.lang.String storage 
  >> is used
  >> with String.getBytes() so we should be able to avoid unicode issues.  
  >> If Java
  >> has a correct java.lang.String representation it's possible easily add 
  >> unicode
  >> support just by serializing the byte representation. (Note that the 
  >> JDK says
  >> that the DEFAULT system char encoding is used so if this is ever 
  >> changed it
  >> might break the index)
  >> While Linux and modern versions of Windows (not sure about OSX) 
  >> support 64bit
  >> filesystems the 4G storage boundary of 32bit filesystems (ext2 is an 
  >> example)
  >> are an issue.  Using smaller indexes can prevent this but eventually 
  >> segment
  >> lookup in the filesystem will be slow.  This will only happen within 
  >> terabyte
  >> storage systems so hopefully the developer has migrated to another 
  >> (modern)
  >> filesystem such as XFS.
  >> ** FEATURES **
  >>   - Must be able to replicate indexes easily to other hosts.
  >>   - Adding content to the index must be CHEAP
  >>   - Deletes need to be cheap (these are cheap for older content.  Just 
  >> discard
  >>     older indexes)
  >>   - Filesystem needs to be able to optimize storage
  >>   - Must support UNICODE and binary content (images, blobs, byte arrays,
  >>     serialized objects, etc)
  >>   - Filesystem metadata operations should be fast.  Since content is 
  >> kept in
  >>     LARGE indexes this is easy to avoid.
  >>   - Migration to the new system from legacy indexes should be fast and
  >>     painless for future developers


  Use Rojo (RSS/Atom aggregator).  Visit Ask me for an 
  invite!  Also see #rojo if you want to chat.

  Rojo is Hiring! -

  If you're interested in RSS, Weblogs, Social Networking, etc... then you 
  should work for Rojo!  If you recommend someone and we hire them you'll 
  get a free iPod!
  Kevin A. Burton, Location - San Francisco, CA
         AIM/YIM - sfburtonator,  Web -
  GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message