lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devon H. O'Dell" <devon.od...@gmail.com>
Subject Re: NIO2 Directory implementations
Date Sun, 17 Mar 2013 20:10:47 GMT
2013/3/17 Michael McCandless <lucene@mikemccandless.com>:
> Hi Michael (directly CC'd this time...),
>
> Maybe you're not subscribed to the list?  Your first email got some
> responses, eg:
>
>     http://lucene.markmail.org/thread/lrv7miivzmjm3ml5

Indeed, he's not, I didn't auto-subscribe him when putting his message
through to the list. Might be a good idea to subscribe; looks like
interesting and promising work :D

--dho

> Net/net, these new directory implementations sound exciting!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Mar 17, 2013 at 4:03 PM, Michael Poindexter
> <staticsnow@gmail.com> wrote:
>> As part of a project using Lucene I have implemented a trio of Directories
>> roughly corresponding to the FSDirectory implementations in core.  These
>> directory implementations use the NIO2 API's in JDK7 when opening files.
>>  This ensures that on Windows the files are opened in a mode that allows
>> deletion even if the file is open elsewhere.
>>
>> 1.) JDK7MMapDirectory - Roughly the same as MMapDirectory.  Uses
>> FileChannel.open (instead of RandomAccessFile) to create a FileChannel that
>> then has map() called on it to create the mapped buffers.
>> 2.) JDK7NIOFSDirectory - Roughly the same as NIOFSDirectory, but uses
>> FileChannel.open to create the file channel instead of RandomAccessFile.
>> 3.) JDK7AsyncFSDirectory - This one is new and different.  I needed a
>> replacement for SimpleFSDirectory (that was not susceptible to problems if
>> interrupt()'ed) and did not have the synchronization problems on Windows of
>> NIOFSDirectory.  This one is used where SimpleFSDirectory could have been
>> used, but uses an AsynchronousFileChannel to do it's work.  The actual
>> operation is still synchronous, but on Windows AsynchronousFileChannel uses
>> overlapped IO, and hence does not require synchronization on the position
>> and should be safe for interrupts.
>>
>> A couple of questions:
>> 1.)  Is there any interest in me contributing these to Lucene?  They
>> require JDK7+, but perhaps they could go in a contrib module?
>>
>> 2.)  While implementing these I noticed the implementation of
>> FSDirectory.sync seems a little strange:  It just opens a new
>> RandomAccessFile and forces a sync using it.  The JavaDocs seem to imply
>> that this would force a sync on the file handle associated with the
>> RandomAccessFile, but that's not the file handle that was written to as
>> part of an IndexOutput.  On Windows at least this won't matter, but it
>> seems theoretically wrong...i.e. according to the JavaDoc on a given
>> platform this style of operation could have no impact if I am understanding
>> it correctly.  It seems like maybe it would be better to have a sync() call
>> on an IndexOutput that can be called before closing it...am I missing
>> something here?
>>
>> 3.)  What is the best way to go about benchmarking/testing these new
>> implementations to compare against the core FSDirectory implementations?
>>  I've seen some references to randomized tests and benchmarks on the
>> developer pages on the Lucene website, but I didn't see anything that was
>> along the lines of "Here's how to run the benchmarks"...any pointers would
>> be much appreciated.
>>
>> Thanks,
>> Mike Poindexter
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message