Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <269467401.1214868765194.JavaMail.jira@brutus>
Date: Mon, 30 Jun 2008 16:32:45 -0700 (PDT)
From: "Brian Pinkerton (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Issue Comment Edited: (LUCENE-753) Use NIO positional read
 to avoid synchronization in FSIndexInput
In-Reply-To: <3846077.1166552421090.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609401#action_12609401 ] 

bripink edited comment on LUCENE-753 at 6/30/08 4:32 PM:
-----------------------------------------------------------------

I was curious about the discrepancy between the ChannelPread implementation and the SeparateFile implementation that Yonik saw.  At least on Mac OS X, the kernel implementation of read is virtually the same as pread, so there shouldn't be any appreciable performance difference unless the VM is doing something funny.  Sure enough, the implementations of read() under RandomAccessFile and read() under FileChannel are totally different.  The former relies on a buffer allocated either on the stack or by malloc, while the latter allocates a native buffer and copies the results to the original array.

Switching to a native buffer in the benchmark yields identical results for ChannelPread and SeparateFile on 1.5 and 1.6 on OS X.  I'm attaching an implementation of ChannelPreadDirect that uses a native buffer.

This may be a moot point because any implementation inside Lucene needs to consume a byte[] and not a ByteBuffer, but at least it's informative.


      was (Author: bripink):
    I was curious about the discrepancy between the ChannelPread implementation and the SeparateFile implementation that Yonik saw.  At least on Mac OS X, the kernel implementation of read is virtually the same as pread, so there shouldn't be any appreciable performance difference unless the VM is doing something funny.  Sure enough, the implementations of read() under RandomAccessFile and read() under FileChannel are totally different.  The former relies on either a buffer allocated either on the stack or by malloc, while the latter allocates a native buffer and copies the results to the original array.

Switching to a native buffer in the benchmark yields identical results for ChannelPread and SeparateFile on 1.5 and 1.6 on OS X.  I'm attaching an implementation of ChannelPreadDirect that uses a native buffer.

This may be a moot point because any implementation inside Lucene needs to consume a byte[] and not a ByteBuffer, but at least it's informative.


> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
>                 Key: LUCENE-753
>                 URL: https://issues.apache.org/jira/browse/LUCENE-753
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java, FSIndexInput.patch, FSIndexInput.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying file.
> This could mitigate any MT performance drop caused by reducing the number of files in the index format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org