lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Date Mon, 30 Jun 2008 22:14:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609383#action_12609383
] 

Michael McCandless commented on LUCENE-753:
-------------------------------------------

OK I re-ran only PooledPread, SeparateFile and ChannelPread since they
are the "leading contenders" on all platforms.

Also, I changed to serial=false.

Now the results are very close on all but windows, but on windows I'm
seeing the opposite of what Yonik saw: PooledPread is slowest, and
SeparateFile is fastest.  But this is a laptop (Win XP Pro), and it is
JRE 1.4.  Also I ran with pool size == number of threads == 4.


Mac OS X 10.5.3, single WD Velociraptor hard drive, Sun JRE 1.6.0_05

{code}
config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=120807, MB/sec=222.20190551871994

config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830326, ms=119641, MB/sec=224.36744594244448

config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=119217, MB/sec=225.1654176837196
{code}


Linux 2.6.22.1, 6-drive RAID 5 array, Sun JRE 1.6.0_06

{code}
config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=52613, MB/sec=510.2074696367818

config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=52715, MB/sec=509.22025230010433

config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=53792, MB/sec=499.0248661511005
{code}


WIN XP PRO, laptop, Sun JRE 1.4.2_15:

{code}
config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=209956, MB/sec=127.85319590771401

config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=89101, MB/sec=301.27098012367986

config: impl=ChannelPread serial=false nThreads=4 iterations=100 bufsize=1024 poolsize=4 filelen=67108864
answer=-23830370, ms=184087, MB/sec=145.81988733587923
{code}

> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
>                 Key: LUCENE-753
>                 URL: https://issues.apache.org/jira/browse/LUCENE-753
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java,
FileReadTest.java, FileReadTest.java, FSIndexInput.patch, FSIndexInput.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying
file.
> This could mitigate any MT performance drop caused by reducing the number of files in
the index format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message