lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 19:13:07 GMT
Its the same test file - everything the same except one file is on local
ext4 hd and the copy is on ramdisk.

I havn't yet looked into what the answer corresponds to. I wonder if the
RAM disk is getting made as ext3?

note:also I give the JVM RAM a bit larger than the file size, and the OS
has plenty of RAM beyond the file size.

Here are two runs on the hardrive/cached side (not ramdisk). Again, a
bit slower than the RAM disk:

config: impl=ChannelPread serial=false nThreads=4 iterations=100
bufsize=1024 poolsize=2 filelen=730554368
answer=-282295361, ms=766340, MB/sec=381.3212767179059


config: impl=ClassicFile serial=false nThreads=4 iterations=100
bufsize=1024 poolsize=2 filelen=730554368
answer=-282295361, ms=852010, MB/sec=342.9792457835002



Yonik Seeley wrote:
> It's been a while since I wrote that benchmarker... is it OK that the
> answer is different?  Did you use the same test file?
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmiller@gmail.com> wrote:
>   
>> The results:
>>
>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>> bufsize=1024 poolsize=2 filelen=730554368
>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>
>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>> bufsize=1024 poolsize=2 filelen=730554368
>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>
>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
>> bufsize=1024 poolsize=2 filelen=730554368
>> answer=-282295361, ms=632253, MB/sec=462.19115955163517
>>
>> config: impl=PooledPread serial=false nThreads=4 iterations=100
>> bufsize=1024 poolsize=2 filelen=730554368
>> answer=-282295361, ms=774664, MB/sec=377.2238637654518
>>
>> ClassicFile was heading for the same fate as ChannelFile.
>>
>>
>> I'll have to check what its like on the file system - but it appears
>> just ridiculously slower. Even with SeparateFile, All 4 cores are bouncing
>> from 0-12% independently and really favoring the low end of that.
>> ChannelPread appears no better.
>>
>> There are results from other OS's/setups in the JIRA issue.
>>
>> I'm using ext4.
>>
>> Uwe Schindler wrote:
>>     
>>> How does a conventional file system compare?
>>>
>>> -----
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>>> Sent: Tuesday, September 15, 2009 7:15 PM
>>>> To: java-user@lucene.apache.org
>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>>>
>>>> Mark Miller wrote:
>>>>
>>>>         
>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk - with
>>>>> SeparateFile all 4 of my cores are immediately pinned and remain so.
>>>>> With ChannelFile, all 4 cores hover 20-30%.
>>>>>
>>>>> It would appear it may not be a good idea to use NIOFSDirectory on
>>>>>
>>>>>           
>>>> ramdisks.
>>>>
>>>>         
>>>>> Even still though - it looks like you have a further issue - your Lucene
>>>>> 2.9 old-api results don't use it, and are still not good.
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> The quick results:
>>>>
>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o
>>>> size=1G,nr_inodes=200k,mode=01777
>>>>
>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>
>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>         
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>       
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


-- 
- Mark

http://www.lucidimagination.com




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message