lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 19:25:00 GMT
I just really I hadn't sent this one. Here are results from the harddrive:

It looks like its closer to the same speed on the hardrive once
everything is loaded in the system cache (as you'd expect). SeparateFile
was 1200 vs almost 1700 on RAMDISK. ChannelPread looked a lot closer though.


- Mark

Tests from harddisk (filesystem cache warmed):

config: impl=SeparateFile serial=false nThreads=4 iterations=100
bufsize=1024 poolsize=2 filelen=730554368
answer=-282293977, ms=238230, MB/sec=1226.6370616630988

config: impl=ChannelPread serial=false nThreads=4 iterations=100
bufsize=1024 poolsize=2 filelen=730554368
answer=-282295361, ms=766340, MB/sec=381.3212767179059


Mark Miller wrote:
> Michael McCandless wrote:
>   
>> I don't like that the answer is different... but it's really really
>> odd that it's different-yet-almost-the-same.
>>
>> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs?
>> (Because the top 2 entries of your 4 results match the first set of 2
>> entries you sent... so I'm thinking these 4 were actually tmpfs not
>> ext4).
>>   
>>     
> Those 4 were tmpfs - I mention ext4 at the end because I had just given
> a feel for the hardrive tests and wanted to note it was from ext4 - the
> results are def ramdisk though.
>   
>> What JRE/OS, linux, kernel versions, and hardware, are you running on?
>>   
>>     
> These are on:
> Ubuntu Karmic Koala 9.10, currently updated
> java-1.5.0-sun-1.5.0.20
> 2.6.31-10-generic
>
> RAM: 3.9 Gig
> 4 core Intel Core2 duo 2.0GHz
>
> Slow 5200 rpm laptop drives.
>
>   
>> The gains of SeparateFile over all else are stunning.  And, quite
>> different from the linux tests I had run under LUCENE-753.  Maybe we
>> need to revert FSDir.open to return SimpleFSDir again, on non-Windows
>> hosts.  But then we don't have good concurrency...
>>
>> Mike
>>
>> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley
>> <yonik.seeley@lucidimagination.com> wrote:
>>   
>>     
>>> It's been a while since I wrote that benchmarker... is it OK that the
>>> answer is different?  Did you use the same test file?
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmiller@gmail.com> wrote:
>>>     
>>>       
>>>> The results:
>>>>
>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>
>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>
>>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517
>>>>
>>>> config: impl=PooledPread serial=false nThreads=4 iterations=100
>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518
>>>>
>>>> ClassicFile was heading for the same fate as ChannelFile.
>>>>
>>>>
>>>> I'll have to check what its like on the file system - but it appears
>>>> just ridiculously slower. Even with SeparateFile, All 4 cores are bouncing
>>>> from 0-12% independently and really favoring the low end of that.
>>>> ChannelPread appears no better.
>>>>
>>>> There are results from other OS's/setups in the JIRA issue.
>>>>
>>>> I'm using ext4.
>>>>
>>>> Uwe Schindler wrote:
>>>>       
>>>>         
>>>>> How does a conventional file system compare?
>>>>>
>>>>> -----
>>>>> Uwe Schindler
>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>>> http://www.thetaphi.de
>>>>> eMail: uwe@thetaphi.de
>>>>>
>>>>>
>>>>>         
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>>>>> Sent: Tuesday, September 15, 2009 7:15 PM
>>>>>> To: java-user@lucene.apache.org
>>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>>>>>
>>>>>> Mark Miller wrote:
>>>>>>
>>>>>>           
>>>>>>             
>>>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk
- with
>>>>>>> SeparateFile all 4 of my cores are immediately pinned and remain
so.
>>>>>>> With ChannelFile, all 4 cores hover 20-30%.
>>>>>>>
>>>>>>> It would appear it may not be a good idea to use NIOFSDirectory
on
>>>>>>>
>>>>>>>             
>>>>>>>               
>>>>>> ramdisks.
>>>>>>
>>>>>>           
>>>>>>             
>>>>>>> Even still though - it looks like you have a further issue -
your Lucene
>>>>>>> 2.9 old-api results don't use it, and are still not good.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             
>>>>>>>               
>>>>>> The quick results:
>>>>>>
>>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o
>>>>>> size=1G,nr_inodes=200k,mode=01777
>>>>>>
>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>>>
>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Mark
>>>>>>
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>           
>>>>>>             
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>         
>>>>>           
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>       
>>>>         
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>     
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>   
>>     
>
>
>   


-- 
- Mark

http://www.lucidimagination.com




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message