lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 19:18:03 GMT
Michael McCandless wrote:
> I don't like that the answer is different... but it's really really
> odd that it's different-yet-almost-the-same.
>
> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs?
> (Because the top 2 entries of your 4 results match the first set of 2
> entries you sent... so I'm thinking these 4 were actually tmpfs not
> ext4).
>   
Those 4 were tmpfs - I mention ext4 at the end because I had just given
a feel for the hardrive tests and wanted to note it was from ext4 - the
results are def ramdisk though.
> What JRE/OS, linux, kernel versions, and hardware, are you running on?
>   
These are on:
Ubuntu Karmic Koala 9.10, currently updated
java-1.5.0-sun-1.5.0.20
2.6.31-10-generic

RAM: 3.9 Gig
4 core Intel Core2 duo 2.0GHz

Slow 5200 rpm laptop drives.

> The gains of SeparateFile over all else are stunning.  And, quite
> different from the linux tests I had run under LUCENE-753.  Maybe we
> need to revert FSDir.open to return SimpleFSDir again, on non-Windows
> hosts.  But then we don't have good concurrency...
>
> Mike
>
> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley
> <yonik.seeley@lucidimagination.com> wrote:
>   
>> It's been a while since I wrote that benchmarker... is it OK that the
>> answer is different?  Did you use the same test file?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmiller@gmail.com> wrote:
>>     
>>> The results:
>>>
>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>
>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>
>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517
>>>
>>> config: impl=PooledPread serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518
>>>
>>> ClassicFile was heading for the same fate as ChannelFile.
>>>
>>>
>>> I'll have to check what its like on the file system - but it appears
>>> just ridiculously slower. Even with SeparateFile, All 4 cores are bouncing
>>> from 0-12% independently and really favoring the low end of that.
>>> ChannelPread appears no better.
>>>
>>> There are results from other OS's/setups in the JIRA issue.
>>>
>>> I'm using ext4.
>>>
>>> Uwe Schindler wrote:
>>>       
>>>> How does a conventional file system compare?
>>>>
>>>> -----
>>>> Uwe Schindler
>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>> http://www.thetaphi.de
>>>> eMail: uwe@thetaphi.de
>>>>
>>>>
>>>>         
>>>>> -----Original Message-----
>>>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>>>> Sent: Tuesday, September 15, 2009 7:15 PM
>>>>> To: java-user@lucene.apache.org
>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>>>>
>>>>> Mark Miller wrote:
>>>>>
>>>>>           
>>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk -
with
>>>>>> SeparateFile all 4 of my cores are immediately pinned and remain
so.
>>>>>> With ChannelFile, all 4 cores hover 20-30%.
>>>>>>
>>>>>> It would appear it may not be a good idea to use NIOFSDirectory on
>>>>>>
>>>>>>             
>>>>> ramdisks.
>>>>>
>>>>>           
>>>>>> Even still though - it looks like you have a further issue - your
Lucene
>>>>>> 2.9 old-api results don't use it, and are still not good.
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> The quick results:
>>>>>
>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o
>>>>> size=1G,nr_inodes=200k,mode=01777
>>>>>
>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>>
>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>           
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>         
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


-- 
- Mark

http://www.lucidimagination.com




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message