lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 20:09:55 GMT
Disturbed reminds me of the owl from sword in the stone ;)

Thats a great one liner - now I am completely disturbed.

Sorry - I've been known to do that -

The two results that I say specifically are from the harddisk - those
are from the harddisk and are ext4. They are a tad slower than the
ramdisk results.

Everything else is the ramdisk - and I wasn't sure if thats ext4 or ext3
(and my mind hasn't wandered towards finding the answer yet). I didn't
specify when creating it.

- Mark

Uwe Schindler wrote:
> Now I am completely disturbed. Which numbers come from which filesystem?
> Ext4 on HDD, tmpfs (which is a filesystem of its own), ext3 on HDD,...
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>   
>> -----Original Message-----
>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
>> Seeley
>> Sent: Tuesday, September 15, 2009 9:33 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>
>> Remember to disable CPU frequency scaling when benchmarking... some
>> things with IO cause the freq to drop, and when it's CPU bound again
>> it takes a while for Linux to scale up the freq again.
>>
>> For example, on my ubuntu box, ChannelFile went from 100MB/sec to
>> 388MB/sec.  This effect probably won't be uniform across
>> implementations either.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>> On Tue, Sep 15, 2009 at 3:25 PM, Mark Miller <markrmiller@gmail.com>
>> wrote:
>>     
>>> I just really I hadn't sent this one. Here are results from the
>>>       
>> harddrive:
>>     
>>> It looks like its closer to the same speed on the hardrive once
>>> everything is loaded in the system cache (as you'd expect). SeparateFile
>>> was 1200 vs almost 1700 on RAMDISK. ChannelPread looked a lot closer
>>>       
>> though.
>>     
>>> - Mark
>>>
>>> Tests from harddisk (filesystem cache warmed):
>>>
>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282293977, ms=238230, MB/sec=1226.6370616630988
>>>
>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=766340, MB/sec=381.3212767179059
>>>
>>>
>>> Mark Miller wrote:
>>>       
>>>> Michael McCandless wrote:
>>>>
>>>>         
>>>>> I don't like that the answer is different... but it's really really
>>>>> odd that it's different-yet-almost-the-same.
>>>>>
>>>>> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs?
>>>>> (Because the top 2 entries of your 4 results match the first set of 2
>>>>> entries you sent... so I'm thinking these 4 were actually tmpfs not
>>>>> ext4).
>>>>>
>>>>>
>>>>>           
>>>> Those 4 were tmpfs - I mention ext4 at the end because I had just given
>>>> a feel for the hardrive tests and wanted to note it was from ext4 - the
>>>> results are def ramdisk though.
>>>>
>>>>         
>>>>> What JRE/OS, linux, kernel versions, and hardware, are you running on?
>>>>>
>>>>>
>>>>>           
>>>> These are on:
>>>> Ubuntu Karmic Koala 9.10, currently updated
>>>> java-1.5.0-sun-1.5.0.20
>>>> 2.6.31-10-generic
>>>>
>>>> RAM: 3.9 Gig
>>>> 4 core Intel Core2 duo 2.0GHz
>>>>
>>>> Slow 5200 rpm laptop drives.
>>>>
>>>>
>>>>         
>>>>> The gains of SeparateFile over all else are stunning.  And, quite
>>>>> different from the linux tests I had run under LUCENE-753.  Maybe we
>>>>> need to revert FSDir.open to return SimpleFSDir again, on non-Windows
>>>>> hosts.  But then we don't have good concurrency...
>>>>>
>>>>> Mike
>>>>>
>>>>> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley
>>>>> <yonik.seeley@lucidimagination.com> wrote:
>>>>>
>>>>>
>>>>>           
>>>>>> It's been a while since I wrote that benchmarker... is it OK that
the
>>>>>> answer is different?  Did you use the same test file?
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmiller@gmail.com>
>>>>>>             
>> wrote:
>>     
>>>>>>             
>>>>>>> The results:
>>>>>>>
>>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>>>>
>>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>>>>
>>>>>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
>>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517
>>>>>>>
>>>>>>> config: impl=PooledPread serial=false nThreads=4 iterations=100
>>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518
>>>>>>>
>>>>>>> ClassicFile was heading for the same fate as ChannelFile.
>>>>>>>
>>>>>>>
>>>>>>> I'll have to check what its like on the file system - but it
appears
>>>>>>> just ridiculously slower. Even with SeparateFile, All 4 cores
are
>>>>>>>               
>> bouncing
>>     
>>>>>>> from 0-12% independently and really favoring the low end of that.
>>>>>>> ChannelPread appears no better.
>>>>>>>
>>>>>>> There are results from other OS's/setups in the JIRA issue.
>>>>>>>
>>>>>>> I'm using ext4.
>>>>>>>
>>>>>>> Uwe Schindler wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> How does a conventional file system compare?
>>>>>>>>
>>>>>>>> -----
>>>>>>>> Uwe Schindler
>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>>>>>> http://www.thetaphi.de
>>>>>>>> eMail: uwe@thetaphi.de
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>>>>>>>> Sent: Tuesday, September 15, 2009 7:15 PM
>>>>>>>>> To: java-user@lucene.apache.org
>>>>>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>>>>>>>>
>>>>>>>>> Mark Miller wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Indeed - I just ran the FileReaderTest on a Linux
tmpfs ramdisk -
>>>>>>>>>>                     
>> with
>>     
>>>>>>>>>> SeparateFile all 4 of my cores are immediately pinned
and remain
>>>>>>>>>>                     
>> so.
>>     
>>>>>>>>>> With ChannelFile, all 4 cores hover 20-30%.
>>>>>>>>>>
>>>>>>>>>> It would appear it may not be a good idea to use
NIOFSDirectory
>>>>>>>>>>                     
>> on
>>     
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> ramdisks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Even still though - it looks like you have a further
issue - your
>>>>>>>>>>                     
>> Lucene
>>     
>>>>>>>>>> 2.9 old-api results don't use it, and are still not
good.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> The quick results:
>>>>>>>>>
>>>>>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o
>>>>>>>>> size=1G,nr_inodes=200k,mode=01777
>>>>>>>>>
>>>>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>>>>>>
>>>>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>>>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> - Mark
>>>>>>>>>
>>>>>>>>> http://www.lucidimagination.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------
>>>>>>>>>                   
>> ---
>>     
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> -------------------------------------------------------------------
>>>>>>>>                 
>> --
>>     
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> --
>>>>>>> - Mark
>>>>>>>
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --------------------------------------------------------------------
>>>>>>>               
>> -
>>     
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>
>>>>         
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>     
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message