Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of markrmiller@gmail.com
 designates 209.85.220.218 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type
         :content-transfer-encoding;
        b=XVVAxDpCrD3Bnr+qxeL/zA30ayx02pgOP0DPNMJKq6joljyMHRZJ/VlJU/8a98ABNg
         836gCykwzXSJN+qcMDdw4nUHRMSE1Kue9sg3awjwuTcjcjwuHCXtojbuO3iyEP/qGxe1
         Uyi70selh3SPub/duOrSfBd9S5fXg+EF5BYS8=
Message-ID: <4AAFE86B.2090509@gmail.com>
Date: Tue, 15 Sep 2009 15:18:03 -0400
From: Mark Miller <markrmiller@gmail.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
References: <4AAF96E6.7000909@net-m.de>
 <4AAFAC95.4050307@gmail.com>	 <4AAFB183.4070006@net-m.de>
 <4AAFB2EB.4020403@gmail.com>	 <5985A0861800491EAFFCD4B0B61351A0@VEGA>
 <4AAFC4B8.9040608@gmail.com>	 <4AAFCBB2.8090601@gmail.com>
 <020FFC0B85B44457BA5D38315A91422C@VEGA>	 <4AAFDA7D.8010202@gmail.com>
	 <c68e39170909151159m493c8c80pc6e81e7f021a1c47@mail.gmail.com>
 <9ac0c6aa0909151210s3c20b9f9mfad6c676d65fd96b@mail.gmail.com>
In-Reply-To: <9ac0c6aa0909151210s3c20b9f9mfad6c676d65fd96b@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Michael McCandless wrote:
> I don't like that the answer is different... but it's really really
> odd that it's different-yet-almost-the-same.
>
> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs?
> (Because the top 2 entries of your 4 results match the first set of 2
> entries you sent... so I'm thinking these 4 were actually tmpfs not
> ext4).
>   
Those 4 were tmpfs - I mention ext4 at the end because I had just given
a feel for the hardrive tests and wanted to note it was from ext4 - the
results are def ramdisk though.
> What JRE/OS, linux, kernel versions, and hardware, are you running on?
>   
These are on:
Ubuntu Karmic Koala 9.10, currently updated
java-1.5.0-sun-1.5.0.20
2.6.31-10-generic

RAM: 3.9 Gig
4 core Intel Core2 duo 2.0GHz

Slow 5200 rpm laptop drives.

> The gains of SeparateFile over all else are stunning.  And, quite
> different from the linux tests I had run under LUCENE-753.  Maybe we
> need to revert FSDir.open to return SimpleFSDir again, on non-Windows
> hosts.  But then we don't have good concurrency...
>
> Mike
>
> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley
> <yonik.seeley@lucidimagination.com> wrote:
>   
>> It's been a while since I wrote that benchmarker... is it OK that the
>> answer is different?  Did you use the same test file?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmiller@gmail.com> wrote:
>>     
>>> The results:
>>>
>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>
>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>
>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517
>>>
>>> config: impl=PooledPread serial=false nThreads=4 iterations=100
>>> bufsize=1024 poolsize=2 filelen=730554368
>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518
>>>
>>> ClassicFile was heading for the same fate as ChannelFile.
>>>
>>>
>>> I'll have to check what its like on the file system - but it appears
>>> just ridiculously slower. Even with SeparateFile, All 4 cores are bouncing
>>> from 0-12% independently and really favoring the low end of that.
>>> ChannelPread appears no better.
>>>
>>> There are results from other OS's/setups in the JIRA issue.
>>>
>>> I'm using ext4.
>>>
>>> Uwe Schindler wrote:
>>>       
>>>> How does a conventional file system compare?
>>>>
>>>> -----
>>>> Uwe Schindler
>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>> http://www.thetaphi.de
>>>> eMail: uwe@thetaphi.de
>>>>
>>>>
>>>>         
>>>>> -----Original Message-----
>>>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>>>> Sent: Tuesday, September 15, 2009 7:15 PM
>>>>> To: java-user@lucene.apache.org
>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>>>>
>>>>> Mark Miller wrote:
>>>>>
>>>>>           
>>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk - with
>>>>>> SeparateFile all 4 of my cores are immediately pinned and remain so.
>>>>>> With ChannelFile, all 4 cores hover 20-30%.
>>>>>>
>>>>>> It would appear it may not be a good idea to use NIOFSDirectory on
>>>>>>
>>>>>>             
>>>>> ramdisks.
>>>>>
>>>>>           
>>>>>> Even still though - it looks like you have a further issue - your Lucene
>>>>>> 2.9 old-api results don't use it, and are still not good.
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> The quick results:
>>>>>
>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o
>>>>> size=1G,nr_inodes=200k,mode=01777
>>>>>
>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
>>>>>
>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
>>>>> bufsize=1024 poolsize=2 filelen=730554368
>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
>>>>>
>>>>>
>>>>> --
>>>>> - Mark
>>>>>
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>           
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>         
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


-- 
- Mark

http://www.lucidimagination.com


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org