lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 19:44:13 GMT
Now I am completely disturbed. Which numbers come from which filesystem?
Ext4 on HDD, tmpfs (which is a filesystem of its own), ext3 on HDD,...

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Tuesday, September 15, 2009 9:33 PM
> To: java-user@lucene.apache.org
> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
> 
> Remember to disable CPU frequency scaling when benchmarking... some
> things with IO cause the freq to drop, and when it's CPU bound again
> it takes a while for Linux to scale up the freq again.
> 
> For example, on my ubuntu box, ChannelFile went from 100MB/sec to
> 388MB/sec.  This effect probably won't be uniform across
> implementations either.
> 
> -Yonik
> http://www.lucidimagination.com
> 
> On Tue, Sep 15, 2009 at 3:25 PM, Mark Miller <markrmiller@gmail.com>
> wrote:
> > I just really I hadn't sent this one. Here are results from the
> harddrive:
> >
> > It looks like its closer to the same speed on the hardrive once
> > everything is loaded in the system cache (as you'd expect). SeparateFile
> > was 1200 vs almost 1700 on RAMDISK. ChannelPread looked a lot closer
> though.
> >
> >
> > - Mark
> >
> > Tests from harddisk (filesystem cache warmed):
> >
> > config: impl=SeparateFile serial=false nThreads=4 iterations=100
> > bufsize=1024 poolsize=2 filelen=730554368
> > answer=-282293977, ms=238230, MB/sec=1226.6370616630988
> >
> > config: impl=ChannelPread serial=false nThreads=4 iterations=100
> > bufsize=1024 poolsize=2 filelen=730554368
> > answer=-282295361, ms=766340, MB/sec=381.3212767179059
> >
> >
> > Mark Miller wrote:
> >> Michael McCandless wrote:
> >>
> >>> I don't like that the answer is different... but it's really really
> >>> odd that it's different-yet-almost-the-same.
> >>>
> >>> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs?
> >>> (Because the top 2 entries of your 4 results match the first set of 2
> >>> entries you sent... so I'm thinking these 4 were actually tmpfs not
> >>> ext4).
> >>>
> >>>
> >> Those 4 were tmpfs - I mention ext4 at the end because I had just given
> >> a feel for the hardrive tests and wanted to note it was from ext4 - the
> >> results are def ramdisk though.
> >>
> >>> What JRE/OS, linux, kernel versions, and hardware, are you running on?
> >>>
> >>>
> >> These are on:
> >> Ubuntu Karmic Koala 9.10, currently updated
> >> java-1.5.0-sun-1.5.0.20
> >> 2.6.31-10-generic
> >>
> >> RAM: 3.9 Gig
> >> 4 core Intel Core2 duo 2.0GHz
> >>
> >> Slow 5200 rpm laptop drives.
> >>
> >>
> >>> The gains of SeparateFile over all else are stunning.  And, quite
> >>> different from the linux tests I had run under LUCENE-753.  Maybe we
> >>> need to revert FSDir.open to return SimpleFSDir again, on non-Windows
> >>> hosts.  But then we don't have good concurrency...
> >>>
> >>> Mike
> >>>
> >>> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley
> >>> <yonik.seeley@lucidimagination.com> wrote:
> >>>
> >>>
> >>>> It's been a while since I wrote that benchmarker... is it OK that the
> >>>> answer is different?  Did you use the same test file?
> >>>>
> >>>> -Yonik
> >>>> http://www.lucidimagination.com
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmiller@gmail.com>
> wrote:
> >>>>
> >>>>
> >>>>> The results:
> >>>>>
> >>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
> >>>>> bufsize=1024 poolsize=2 filelen=730554368
> >>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
> >>>>>
> >>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
> >>>>> bufsize=1024 poolsize=2 filelen=730554368
> >>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
> >>>>>
> >>>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100
> >>>>> bufsize=1024 poolsize=2 filelen=730554368
> >>>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517
> >>>>>
> >>>>> config: impl=PooledPread serial=false nThreads=4 iterations=100
> >>>>> bufsize=1024 poolsize=2 filelen=730554368
> >>>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518
> >>>>>
> >>>>> ClassicFile was heading for the same fate as ChannelFile.
> >>>>>
> >>>>>
> >>>>> I'll have to check what its like on the file system - but it appears
> >>>>> just ridiculously slower. Even with SeparateFile, All 4 cores are
> bouncing
> >>>>> from 0-12% independently and really favoring the low end of that.
> >>>>> ChannelPread appears no better.
> >>>>>
> >>>>> There are results from other OS's/setups in the JIRA issue.
> >>>>>
> >>>>> I'm using ext4.
> >>>>>
> >>>>> Uwe Schindler wrote:
> >>>>>
> >>>>>
> >>>>>> How does a conventional file system compare?
> >>>>>>
> >>>>>> -----
> >>>>>> Uwe Schindler
> >>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
> >>>>>> http://www.thetaphi.de
> >>>>>> eMail: uwe@thetaphi.de
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Mark Miller [mailto:markrmiller@gmail.com]
> >>>>>>> Sent: Tuesday, September 15, 2009 7:15 PM
> >>>>>>> To: java-user@lucene.apache.org
> >>>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
> >>>>>>>
> >>>>>>> Mark Miller wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs
ramdisk -
> with
> >>>>>>>> SeparateFile all 4 of my cores are immediately pinned
and remain
> so.
> >>>>>>>> With ChannelFile, all 4 cores hover 20-30%.
> >>>>>>>>
> >>>>>>>> It would appear it may not be a good idea to use NIOFSDirectory
> on
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ramdisks.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Even still though - it looks like you have a further
issue - your
> Lucene
> >>>>>>>> 2.9 old-api results don't use it, and are still not
good.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> The quick results:
> >>>>>>>
> >>>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o
> >>>>>>> size=1G,nr_inodes=200k,mode=01777
> >>>>>>>
> >>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100
> >>>>>>> bufsize=1024 poolsize=2 filelen=730554368
> >>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938
> >>>>>>>
> >>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100
> >>>>>>> bufsize=1024 poolsize=2 filelen=730554368
> >>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> - Mark
> >>>>>>>
> >>>>>>> http://www.lucidimagination.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ------------------------------------------------------------------
> ---
> >>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> -------------------------------------------------------------------
> --
> >>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>> - Mark
> >>>>>
> >>>>> http://www.lucidimagination.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --------------------------------------------------------------------
> -
> >>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
> > --
> > - Mark
> >
> > http://www.lucidimagination.com
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message