lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 16:37:09 GMT
Maybe Linux has some problems with NIO on tmpfs/other ramdisks. What Linux
do you use, 64bit or 32bit JVM and kernel, ram fs type?

If you have 64 bit and you stored your index in Linux tmpfs (not the old RAM
fs), the fastest would be MMapDirectory, as the tmpfs RAM can be directly
used when mapped into the JVM address space.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Tuesday, September 15, 2009 5:30 PM
> To: java-user@lucene.apache.org
> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
> 
> Thomas Becker wrote:
> > Hey Mark,
> >
> > yes. I'm running the app on unix. You see the difference between 2.9 and
> 2.4 here:
> >
> > http://ankeschwarzer.de/tmp/graph.jpg
> >
> Right - I know your measurements showed a difference (and will keep that
> in mind) - but the profiling results then seem
> oddly similar.
> 
> > 2.4 responds much quicker thus increasing throughput severly. I'm having
> a
> > single segment only:
> >
> > -rw-r--r-- 1 asuser asgroup         20 Sep  9 16:40 segments.gen
> > -rw-r--r-- 1 asuser asgroup        294 Sep  9 16:44 segments_1o
> > -rw-r--r-- 1 asuser asgroup 2810603184 Sep  9 16:44 _7c.cfs
> >
> > The FileChannel.read hotspot is indeed strange. Especially if taking
> into
> > account that the index is lying on a tmpfs (in memory). And 2.4 should
> be using
> > NIOFSDirectory as well?! Will check that.
> >
> 2.4 did not use NIOFSDirectory by default - you would have had to
> manually specified it. Now its used by default if its detects a non
> Windows OS.
> 
> Any particular reason your profiling output does not show invocations?
> Its not essential most likely, but I've found it to be helpful in
> comparisons.
> 
> We are about to release 2.9, and its been such a long haul, I'd hate to
> see a release with an unknown performance trap.
> 
> --
> - Mark
> 
> http://www.lucidimagination.com
> 
> 
> > Thanks a lot for your support!
> >
> > Cheers,
> > Thomas
> >
> > Mark Miller wrote:
> >
> >> A few quick notes -
> >>
> >> Lucene 2.9 old api doesn't appear much worse than Lucene 2.4?
> >>
> >> You save a lot with the new Intern impl, because thats not a hotspot
> >> anymore. But then,
> >> RandomAccessFile seeks end up being a lot more of the pie. They look
> >> fairly similar in speed overall?
> >>
> >> It looks like the major bottleneck with 2.9 new api is that its using
> >> NIOFSDirectory (your on unix I guess, and it now
> >> defaults to that on non Windows os's), and that appears to be a real
> >> killer for you. Its taking half the time for its
> >> reads.  ???
> >>
> >> No conclusions yet, but I'm looking it over. Some other guys will come
> >> in with some ideas as well.
> >>
> >> Do confirm that those profiling results are on a single segment though.
> >>
> >> - Mark
> >>
> >>
> >> Mark Miller wrote:
> >>
> >>> Thomas Becker wrote:
> >>>
> >>>
> >>>> Here's the results of profiling 10 different search requests:
> >>>>
> >>>> http://ankeschwarzer.de/tmp/lucene_24_oldapi.png
> >>>> http://ankeschwarzer.de/tmp/lucene_29_oldapi.png
> >>>> http://ankeschwarzer.de/tmp/lucene_29_newapi.png
> >>>>
> >>>> But you already gave me a good hint. The index being used is an old
> one build
> >>>> with lucene 2.4. I will now try a freshly build 2.9 index and see if
> performance
> >>>> improves. Maybe that already solves the issue...stupid me...
> >>>>
> >>>>
> >>>>
> >>> That shouldn't be an issue unless there is some odd bug.
> >>>
> >>>
> >>>
> >>>> We're updating the index every 30 min. at the moment and it gets
> optimized after
> >>>> each update.
> >>>>
> >>>>
> >>>>
> >>> So this profiling is on an optimized index (eg a single segment) ?
> >>> That would be odd indeed, and possibly point to some of the scoring
> changes.
> >>>
> >>>
> >>>
> >>>> Mark Miller wrote:
> >>>>
> >>>>
> >>>>
> >>>>> Thomas Becker wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Hey Mark,
> >>>>>>
> >>>>>> thanks for your reply. Will do. Results will follow in a couple
of
> minutes.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> Thanks, awesome.
> >>>>>
> >>>>> Also, how many segments (approx) are in your index? If there are
a
> lot,
> >>>>> have you/can you try the same tests on an optimized index? Don't
> want to
> >>>>> get ahead of the profiling results, but just to continue the info
> loop.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
> >
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message