lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 16:45:44 GMT
Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk - with
SeparateFile all 4 of my cores are immediately pinned and remain so.
With ChannelFile, all 4 cores hover 20-30%.

It would appear it may not be a good idea to use NIOFSDirectory on ramdisks.

Even still though - it looks like you have a further issue - your Lucene
2.9 old-api results don't use it, and are still not good.

-- 
- Mark

http://www.lucidimagination.com


Uwe Schindler wrote:
> Maybe Linux has some problems with NIO on tmpfs/other ramdisks. What Linux
> do you use, 64bit or 32bit JVM and kernel, ram fs type?
>
> If you have 64 bit and you stored your index in Linux tmpfs (not the old RAM
> fs), the fastest would be MMapDirectory, as the tmpfs RAM can be directly
> used when mapped into the JVM address space.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>   
>> -----Original Message-----
>> From: Mark Miller [mailto:markrmiller@gmail.com]
>> Sent: Tuesday, September 15, 2009 5:30 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>
>> Thomas Becker wrote:
>>     
>>> Hey Mark,
>>>
>>> yes. I'm running the app on unix. You see the difference between 2.9 and
>>>       
>> 2.4 here:
>>     
>>> http://ankeschwarzer.de/tmp/graph.jpg
>>>
>>>       
>> Right - I know your measurements showed a difference (and will keep that
>> in mind) - but the profiling results then seem
>> oddly similar.
>>
>>     
>>> 2.4 responds much quicker thus increasing throughput severly. I'm having
>>>       
>> a
>>     
>>> single segment only:
>>>
>>> -rw-r--r-- 1 asuser asgroup         20 Sep  9 16:40 segments.gen
>>> -rw-r--r-- 1 asuser asgroup        294 Sep  9 16:44 segments_1o
>>> -rw-r--r-- 1 asuser asgroup 2810603184 Sep  9 16:44 _7c.cfs
>>>
>>> The FileChannel.read hotspot is indeed strange. Especially if taking
>>>       
>> into
>>     
>>> account that the index is lying on a tmpfs (in memory). And 2.4 should
>>>       
>> be using
>>     
>>> NIOFSDirectory as well?! Will check that.
>>>
>>>       
>> 2.4 did not use NIOFSDirectory by default - you would have had to
>> manually specified it. Now its used by default if its detects a non
>> Windows OS.
>>
>> Any particular reason your profiling output does not show invocations?
>> Its not essential most likely, but I've found it to be helpful in
>> comparisons.
>>
>> We are about to release 2.9, and its been such a long haul, I'd hate to
>> see a release with an unknown performance trap.
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>     
>>> Thanks a lot for your support!
>>>
>>> Cheers,
>>> Thomas
>>>
>>> Mark Miller wrote:
>>>
>>>       
>>>> A few quick notes -
>>>>
>>>> Lucene 2.9 old api doesn't appear much worse than Lucene 2.4?
>>>>
>>>> You save a lot with the new Intern impl, because thats not a hotspot
>>>> anymore. But then,
>>>> RandomAccessFile seeks end up being a lot more of the pie. They look
>>>> fairly similar in speed overall?
>>>>
>>>> It looks like the major bottleneck with 2.9 new api is that its using
>>>> NIOFSDirectory (your on unix I guess, and it now
>>>> defaults to that on non Windows os's), and that appears to be a real
>>>> killer for you. Its taking half the time for its
>>>> reads.  ???
>>>>
>>>> No conclusions yet, but I'm looking it over. Some other guys will come
>>>> in with some ideas as well.
>>>>
>>>> Do confirm that those profiling results are on a single segment though.
>>>>
>>>> - Mark
>>>>
>>>>
>>>> Mark Miller wrote:
>>>>
>>>>         
>>>>> Thomas Becker wrote:
>>>>>
>>>>>
>>>>>           
>>>>>> Here's the results of profiling 10 different search requests:
>>>>>>
>>>>>> http://ankeschwarzer.de/tmp/lucene_24_oldapi.png
>>>>>> http://ankeschwarzer.de/tmp/lucene_29_oldapi.png
>>>>>> http://ankeschwarzer.de/tmp/lucene_29_newapi.png
>>>>>>
>>>>>> But you already gave me a good hint. The index being used is an old
>>>>>>             
>> one build
>>     
>>>>>> with lucene 2.4. I will now try a freshly build 2.9 index and see
if
>>>>>>             
>> performance
>>     
>>>>>> improves. Maybe that already solves the issue...stupid me...
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> That shouldn't be an issue unless there is some odd bug.
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> We're updating the index every 30 min. at the moment and it gets
>>>>>>             
>> optimized after
>>     
>>>>>> each update.
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> So this profiling is on an optimized index (eg a single segment) ?
>>>>> That would be odd indeed, and possibly point to some of the scoring
>>>>>           
>> changes.
>>     
>>>>>
>>>>>           
>>>>>> Mark Miller wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Thomas Becker wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Hey Mark,
>>>>>>>>
>>>>>>>> thanks for your reply. Will do. Results will follow in a
couple of
>>>>>>>>                 
>> minutes.
>>     
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Thanks, awesome.
>>>>>>>
>>>>>>> Also, how many segments (approx) are in your index? If there
are a
>>>>>>>               
>> lot,
>>     
>>>>>>> have you/can you try the same tests on an optimized index? Don't
>>>>>>>               
>> want to
>>     
>>>>>>> get ahead of the profiling results, but just to continue the
info
>>>>>>>               
>> loop.
>>     
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>
>>>>>>             
>>>>>           
>>>       
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>     
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message