lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Becker <thomas.bec...@net-m.de>
Subject Re: lucene 2.9.0RC4 slower than 2.4.1?
Date Tue, 15 Sep 2009 15:59:40 GMT
Hi Uwe,

already done. See my last message.

Cheers,
Thomas

Uwe Schindler wrote:
> On 2.9. NIOFS is only used, if you use FSDirectory.open() instead of
> FSDirectory.getDirectory (Deprecated). Can you compare when you use instead
> of FSDirectory.open() the direct ctor of SimpleFSDir vs. NIOFSDir vs.
> MMapDir and compare?
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
>> -----Original Message-----
>> From: Mark Miller [mailto:markrmiller@gmail.com]
>> Sent: Tuesday, September 15, 2009 5:30 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
>>
>> Thomas Becker wrote:
>>> Hey Mark,
>>>
>>> yes. I'm running the app on unix. You see the difference between 2.9 and
>> 2.4 here:
>>> http://ankeschwarzer.de/tmp/graph.jpg
>>>
>> Right - I know your measurements showed a difference (and will keep that
>> in mind) - but the profiling results then seem
>> oddly similar.
>>
>>> 2.4 responds much quicker thus increasing throughput severly. I'm having
>> a
>>> single segment only:
>>>
>>> -rw-r--r-- 1 asuser asgroup         20 Sep  9 16:40 segments.gen
>>> -rw-r--r-- 1 asuser asgroup        294 Sep  9 16:44 segments_1o
>>> -rw-r--r-- 1 asuser asgroup 2810603184 Sep  9 16:44 _7c.cfs
>>>
>>> The FileChannel.read hotspot is indeed strange. Especially if taking
>> into
>>> account that the index is lying on a tmpfs (in memory). And 2.4 should
>> be using
>>> NIOFSDirectory as well?! Will check that.
>>>
>> 2.4 did not use NIOFSDirectory by default - you would have had to
>> manually specified it. Now its used by default if its detects a non
>> Windows OS.
>>
>> Any particular reason your profiling output does not show invocations?
>> Its not essential most likely, but I've found it to be helpful in
>> comparisons.
>>
>> We are about to release 2.9, and its been such a long haul, I'd hate to
>> see a release with an unknown performance trap.
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>> Thanks a lot for your support!
>>>
>>> Cheers,
>>> Thomas
>>>
>>> Mark Miller wrote:
>>>
>>>> A few quick notes -
>>>>
>>>> Lucene 2.9 old api doesn't appear much worse than Lucene 2.4?
>>>>
>>>> You save a lot with the new Intern impl, because thats not a hotspot
>>>> anymore. But then,
>>>> RandomAccessFile seeks end up being a lot more of the pie. They look
>>>> fairly similar in speed overall?
>>>>
>>>> It looks like the major bottleneck with 2.9 new api is that its using
>>>> NIOFSDirectory (your on unix I guess, and it now
>>>> defaults to that on non Windows os's), and that appears to be a real
>>>> killer for you. Its taking half the time for its
>>>> reads.  ???
>>>>
>>>> No conclusions yet, but I'm looking it over. Some other guys will come
>>>> in with some ideas as well.
>>>>
>>>> Do confirm that those profiling results are on a single segment though.
>>>>
>>>> - Mark
>>>>
>>>>
>>>> Mark Miller wrote:
>>>>
>>>>> Thomas Becker wrote:
>>>>>
>>>>>
>>>>>> Here's the results of profiling 10 different search requests:
>>>>>>
>>>>>> http://ankeschwarzer.de/tmp/lucene_24_oldapi.png
>>>>>> http://ankeschwarzer.de/tmp/lucene_29_oldapi.png
>>>>>> http://ankeschwarzer.de/tmp/lucene_29_newapi.png
>>>>>>
>>>>>> But you already gave me a good hint. The index being used is an old
>> one build
>>>>>> with lucene 2.4. I will now try a freshly build 2.9 index and see
if
>> performance
>>>>>> improves. Maybe that already solves the issue...stupid me...
>>>>>>
>>>>>>
>>>>>>
>>>>> That shouldn't be an issue unless there is some odd bug.
>>>>>
>>>>>
>>>>>
>>>>>> We're updating the index every 30 min. at the moment and it gets
>> optimized after
>>>>>> each update.
>>>>>>
>>>>>>
>>>>>>
>>>>> So this profiling is on an optimized index (eg a single segment) ?
>>>>> That would be odd indeed, and possibly point to some of the scoring
>> changes.
>>>>>
>>>>>
>>>>>> Mark Miller wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thomas Becker wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hey Mark,
>>>>>>>>
>>>>>>>> thanks for your reply. Will do. Results will follow in a
couple of
>> minutes.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks, awesome.
>>>>>>>
>>>>>>> Also, how many segments (approx) are in your index? If there
are a
>> lot,
>>>>>>> have you/can you try the same tests on an optimized index? Don't
>> want to
>>>>>>> get ahead of the profiling results, but just to continue the
info
>> loop.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

-- 
Thomas Becker
Senior JEE Developer

net mobile AG
Zollhof 17
40221 Düsseldorf
GERMANY

Phone:    +49 211 97020-195
Fax:      +49 211 97020-949
Mobile:   +49 173 5146567 (private)
E-Mail:   mailto:thomas.becker@net-m.de
Internet: http://www.net-m.de

Registergericht:  Amtsgericht Düsseldorf, HRB 48022
Vorstand:         Theodor Niehues (Vorsitzender), Frank Hartmann,
                 Kai Markus Kulas, Dieter Plassmann
Vorsitzender des
Aufsichtsrates:   Dr. Michael Briem

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message