Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of markrmiller@gmail.com
 designates 209.85.220.218 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type
         :content-transfer-encoding;
        b=FDbToqifG2g54eAbDd6Sz1NrNSOS+NuHnog72THSNm0BA5+5WZRLy42LrL12lOacGY
         BWhttVRxRFQ/okinuDTMKz8SiJ9S9LOCvL0Mg3y1YITaPerGzxYrkiqGzVa7zMRGPbea
         bdQjuxQSdsGnyRMCsglq/Uw9xwfk6Y/0D5LPE=
Message-ID: <4AAFB2EB.4020403@gmail.com>
Date: Tue, 15 Sep 2009 11:29:47 -0400
From: Mark Miller <markrmiller@gmail.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: lucene 2.9.0RC4 slower than 2.4.1?
References: <4AAF96E6.7000909@net-m.de> <4AAF9700.9010509@net-m.de>
 <4AAF98D1.5010207@net-m.de> <4AAF9B03.1040708@gmail.com>
 <4AAF9F17.2030307@net-m.de> <4AAFA0EA.2090300@gmail.com>
 <4AAFA800.5040600@net-m.de> <4AAFAA41.1070103@gmail.com>
 <4AAFAC95.4050307@gmail.com> <4AAFB183.4070006@net-m.de>
In-Reply-To: <4AAFB183.4070006@net-m.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thomas Becker wrote:
> Hey Mark,
>
> yes. I'm running the app on unix. You see the difference between 2.9 and 2.4 here:
>
> http://ankeschwarzer.de/tmp/graph.jpg
>   
Right - I know your measurements showed a difference (and will keep that
in mind) - but the profiling results then seem
oddly similar.

> 2.4 responds much quicker thus increasing throughput severly. I'm having a
> single segment only:
>
> -rw-r--r-- 1 asuser asgroup         20 Sep  9 16:40 segments.gen
> -rw-r--r-- 1 asuser asgroup        294 Sep  9 16:44 segments_1o
> -rw-r--r-- 1 asuser asgroup 2810603184 Sep  9 16:44 _7c.cfs
>
> The FileChannel.read hotspot is indeed strange. Especially if taking into
> account that the index is lying on a tmpfs (in memory). And 2.4 should be using
> NIOFSDirectory as well?! Will check that.
>   
2.4 did not use NIOFSDirectory by default - you would have had to
manually specified it. Now its used by default if its detects a non
Windows OS.

Any particular reason your profiling output does not show invocations?
Its not essential most likely, but I've found it to be helpful in
comparisons.

We are about to release 2.9, and its been such a long haul, I'd hate to
see a release with an unknown performance trap.

-- 
- Mark

http://www.lucidimagination.com


> Thanks a lot for your support!
>
> Cheers,
> Thomas
>
> Mark Miller wrote:
>   
>> A few quick notes -
>>
>> Lucene 2.9 old api doesn't appear much worse than Lucene 2.4?
>>
>> You save a lot with the new Intern impl, because thats not a hotspot
>> anymore. But then,
>> RandomAccessFile seeks end up being a lot more of the pie. They look
>> fairly similar in speed overall?
>>
>> It looks like the major bottleneck with 2.9 new api is that its using
>> NIOFSDirectory (your on unix I guess, and it now
>> defaults to that on non Windows os's), and that appears to be a real
>> killer for you. Its taking half the time for its
>> reads.  ???
>>
>> No conclusions yet, but I'm looking it over. Some other guys will come
>> in with some ideas as well.
>>
>> Do confirm that those profiling results are on a single segment though.
>>
>> - Mark
>>
>>
>> Mark Miller wrote:
>>     
>>> Thomas Becker wrote:
>>>   
>>>       
>>>> Here's the results of profiling 10 different search requests:
>>>>
>>>> http://ankeschwarzer.de/tmp/lucene_24_oldapi.png
>>>> http://ankeschwarzer.de/tmp/lucene_29_oldapi.png
>>>> http://ankeschwarzer.de/tmp/lucene_29_newapi.png
>>>>
>>>> But you already gave me a good hint. The index being used is an old one build
>>>> with lucene 2.4. I will now try a freshly build 2.9 index and see if performance
>>>> improves. Maybe that already solves the issue...stupid me...
>>>>   
>>>>     
>>>>         
>>> That shouldn't be an issue unless there is some odd bug.
>>>
>>>   
>>>       
>>>> We're updating the index every 30 min. at the moment and it gets optimized after
>>>> each update.
>>>>   
>>>>     
>>>>         
>>> So this profiling is on an optimized index (eg a single segment) ?
>>> That would be odd indeed, and possibly point to some of the scoring changes.
>>>
>>>   
>>>       
>>>> Mark Miller wrote:
>>>>   
>>>>     
>>>>         
>>>>> Thomas Becker wrote:
>>>>>     
>>>>>       
>>>>>           
>>>>>> Hey Mark,
>>>>>>
>>>>>> thanks for your reply. Will do. Results will follow in a couple of minutes.
>>>>>>
>>>>>>
>>>>>>   
>>>>>>       
>>>>>>         
>>>>>>             
>>>>> Thanks, awesome.
>>>>>
>>>>> Also, how many segments (approx) are in your index? If there are a lot,
>>>>> have you/can you try the same tests on an optimized index? Don't want to
>>>>> get ahead of the profiling results, but just to continue the info loop.
>>>>>
>>>>>     
>>>>>       
>>>>>           
>>>>   
>>>>     
>>>>         
>>>   
>>>       
>>     
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org