Mailing-List: contact dev-help@jackrabbit.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@jackrabbit.apache.org
Received-SPF: pass (herse.apache.org: domain of
 gcaj-jackrabbit-dev@m.gmane.org designates 80.91.229.2 as permitted sender)
To: dev@jackrabbit.apache.org
From: Christoph Kiehl <christoph@sulu3000.de>
Subject: Re: Optimize search performance
Date: Tue, 19 Jun 2007 18:18:50 +0200
Lines: 42
Message-ID: <f58vir$l2a$1@sea.gmane.org>
References: <f4bc62$buv$1@sea.gmane.org> <466D04F8.8030304@day.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Thunderbird 2.0.0.4 (Windows/20070604)
In-Reply-To: <466D04F8.8030304@day.com>
Sender: news <news@sea.gmane.org>

Marcel Reutegger wrote:

>> 2. Lucene uses the FieldCaches to speed up sorting and range queries 
>> which is exactly what we are after. Those FieldCaches are per 
>> IndexReader.
>> Jackrabbit uses an IndexSearcher which searches on a single 
>> IndexReader which is most likely to be an instance of 
>> CachingMultiReader. So on every search which builds up a FieldCache 
>> this FieldCache instance is associated with this instance of a 
>> CachingMultiReader. On successive queries which operate on this 
>> CachingMultiReader you will get a tremendous speedup for queries which 
>> can reuse  those associated FieldCache instances.
>> The problem is that Jackrabbit creates a new CachingMultiReader 
>> _everytime_ one of the underlying indexes are modified. This means if 
>> you just change _one_ item in the repository you will need to rebuild 
>> all those FieldCaches because the existing FieldCaches are associated 
>> with the old instance of CachingMultiReader.
>> This does not only lead to slow search response times for queries 
>> which contains range queries or are sorted by a field but also leads 
>> to massive memory consumption (depending on the size of your indexes) 
>> because there might be multiple instances of CachingMultiReaders in 
>> use if you have a scenario where a lot of queries and item 
>> modifications are executed concurrently.
>> As far as I understand the solution is to use a MultiSearcher which 
>> uses multiple IndexReaders. Since due to the merging strategy most of 
>> the indexes are stable this means the FieldCaches can be used for a 
>> much longer time.
> 
> Using a multi searcher means that you must be able to execute a query on 
> each of the index segments independently. this is not possible because 
> hierarchy information is always spread across multiple segments. e.g. a 
> node in one segment may reference a parent in another segment.

I just created an issue [1] to which I attached an initial patch which works 
quite well for us. It doesn't use MultiSearcher but extends 
SharedFieldSortComparator to be aware of the underlying index segments. Could 
you please review the patch?

Cheers,
Christoph

[1] http://issues.apache.org/jira/browse/JCR-974