Mailing-List: contact dev-help@jackrabbit.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@jackrabbit.apache.org
Received-SPF: pass (herse.apache.org: domain of mreutegg@gmail.com designates
 66.249.92.168 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding:sender;
        b=khhrl1hUP5DdegT98x8gB0t2YXys63TWuA6Zz28aTDhGlQ1SFjgD3OiOXEoN/Vf1d+KTnDNj9Xh0Zd6e8QYGmsop+3djWa0L/LpEbdVFAqY4RJVlqrL8j2aYzxM1YUJfgDfONujkBFFwZc9VEAcDKUi8KKmTTK51pY6kF5XAOrU=
Message-ID: <466D04F8.8030304@day.com>
Date: Mon, 11 Jun 2007 10:16:56 +0200
From: Marcel Reutegger <marcel.reutegger@day.com>
User-Agent: Thunderbird 1.5.0.12 (Windows/20070509)
MIME-Version: 1.0
To: dev@jackrabbit.apache.org
Subject: Re: Optimize search performance
References: <f4bc62$buv$1@sea.gmane.org>
In-Reply-To: <f4bc62$buv$1@sea.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Marcel Reutegger <mreutegg@gmail.com>

Hi Christoph,

Christoph Kiehl wrote:
> I had a look at the search related code during the last days, because we 
> need better performance for range queries on date fields as well as for 
> sorting by date fields. These are my thoughts so far:
> 
> 1. Wouldn't it make sense to exclude the index for the "jcr:system" tree 
> (which is located at repository/index by default) if the query to 
> execute doesn't include items from the "jcr:system" tree.
> Take for example a query like "my:app//element(*, foo:bar)". This query 
> only searches for nodes located under "my:app" which excludes nodes from 
> "jcr:system" and therefore doesn't need to search in the "jcr:system" 
> index.

I think this is doable. Can you please file a jira issue about this?

> As the "jcr:system" might grow quite quickly if you create a lot 
> versions it might be worth to exclude it.
> I'm not sure though how hard it would be to find out if a query needs to 
> include the "jcr:system" index.

There are two relevant nodes in the query tree to find that out.

- what's the first location step and does it include the jcr:system tree? I 
think that's an easy one.
- does the query contain a jcr:deref node? If there is an intermediate result of 
a query may dereference into the jcr:system tree.

> 2. Lucene uses the FieldCaches to speed up sorting and range queries 
> which is exactly what we are after. Those FieldCaches are per IndexReader.
> Jackrabbit uses an IndexSearcher which searches on a single IndexReader 
> which is most likely to be an instance of CachingMultiReader. So on 
> every search which builds up a FieldCache this FieldCache instance is 
> associated with this instance of a CachingMultiReader. On successive 
> queries which operate on this CachingMultiReader you will get a 
> tremendous speedup for queries which can reuse  those associated 
> FieldCache instances.
> The problem is that Jackrabbit creates a new CachingMultiReader 
> _everytime_ one of the underlying indexes are modified. This means if 
> you just change _one_ item in the repository you will need to rebuild 
> all those FieldCaches because the existing FieldCaches are associated 
> with the old instance of CachingMultiReader.
> This does not only lead to slow search response times for queries which 
> contains range queries or are sorted by a field but also leads to 
> massive memory consumption (depending on the size of your indexes) 
> because there might be multiple instances of CachingMultiReaders in use 
> if you have a scenario where a lot of queries and item modifications are 
> executed concurrently.
> As far as I understand the solution is to use a MultiSearcher which uses 
> multiple IndexReaders. Since due to the merging strategy most of the 
> indexes are stable this means the FieldCaches can be used for a much 
> longer time.

this is all correct but does not work because. and you actually already found 
out why:

> I just tried to quickly modify SearchIndex to use a MultiSearcher with 
> multiple IndexReaders wrapped by IndexSearchers but wasn't successful 
> because somewhere in DescendantSelfAxisWeight the index readers are 
> required to implement HierarchyResolver which ReadOnlyIndexReader doesn't.

Using a multi searcher means that you must be able to execute a query on each of 
the index segments independently. this is not possible because hierarchy 
information is always spread across multiple segments. e.g. a node in one 
segment may reference a parent in another segment.

there's also another reason why a multi searcher is not the best solution. it 
requires that the fields of a returned FieldDoc contain the values of the 
indexed property. If there are lots of values to order the complete set of 
values needs to be read into memory. With the current implementation this is not 
needed because there is just a single FieldCache that uses integers instead of 
the real value. See class SharedFieldSortComparator [1]. the downside of this 
approach is that you cannot do a merge sort just using those integers.

a viable solution maybe is a combination of both approaches. use a FieldCache 
per index segment (which allows us to cache them for a longer period) but still 
use integer values for ordering of nodes within a segment. Then do a merge sort 
with a modified SharedFieldSortComparator that reads property values from the 
item state manager when nodes are compared across index segments. even though 
this requires reading property state, the performance shouldn't suffer too much, 
I think. the properties would be read anyway when the query result is iterated, 
so it shouldn't harm if they are read already during query execution.

regards
  marcel

[1] 
https://svn.apache.org/repos/asf/jackrabbit/tags/1.3/jackrabbit-core/src/main/java/org/apache/jackrabbit/core/query/lucene/SharedFieldSortComparator.java