Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of brianjriddle@gmail.com
 designates 64.233.182.193 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=YQdkzoXIFHjFrL78ge44h/JxYyuZe+p+P1OAcDGv9IO4tZwlLdGJY9NgZPhmvtUxLOwJc0E4hStps1M2Plqh3o+ySmqbn7YTreyboQRYmABYcnTqfIhVUFZDDPOkc34UDWThEWVEuQ0rvvJzLRqyE0BK+1gBvd1FFCCllGCg7og=
Message-ID: <516b77c50603152226s68771186pb0bdc6af9acd9768@mail.gmail.com>
Date: Thu, 16 Mar 2006 07:26:52 +0100
From: "Brian Riddle" <brianjriddle@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: FunctionQuery example request
In-Reply-To: <20060315191747.82758.qmail@web30415.mail.mud.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_2385_17599761.1142490412045"
References: <20060315191747.82758.qmail@web30415.mail.mud.yahoo.com>

------=_Part_2385_17599761.1142490412045
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hej Paul,

I have implemented the DistanceComparatorSource
> example from Lucene In Action (my Bible) and it works
> great. We are now in the situation where we have
> nearly a million documents in our index and the
> performance of this implementation has degraded.
>

I have had the same problem with DistanceComparatorSource from Lucene In
Action.  After doing some profiling
found that if you do not implement equals and hashcode in at least class
that implments SortComparatorSort
a memory leak is created.

The sorting api in lucene keeps a cache of SortComparators in
org.apache.lucene.search.FieldCacheImpl.
This cache is based on 3 things
1) IndexReader
2) Field you are sorting
3) Compartor you are using

If your compartor does not implement equals and hashcode your are getting
penalized twice as the int[] you are using is being created *everytime* the
sort is used and the internal cache in FieldCacheImpl grows overtime.

We had a similar degradation when using lucene until we implemented a equal=
s
and hascode in out SortComparatorSort.

We were using lucene-1.4.3 at the time and tried different combinations of
versions of java(1.4.2 compared to 1.5) and lucene. In our environment we
found that the best increase was by upgrading to lucene-1.9.1.
A little more info can be found here http://www.lucenebook.com/blog/errata/

/Brian

------=_Part_2385_17599761.1142490412045--