Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of erickerickson@gmail.com
 designates 209.85.134.187 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=bjKm2u8LCwV63ydYzt6815SZI5Hvufv22e0KAQGPXudWIvX1FyvLeIXIsxsU5P99LssoMAZkduEgCIgynK9u752OnsLfjj6GoQWsBVX8SQk+f7MKegZjL087ip7+nLOKPWzZd0Lkh7ocVxInA1CGsf9n7GaiPR39eDcxxJsdVtA=
Message-ID: <359a92830704081352i4c393868xeb9705547df3267d@mail.gmail.com>
Date: Sun, 8 Apr 2007 16:52:31 -0400
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Re[2]: Out of memory exception for big indexes
In-Reply-To: <f36128690704081158g3eb47189q2cd1d03bc89145df@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_16501_8231620.1176065551323"
References: <46162A72.50400@sirma.bg>
	 <359a92830704060730l76d38d22g44be77aa3bff3845@mail.gmail.com>
	 <Pine.LNX.4.58.0704061101020.15801@hal.rescomp.berkeley.edu>
	 <872e2d490704061203k6923dbd2rd6ffe0117a46837@mail.gmail.com>
	 <f36128690704072203i3ecf944bidbbb8769681b1ea4@mail.gmail.com>
	 <814127511.20070408213229@gmail.com>
	 <f36128690704081158g3eb47189q2cd1d03bc89145df@mail.gmail.com>

------=_Part_16501_8231620.1176065551323
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

It *is* a bit confusing, since every search is sorted, kinda....

Practically, a sorted query is one where you call one of the search
methods (on, say, Searcher) with a Sort object, which sorts
on one or more of the fields in your index (which ones are
used are specified in the (array of) Sort objects).

Searches that do NOT have a Sort object default to using
relevance ranking, which is not nearly so memory-intensive. This is,
after all, one float or so....

The difference is that the fields referenced in the Sort object
have to be read into memory and compared against all other
values, and the aggregate may be quite large memory-wise.

Erick

On 4/8/07, Nilesh Bansal <nileshbansal@gmail.com> wrote:
>
> On 4/8/07, Artem <abublic@gmail.com> wrote:
> > I must note that my patch only helps in lucene-OOM situations related to
> > _sorted_ queries. If this is your case than I think yes it will help.
> Probably a newbie question, but can you please explain what sorted
> queries mean? Is simple keyword search a sorted query?
>
> > In my app currently index is not so big, only 1mln docs. With the patch
> applied
> > sample query giving first 30 of 120,000 sorted results made memory
> consumption
> > jump from 18M to 20M according to jconsole.
> >
> > NB> It seems that there are some issues with this patch and that was the
> > NB> reason it is not yet in the main source tree. Can someone please
> > NB> summerize what are the downsides of using such an approach. It will
> be
> > NB> really good if Lucene had it in main source tree and a flag to turn
> ON
> > NB> or OFF this feature.
> >
> > First there's performance cost (for second and further queries with the
> > same IndexSearcher). In default implementation all the index values of
> sorted
> > field are cached during the first sorted search - this takes memory and
> time;
> > but next queries run fast if there still some memory left. My
> implementation
> > doesn't cache field values but loads them from respective documents on
> the fly -
> > so it's slower but takes less memory. The query mentioned took about 3s
> (with
> > rather small sorted fields values - about 20-100 chars).
> > There's a limitation also - my implementation requires sorted field to
> be
> > "stored" in index (Field.Store.YES in doc.add())
> >
> > NB> Bublic, can you tell me what exactly I need to do if I want to use
> this patch?
> >
> > You can include StoredFieldSortFactory class source file into your
> sources and
> > then use StoredFieldSortFactory.create(sortFieldName, sortDescending) to
> get
> > Sort object for sorting query.
> > StoredFieldSortFactory source file can be extracted from LUCENE-769
> patch or
> > from sharehound sources:
> http://sharehound.cvs.sourceforge.net/*checkout*/sharehound/jNetCrawler/src/java/org/apache/lucene/search/StoredFieldSortFactory.java
> >
> > Regards,
> > Artem
> >
> > NB> thanks
> > NB> Nilesh
> >
> > NB> On 4/6/07, Bublic Online <abublic@gmail.com> wrote:
> > >> Hi Ivan, Chris and all!
> > >>
> > >> I'm that contributor of LUCENE-769 and I recommend it too :)
> > >> OutOfMemory error was one of main reasons for me to make it.
> > >>
> > >> Regards,
> > >> Artem Vasiliev
> > >>
> > >> On 4/6/07, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> > >> >
> > >> >
> > >> > : The problem I suspect is the sorting. As I understand, Lucene
> > >> > : builds internal caches for sorting and I suspect that this is the
> root
> > >> > : of your problem. You can test this by trying your problem queries
> > >> > : without sorting.
> > >> >
> > >> > if Sorting really is the cause of your problems, you may want to
> try out
> > >> > this patch...
> > >> >
> > >> > https://issues.apache.org/jira/browse/LUCENE-769
> > >> >
> > >> > ...it *may* be advantageous in situations where memory is your most
> > >> > constrained resource, and you are willing to sacrifice speed for
> sorting
> > >> > ... it looks promising to me, but there haven't been any convincing
> > >> > usecases/benchmarks of people finding it beneficial (other then the
> > >> > original contributor)
> > >> >
> > >> > if you do try it, please post your comments in the issue.
> > >> >
> > >> >
> > >> >
> > >> > -Hoss
> > >> >
> > >> >
> > >> >
> ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >> >
> > >> >
> > >>
> >
> >
> >
> >
> >
> > --
> > Best regards,
> >  Artem                            mailto:abublic@gmail.com
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> Nilesh Bansal.
> http://queens.db.toronto.edu/~nilesh/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_16501_8231620.1176065551323--