Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of ian.lea@gmail.com designates
 209.85.216.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=bC3WRQ8C3PFdcPyppNkedzfsK+QwRLpvcSLdoEugtbwASwxpru7/sHOIBTICqgoyVr
         55WFbqjl7AmEf8VtdhQJ4GuYrw4kcdQExilUyBDpVkJg5F6ag+ZG2lTzq2zmW4dWLr2k
         pbB26ujyzMTnzlTcy6Qm2mzefHpx1JEawwCyU=
MIME-Version: 1.0
In-Reply-To: 
 <AA27F54EFDA0CE43B33EBDB7796323740CA2761A19@PUNITPMBX02.ad.infosys.com>
References: 
 <AA27F54EFDA0CE43B33EBDB7796323740CA26DAB52@PUNITPMBX02.ad.infosys.com>
 <AANLkTikjV11DwFig21TQqGVokB5g2Hy3Cd9QhuM8mxFh@mail.gmail.com>
 <AA27F54EFDA0CE43B33EBDB7796323740CA2760D60@PUNITPMBX02.ad.infosys.com>
 <AANLkTink0QeB+2PpzwFjztZg_1VxwoEjjVrFA9SPwWfj@mail.gmail.com>
 <AA27F54EFDA0CE43B33EBDB7796323740CA2761A19@PUNITPMBX02.ad.infosys.com>
From: Ian Lea <ian.lea@gmail.com>
Date: Wed, 25 Aug 2010 09:31:55 +0100
Message-ID: <AANLkTimgzckcB1zr7VdcU8cuvHNJmA8PL7EtK01+QejS@mail.gmail.com>
Subject: Re: Sorting a Lucene index
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

1 billion i.e. 1,000,000,000?

Either buy more RAM, lots more RAM, or skip lucene sorting and do your
own sorting for the top n hits.  You might also want to look into
sharding/distributing your index.


--
Ian.


On Wed, Aug 25, 2010 at 6:16 AM, Shelly_Singh <Shelly_Singh@infosys.com> wr=
ote:
> I have 1 bln documents to sort. So, that would mean ( 8 bln bytes =3D=3D =
8GB RAM) bytes.
> All I have is 8 GB on my machine, so I do not think approach would work.
>
> Any other options?
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Thursday, August 19, 2010 7:18 PM
> To: java-user@lucene.apache.org
> Subject: Re: Sorting a Lucene index
>
> You haven't yet told us how many documents you're talking about here, so
> it's
> hard to have a good idea of what solutions are. That said, I'd just try
> sorting first.
> The sorting cache size will be something like (sizeof(int or long)) *
> (number of documents).
> Measure (remember to measure the response after query warmups, the first
> few will be slower because they fill up the cache) THEN fix iff there's a
> problem.
>
> And just forget the idea of inserting your documents in the correct order
> <G>. You
> stated that the documents come in in random order. Document IDs are assig=
ned
> internally to Lucene, and monotonically increasing. I sure don't see how =
you
> can
> reconcile those two things...
>
> But again, just try it with sorting on the numeric field and only fix thi=
ngs
> if you have a
> problem. Lots of work has been put into making Lucene fast, by very brigh=
t
> people. See
> if they've already solved your problem for you...
>
> Best
> Erick.
>
>
> On Thu, Aug 19, 2010 at 1:51 AM, Shelly_Singh <Shelly_Singh@infosys.com>w=
rote:
>
>> Hi Anshum,
>>
>> I require sorted results for all my queries and the field on which I nee=
d
>> sorting is fixed; so this lead to me the idea of storing in sorted order=
 to
>> avoid sorting cost with every query.
>>
>> Thanks and Regards,
>>
>> Shelly Singh
>> Center For KNowledge Driven Information Systems, Infosys
>> Email: shelly_singh@infosys.com
>> Phone: (M) 91 992 369 7200, (VoIP)2022978622
>>
>> -----Original Message-----
>> From: Anshum [mailto:anshumg@gmail.com]
>> Sent: Wednesday, August 18, 2010 5:21 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: Sorting a Lucene index
>>
>> Hi Shelly,
>> The search results so returned are sorted either by relevance, index ord=
er,
>> stored field, or custom order.
>> As you are saying that you would not be able to maintain the index order=
,
>> =A0you would have to do the sort at run time.
>> Sorting on a stored field is not costly and you may use it comfortably.
>> btw,
>> are you facing any issues in sort time or is it a presumption?
>>
>> --
>> Anshum Gupta
>> http://ai-cafe.blogspot.com
>>
>>
>> On Wed, Aug 18, 2010 at 5:12 PM, Shelly_Singh <Shelly_Singh@infosys.com
>> >wrote:
>>
>> > Hi,
>> >
>> > I have a Lucene index that contains a numeric field along with certain
>> > other fields. The order of incoming documents is random and
>> un-predictable.
>> > As a result, while creating an index, I end up adding docs in random
>> order
>> > with respect to the numeric field value.
>> >
>> > For example, documents may be added in following order:
>> > 12,y,d
>> > 100,o,p
>> > 1,x,y
>> > 23,u,i
>> > 31,v,m
>> > 22,b,m
>> > 109,k,l
>> >
>> > My requirement is that at search time, I want the documents in order o=
f
>> the
>> > numeric field.
>> > One, option is to do a score/sort on the numeric field.
>> > But, this may be a costly operation.
>> >
>> > Hence, I am trying to find if there is some way, such that, my stored
>> index
>> > is sorted by itself.
>> >
>> > Please help.
>> >
>> > Thanks and Regards,
>> >
>> > Shelly Singh
>> > Center For KNowledge Driven Information Systems, Infosys
>> > Email: shelly_singh@infosys.com<mailto:shelly_singh@infosys.com>
>> > Phone: (M) 91 992 369 7200, (VoIP)2022978622
>> >
>> >
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org