lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: is this the right way to go?
Date Wed, 09 Jun 2010 20:55:44 GMT
Doing your own sorting is certainly an acceptable thing to do, and for
low numbers of hits might even be "the right way".  There are also
some tips and tricks that you can use to reduce Lucene's memory usage
for sorting such as using NumericField or splitting the sortable
field(s) into chunks, e.g. indexing year/month/day/hh/mm as separate
fields.


--
Ian.


On Wed, Jun 9, 2010 at 9:22 PM, fujian <fujian.z.yang@nokia.com> wrote:
>
>
> Hello,
>
> We are using lucene 2.9.0. and ran into OutOfMemory error when sorting on a
> highly unique field on a big index. After doing some research we learned
> that lucene will load the sort field value for all documents into memory to
> do sorting, and ended up with the OutOfMemory if the index is too big.
>
> The problem is no matter how much heap size you allocated for jvm, you will
> run into this problem eventually when your index is big enough and the sort
> field value is unique enough regardless how small amount of documents
> actually match the query ...
>
> This is a bit ... annoying. I only want to sort the 10 docs that match the
> query, why I need to load millions of field values into memory?
>
> Then we are thinking to disable the sort from lucene search and do the
> sorting ourselves. that is, after getting all matching docs from
> search(since we know the matching docs won't be too much), we do the sorting
> "manually". This requires us write our own sorting methods and it needs to
> support multiple field sort.
>
> so my question is is this the right way to go? or maybe lucene/solr already
> has the similiar thing?
>
> Thank you very much,
> -Fujian
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/is-this-the-right-way-to-go-tp883464p883464.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message