lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Herou" <marcus.he...@tailsweep.com>
Subject Sort suggestion
Date Tue, 29 Jul 2008 19:05:45 GMT
Guys.

I've noticed many having trouble with sorting and OOM. Eventually they solve
it by throwing more memory at the problem.

Should'nt a solution which can sort on disk when neccessary be implemented
in core Lucene ?
Something like this:
http://www.codeodor.com/index.cfm/2007/5/10/Sorting-really-BIG-files/1194

Since you obviously know the result size you can calculate how much memory
is needed for the sort and if the calculated value s higher then a
configurable threshold an external on disk sort is performed and perhaps a
logging message which states something on a WARN level.

Just a thought since I'm about to implement something which could sort any
Comparable object but on disk.

Guess the Hadoop project have the perfect tools for this since everything
the mapred inputfiles are sorted, on disk and huge.

Kindly

//Marcus


-- 
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
http://blogg.tailsweep.com/

Mime
View raw message