lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Details on setting block parameters for Lucene41PostingsFormat
Date Sun, 11 Jan 2015 11:18:25 GMT
On Sat, Jan 10, 2015 at 7:58 PM, Tom Burton-West <tburtonw@umich.edu> wrote:
> Thanks Mike,
>
> We run our Solr 3.x indexing with 10GB/shard.  I've been testing Solr 4
> with 4,6, and 8GB for heap.  As of Friday night when the indexes were about
> half done (about 400GB on disk) only the 4GB had issues.  I'll find out on
> Monday if the other runs had issues.  If we can go from 10GB in Solr 3.x to
> 6GB with Solr 4.x, that will be a significant change.

OK.  It would be good to know where all your RAM is being consumed,
and how much of that is really the terms index: it ought to be a very
small part of it.

> With TermsIndexInterval we traded off less memory use for increased chance
> of disk seeks and more data to be read per seek (and if I remember right,
> that more data was scanned sequentially rather than binary searched.)
> What is the trade-off when increasing the block size?

It's exactly the same tradeoff: blocks will be larger, so there will
be fewer blocks that the terms index must reference (making it
smaller), but more scanning to find your exact term within a given
block.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message