Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Markus Jelsma <markus.jelsma@openindex.io>
Reply-To: markus.jelsma@openindex.io
Organization: Openindex
To: solr-user@lucene.apache.org
Subject: Calculating memory consumption FVH
Date: Sun, 12 Jun 2011 01:27:18 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.35-29-generic; KDE/4.5.5; i686; ; )
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201106120127.18740.markus.jelsma@openindex.io>

Hi,

How can i calculate the memory requirements for highlighting documents where 
the variables are number of documents to highlight and the total size of the 
documents in a single request? I can then multiply a worst-case result set 
with the expected concurrency.

For example, with a _limited_ heap highlighting works perfectly well and is 
only CPU-bound, which is clearly visible when highlighting a result set with 
many very large documents. When such a worst-case query is fired a lot of times 
concurrently Solr is guaranteed to run out of memory.

So, now i'm looking for a method to calcualte per-request memory consumption 
for set-ups with FVH and all term* settings enabled on the highlighted field in 
schema.

Thanks,