Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: Long GC pauses while reading Solr docs using Cursor approach
To: solr-user@lucene.apache.org
References: <CAA70BoXCYskAf4dXGdRwmwKnB5SKPgpb43foSDPz9LK2yKmUZA@mail.gmail.com>
 <0e04d6c8-a3cc-3824-542d-e45024d7876a@elyograg.org>
 <C36D72F2-925B-491D-9FEA-03BF34D0613D@wunderwood.org>
 <CAA70BoXOJQ2Cywaw6_aBu6NnmgWHnZ6dOk+wPZa2vnSYuegaFg@mail.gmail.com>
 <CAF8TkC6URbxpn33Nm8N0n6DYSAB5=xF4etceP=ShHFNg5P-i-Q@mail.gmail.com>
 <CAA70BoWLuhcu0VTk=-Ua4jkN=GNsZZn1ZVUrk2Yj9ZEFW0rxcg@mail.gmail.com>
 <CAN4YXvf_NWEjqbmzwxgKj-oQ1bcQUajVVXe==snZJ7iKv6SHDA@mail.gmail.com>
 <CAA70BoVR9wiN2Zd4w+U=Yy=NzqEtatVMB-g3CSDLyERp_vfR-A@mail.gmail.com>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <a9fc1473-a4ac-84f0-99ca-90e02aa97e40@elyograg.org>
Date: Wed, 12 Apr 2017 20:36:25 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <CAA70BoVR9wiN2Zd4w+U=Yy=NzqEtatVMB-g3CSDLyERp_vfR-A@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 13 Apr 2017 02:36:46 -0000

On 4/12/2017 5:19 PM, Chetas Joshi wrote:
> I am getting back 100K results per page.
> The fields have docValues enabled and I am getting sorted results based on "id" and 2 more fields (String: 32 Bytes and Long: 8 Bytes).
>
> I have a solr Cloud of 80 nodes. There will be one shard that will get top 100K docs from each shard and apply merge sort. So, the max memory usage of any shard could be 40 bytes * 100K * 80 = 320 MB. Why would heap memory usage shoot up from 8 GB to 17 GB?

From what I understand, Java overhead for a String object is 56 bytes
above the actual byte size of the string itself.  And each character in
the string will be two bytes -- Java uses UTF-16 for character
representation internally.  If I'm right about these numbers, it means
that each of those id values will take 120 bytes -- and that doesn't
include the size the actual response (xml, json, etc).

I don't know what the overhead for a long is, but you can be sure that
it's going to take more than eight bytes total memory usage for each one.

Then there is overhead for all the Lucene memory structures required to
execute the query and gather results, plus Solr memory structures to
keep track of everything.  I have absolutely no idea how much memory
Lucene and Solr use to accomplish a query, but it's not going to be
small when you have 200 million documents per shard.

Speaking of Solr memory requirements, under normal query circumstances
the aggregating node is going to receive at least 100K results from
*every* shard in the collection, which it will condense down to the
final result with 100K entries.  The behavior during a cursor-based
request may be more memory-efficient than what I have described, but I
am unsure whether that is the case.

If the cursor behavior is not more efficient, then each entry in those
results will contain the uniqueKey value and the score.  That's going to
be many megabytes for every shard.  If there are 80 shards, it would
probably be over a gigabyte for one request.

Thanks,
Shawn