From solr-user-return-143652-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Tue Sep 11 20:43:51 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E73CC18067A for ; Tue, 11 Sep 2018 20:43:50 +0200 (CEST) Received: (qmail 94917 invoked by uid 500); 11 Sep 2018 18:43:49 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 94859 invoked by uid 99); 11 Sep 2018 18:43:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Sep 2018 18:43:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id B6275C2717 for ; Tue, 11 Sep 2018 18:43:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.102 X-Spam-Level: X-Spam-Status: No, score=-0.102 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=elyograg.org Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id uk4s8vg6UQda for ; Tue, 11 Sep 2018 18:43:46 +0000 (UTC) Received: from frodo.elyograg.org (frodo.elyograg.org [166.70.79.217]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 890A25F1D3 for ; Tue, 11 Sep 2018 18:43:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by frodo.elyograg.org (Postfix) with ESMTP id BB3BB7B2 for ; Tue, 11 Sep 2018 12:43:44 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=elyograg.org; h= content-language:content-transfer-encoding:content-type :content-type:in-reply-to:mime-version:user-agent:date:date :message-id:from:from:references:subject:subject:received :received; s=mail; t=1536691424; bh=ew77Q95FH1p3JY1pX46OsITPQDHV Y/DxRUkKe2Q0JKc=; b=KQXxbpYSLy8jK8QwzSW5QkgIMbqbI5i1kPfdLuJmuwdU vdEon1CHormKcxAe2XnqfmDc3H5OEihYyegU7ITPYWJXigvw6zsIyeNeEqg2uyUh GUDKvD91xC78ytzmwH4WUYQJMdaT4kx4JkpD3ARQYQihpVOP61d/2DYpcE14T9c= X-Virus-Scanned: Debian amavisd-new at frodo.elyograg.org Received: from frodo.elyograg.org ([127.0.0.1]) by localhost (frodo.elyograg.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id OTKEXtkdzi26 for ; Tue, 11 Sep 2018 12:43:44 -0600 (MDT) Received: from [192.168.1.114] (114.int.elyograg.org [192.168.1.114]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: elyograg@elyograg.org) by frodo.elyograg.org (Postfix) with ESMTPSA id 4E6AB751 for ; Tue, 11 Sep 2018 12:43:44 -0600 (MDT) Subject: Re: Solr RSIZE memory overusage To: solr-user@lucene.apache.org References: <1536685666.5107.11.camel@exiger.com> <1536689662.5107.25.camel@exiger.com> From: Shawn Heisey Message-ID: <3c9ff1cb-cb82-c3a7-0295-ce5d9001a804@elyograg.org> Date: Tue, 11 Sep 2018 12:43:44 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <1536689662.5107.25.camel@exiger.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US On 9/11/2018 12:14 PM, Boris Pasko wrote: >> Run top, press shift-M to sort by memory usage, then grab a > atop: http://oi68.tinypic.com/10pokkk.jpg > top: http://oi63.tinypic.com/msbpfp.jpg Looking at the second one: The SHR value is showing 90GB. Your Java process is in actuality only using in the ballpark of 9GB memory -- the difference between RES and SHR.  I have no idea why the SHR value goes so high sometimes, but it does.  This is a strange memory reporting anomaly encountered when using Java software.  The problem might be in the OS, or it might be in Java ... but there IS a reporting problem.  It does seem related to MMap, though.  You said you're using NRTCachingDirectoryFactory ... which *does* use MMap for all index file access. Looks like the amount of memory in the machine is too large for "top" to display the numbers correctly.  See the "+" sign for total memory, buff/cache/ and avail Mem. If we switch over to atop (a program that I did not know about) for a moment, you'll see that there is 113GB used by the disk cache.  So having 101GB (the RSIZE value for the Java process) is simply not possible.  That number is being reported incorrectly. It does look like you've got in the neighborhood of 600GB of index data.  Only 113GB of that data is cached.  For some use cases, this will be plenty of cached data for good performance. For others, it won't be anywhere near enough.  For *perfect* performance, there will be enough memory for ALL of the index data to fit into memory. Thanks, Shawn