Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 62AEB176D8 for ; Wed, 25 Mar 2015 23:16:43 +0000 (UTC) Received: (qmail 99487 invoked by uid 500); 25 Mar 2015 23:16:37 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 99368 invoked by uid 500); 25 Mar 2015 23:16:37 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 99358 invoked by uid 99); 25 Mar 2015 23:16:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2015 23:16:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.138.144.103] (HELO sulu.netzoomi.net) (83.138.144.103) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2015 23:16:11 +0000 Received: from vulcan.netzoomi.net (unknown [212.100.249.54]) by sulu.netzoomi.net (Postfix) with ESMTP id 8111C6A4BB4 for ; Wed, 25 Mar 2015 23:14:48 +0000 (GMT) X-Envelope-From: Received: from vista (cpc7-seve18-2-0-cust228.13-3.cable.virginm.net [86.19.240.229]) by vulcan.netzoomi.net (Postfix) with ESMTPA id 573A11248766 for ; Wed, 25 Mar 2015 23:14:48 +0000 (GMT) From: "Mich Talebzadeh" To: References: In-Reply-To: Subject: RE: Swap requirements Date: Wed, 25 Mar 2015 23:14:35 -0000 Message-ID: <040e01d06751$75d723f0$61856bd0$@co.uk> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_040F_01D06751.75D723F0" X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AdBnUDiyMtqEapfWTNOJymHGAmjfNwAACXVg Content-Language: en-gb X-domainnameshop-MailScanner-Information: Please contact the ISP for more information X-domainnameshop-MailScanner-ID: 8111C6A4BB4.A233E X-domainnameshop-MailScanner: Found to be clean X-domainnameshop-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=1.275, required 5, autolearn=disabled, HTML_MESSAGE 0.00, RDNS_NONE 1.27) X-domainnameshop-MailScanner-SpamScore: 1 X-domainnameshop-MailScanner-From: mich@peridale.co.uk X-domainnameshop-MailScanner-Watermark: 1427930090.9689@Q/kv6vgUaT3ZX9xE2VrMAw X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No This is a multi-part message in MIME format. ------=_NextPart_000_040F_01D06751.75D723F0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Yes I believe that is the case. =20 This is very common from days of Max shared memory on Solaris etc. Large = applications tend to have processes with large virtual address spaces. = This is typically the result of attaching to large shared memory = segments used by applications and large copy-on-write (COW) segments = that get mapped but sometimes never actually get touched. The net effect = of this is that on the host supporting multiple applications, the = virtual address space requirements will grow to be quite large, = typically exceeding the physical memory. Consequently, a fair amount of = swap disk needs to be configured to support these applications with = large virtual address space running concurrently. In the old days this = would typically be 1.2* shared memory segment or RAM =20 =20 HTH =20 Mich Talebzadeh =20 http://talebzadehmich.wordpress.com =20 Publications due shortly: Creating in-memory Data Grid for Trading Systems with Oracle TimesTen = and Coherence Cache =20 NOTE: The information in this email is proprietary and confidential. = This message is for the designated recipient only, if you are not the = intended recipient, you should destroy it immediately. Any information = in this message shall not be understood as given or endorsed by Peridale = Ltd, its subsidiaries or their employees, unless expressly so stated. It = is the responsibility of the recipient to ensure that this email is = virus free, therefore neither Peridale Ltd, its subsidiaries nor their = employees accept any responsibility. =20 From: max scalf [mailto:oracle.blog3@gmail.com]=20 Sent: 25 March 2015 23:05 To: user@hadoop.apache.org Subject: Re: Swap requirements =20 Thank you harsh. Can you please explain what you mean when u said "Just = simple virtual memory used by the process" ? Doesn't virtual memory = means swap? On Wednesday, March 25, 2015, Harsh J wrote: The suggestion (regarding swappiness) is not for disabling swap as much = as it is to 'not using swap (until really necessary)'. When you run a = constant memory-consuming service such as HBase you'd ideally want the = RAM to serve up as much as it can, which setting that swappiness value = helps do (the OS otherwise begins swapping way before its available = physical RAM is nearing full state). =20 The vmem-pmem ratio is something entirely else. The vmem of a process = does not mean swap space usage, just simple virtual memory used by the . = I'd recommend disabling YARN's vmem checks on today's OSes (but keep = pmem checks on). You can read some more on this at = http://www.quora.com/Why-do-some-applications-use-significantly-more-virt= ual-memory-on-RHEL-6-compared-to-RHEL-5 =20 On Thu, Mar 26, 2015 at 3:37 AM, Abdul I Mohammed = > wrote: Thanks Mith...any idea about Yarn.nodemanager.Vmem-pmem-ratio = parameter... If data nodes does not require swap then what about the above parameter? = What is that used for in yarn? =20 --=20 Harsh J ------=_NextPart_000_040F_01D06751.75D723F0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

Yes I = believe that is the case.

 

This is very = common from days of Max shared memory on Solaris etc. Large applications tend to have processes with = large virtual address spaces. This is typically the result of attaching = to large shared memory segments used by applications and large = copy-on-write (COW) segments that get mapped but sometimes never = actually get touched. The net effect of this is that on the host = supporting multiple applications, the virtual address space requirements = will grow to be quite large, typically exceeding the physical memory. = Consequently, a fair amount of swap disk needs to be configured to = support these applications =C2=A0with large virtual address space = running concurrently. In the old days this would typically be 1.2* = shared memory segment or RAM

 

 

HTH

 

M= ich Talebzadeh

<= o:p> 

h= ttp://talebzadehmich.wordpress.com

 

Publications due = shortly:

C= reating in-memory Data Grid for Trading Systems with Oracle TimesTen and = Coherence Cache

<= o:p> 

NO= TE: The information in this email is proprietary and confidential. This = message is for the designated recipient only, if you are not the = intended recipient, you should destroy it immediately. Any information = in this message shall not be understood as given or endorsed by Peridale = Ltd, its subsidiaries or their employees, unless expressly so stated. It = is the responsibility of the recipient to ensure that this email is = virus free, therefore neither Peridale Ltd, its subsidiaries nor their = employees accept any responsibility.

 

From:= max scalf = [mailto:oracle.blog3@gmail.com]
Sent: 25 March 2015 = 23:05
To: user@hadoop.apache.org
Subject: Re: Swap = requirements

 

Thank you = harsh.  Can you please explain what you mean when u said "Just = simple virtual memory used by the process" ?  Doesn't virtual = memory means swap?

On Wednesday, March 25, 2015, Harsh J <harsh@cloudera.com> = wrote:

The suggestion (regarding = swappiness) is not for disabling swap as much as it is to 'not using = swap (until really necessary)'. When you run a constant memory-consuming = service such as HBase you'd ideally want the RAM to serve up as much as = it can, which setting that swappiness value helps do (the OS otherwise = begins swapping way before its available physical RAM is nearing full = state).

 

The vmem-pmem ratio is something entirely else. The = vmem of a process does not mean swap space usage, just simple virtual = memory used by the . I'd recommend disabling YARN's vmem checks on = today's OSes (but keep pmem checks on). You can read some more on this = at http://www.quora.com/Why-do-some-applications-use-signi= ficantly-more-virtual-memory-on-RHEL-6-compared-to-RHEL-5<= /p>

 

On Thu, Mar 26, 2015 at 3:37 AM, Abdul I Mohammed = <oracle.blog3@gmail.com> wrote:

Thanks Mith...any idea about = Yarn.nodemanager.Vmem-pmem-ratio parameter...

If data nodes does = not require swap then what about the above parameter?  What is that = used for in yarn?



 

-- =

Harsh = J

------=_NextPart_000_040F_01D06751.75D723F0--