hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject RE: Swap requirements
Date Wed, 25 Mar 2015 20:55:15 GMT
Hi,

I do not think DataNodes (DN) require any swapping as swap is only used for paging in case
of running out of memory.

NameNodes (NN) behave like an in-memory database IMDB (like Oracle TimesTen) and uses a disk
file system called the FsImage to load the metadata as startup. For the remaining period until
HDFS is shutdown or otherwise NN will use the in memory cache to access metadata. Now the
only occasion I see the swap space will be used is when NN runs out of memory and start swapping
to disk. That is using swap space like memory which is undesirable. Not very clear to me.

HTH

Mich Talebzadeh

http://talebzadehmich.wordpress.com

Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache

NOTE: The information in this email is proprietary and confidential. This message is for the
designated recipient only, if you are not the intended recipient, you should destroy it immediately.
Any information in this message shall not be understood as given or endorsed by Peridale Ltd,
its subsidiaries or their employees, unless expressly so stated. It is the responsibility
of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd,
its subsidiaries nor their employees accept any responsibility.


-----Original Message-----
From: Abdul I Mohammed [mailto:oracle.blog3@gmail.com] 
Sent: 25 March 2015 20:45
To: HDP mailing list
Subject: Swap requirements

Hello all,

I am trying to figure out what are the swap requirements for name node and data nodes?  Some
vendor docs that I read says to set vm.swappiness to 0 which is telling kernel to not to use
swap?  

What is the default settings the community is using for there clusters?  

Also does the below parameter uses swap memory or regular memory in yarn?
Yarn.nodemanager.Vmem-pmem-ratio


Mime
View raw message