hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Brune" <bbr...@decarta.com>
Subject RE: How to find out whether a node is Overloaded from Cpu utilization ?
Date Tue, 17 Jan 2012 16:46:28 GMT

Hi,

The significant factor in cluster loading is memory, not CPU.  Hadoop views the cluster only
with respect to memory and cares not about CPU utilization or Disk saturation.  If you run
too many TaskTrackers, you risk memory overcommit where the Linux OOM will come out of the
closet and randomly kill processes which will ultimately take out the box.  Disk saturation
is another issue that will contribute to datanode timeouts, which lead to a downward spiral
where the namenode will think the datanode is down and start 
replicating its blocks adding to the overall I/O loading contributing to other datanode timeouts.
 If your datanode CPU's are pegged then this contributes as well, but not as much.
Generally, if you carve up your cluster properly, you won't have CPU over utilization issues,
quite the opposite.

-Bill

-----Original Message-----
From: Amandeep Khurana [mailto:amansk@gmail.com]
Sent: Mon 1/16/2012 10:21 PM
To: common-user@hadoop.apache.org
Cc: hadoop-user@lucene.apache.org
Subject: Re: How to find out whether a node is Overloaded from Cpu utilization ?
 
Arun,

I don't think you'll hear a fixed number. Having said that, I have seen CPU
being pegged at 95% during jobs and the cluster working perfectly fine. On
the slaves, if you have nothing else going on, Hadoop only has TaskTrackers
and DataNodes. Those two daemons are relatively light weight in terms of
CPU for the most part. So, you can afford to let your tasks take up a high
%.

Hope that helps.

-Amandeep

On Tue, Jan 17, 2012 at 2:16 PM, ArunKumar <arunk786@gmail.com> wrote:

> Hi  Guys !
>
> When we get CPU utilization value of a node  in hadoop cluster, what
> percent
> value can be considered as overloaded ?
> Say for eg.
>
>        CPU utilization            Node Status
>             85%                      Overloaded
>              20%                        Normal
>
>
> Arun
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-find-out-whether-a-node-is-Overloaded-from-Cpu-utilization-tp3665289p3665289.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>


Mime
View raw message