hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Lehene <cleh...@adobe.com>
Subject Re: High OS Load Numbers when idle
Date Wed, 18 Aug 2010 07:26:47 GMT
George, 

Did you try to monitor the CPU in real-time to see any patterns? Does it do any CPU spikes?
Even so, the load averages should be matched against the number of cores/hyper threads. How
many cores do you see in top? (I personally prefer htop to get a cluster overview in realtime).
If you have 2 CPUs with 2 cores each the maximum load you should have would be 4. However
if the cluster is idle it should go as low as 0.2 - 0.3.  If you don't see the real-time utilization
I'd check if Ganglia is set up correctly.

Cosmin 


On Aug 18, 2010, at 1:11 AM, George Stathis wrote:

> No takers? :-) Am I missing something too obvious?
> 
> On Tue, Aug 17, 2010 at 2:03 PM, George Stathis <gstathis@gmail.com> wrote:
> 
>> Hello,
>> 
>> We have just setup a new cluster on EC2 using Hadoop 0.20.2 and HBase
>> 0.20.3. Our small setup as of right now consists of one master and four
>> slaves with a replication factor of 2:
>> 
>> Master: xLarge instance with 2 CPUs and 17.5 GB RAM - runs 1 namenode, 1
>> secondarynamenode, 1 jobtracker, 1 hbasemaster, 1 zookeeper (uses its' own
>> dedicated EMS drive)
>> Slaves: xLarge instance with 2 CPUs and 17.5 GB RAM each - run 1 datanode,
>> 1 tasktracker, 1 regionserver
>> 
>> We have also installed Ganglia to monitor the cluster stats as we are about
>> to run some performance tests but, right out of the box, we are noticing
>> high system loads (especially on the master node) without any activity
>> happening on the clister. Of course, the CPUs are not being utilized at all,
>> but Ganglia is reporting almost all nodes in the red as the 1, 5 an 15
>> minute load times are all above 100% most of the time (i.e. there are more
>> than two processes at a time competing for the 2 CPUs time).
>> 
>> Question1: is this normal?
>> Question2: does it matter since each process barely uses any of the CPU
>> time?
>> 
>> Thank you in advance and pardon the noob questions.
>> 
>> -GS
>> 


Mime
View raw message