Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of zhengda1936@gmail.com
 designates 209.85.216.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=OOzQR/ZjFscUlj5N1EoEeBCOqA6Y7oF9PmJG9IKFw2K33tQT2eDyQr/wjOwDyvgG/e
         kV3uu3h2j5L6MlF5EIoBBYnZzKN949bovvcU3nogTxvkhzM+DoganTuccLDHg+daQt09
         e+tWKWjcDJGysfYN2QRwUGX3QJ0PxO9qsHaiM=
Message-ID: <4CDC4977.4000608@gmail.com>
Date: Thu, 11 Nov 2010 14:52:23 -0500
From: Da Zheng <zhengda1936@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: monitor the hadoop cluster
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hello,

I wrote a MapReduce program and ran it on a 3-node hadoop cluster, but 
its running time varies a lot, from 2 minutes to 3 minutes. I want to 
understand how time is used by the map phase and the reduce phase, and 
hope to find the place to improve the performance.

Also the current input data is sorted, so I wrote a customized 
partitioner to reduce the data shuffling across the network. I need some 
means to help me observe the data movement.

I know hadoop community developed chukwa for monitoring, but it seems 
very immature right now. I wonder how people monitor hadoop cluster 
right now. Is there a good way to solve my problems listed above?

Thanks,
Da