Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CF96917448 for ; Wed, 3 Jun 2015 21:29:15 +0000 (UTC) Received: (qmail 32183 invoked by uid 500); 3 Jun 2015 21:29:10 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 32077 invoked by uid 500); 3 Jun 2015 21:29:10 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 32067 invoked by uid 99); 3 Jun 2015 21:29:10 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Jun 2015 21:29:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 085CB1A4497 for ; Wed, 3 Jun 2015 21:29:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.969 X-Spam-Level: ** X-Spam-Status: No, score=2.969 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id bCK3wgDS2IMt for ; Wed, 3 Jun 2015 21:29:09 +0000 (UTC) Received: from nk11p00mm-asmtp002.mac.com (nk11p00mm-asmtp002.mac.com [17.158.161.1]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 5A36F20C4B for ; Wed, 3 Jun 2015 21:29:08 +0000 (UTC) Received: from MACProW7SSD (0-46.static.highlandsfibernetwork.com [216.9.0.46]) by nk11p00mm-asmtp002.mac.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Dec 4 2014)) with ESMTPSA id <0NPE003OF0V08Z30@nk11p00mm-asmtp002.mac.com> for user@hadoop.apache.org; Wed, 03 Jun 2015 21:25:49 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.14.151,1.0.33,0.0.0000 definitions=2015-06-03_11:2015-06-03,2015-06-03,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=1 phishscore=0 adultscore=5 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1412110000 definitions=main-1506030265 From: Caesar Samsi To: user@hadoop.apache.org References: In-reply-to: Subject: Monitoring dashboard for Hadoop? Date: Wed, 03 Jun 2015 17:25:43 -0400 Message-id: <040a01d09e43$d95554f0$8bfffed0$@mac.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="----=_NextPart_000_040B_01D09E22.52451480" X-Mailer: Microsoft Outlook 14.0 Thread-index: AQInc/nkMlM9waxUy0ok+YW/qxZQapztny9Q Content-language: en-us This is a multipart message in MIME format. ------=_NextPart_000_040B_01D09E22.52451480 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello, I'm new to Hadoop and successfully built a fully distributed cluster of 3 nodes (1 master, 2 slaves) as a proof of concept. I have some questions below. Is there a dashboard to monitor the progress of a mapreduce computation? 1. I'm looking to ensure the computation gets allocated and uses the correct number of computation nodes 2. Monitor computation on the nodes (up/down/in-progress/completed) 3. If possible direct computation to specific group of nodes (depending on the computation priority). Similarly for HDFS 1. Ensure data file gets replicated to the correct number of nodes 2. If possible prioritize data replication (i.e. replicate data files that are accessed frequently to nodes that have better hardware, so some sort of load balancing distribution) Many Thanks, Caesar. ------=_NextPart_000_040B_01D09E22.52451480 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

I’m = new to Hadoop and successfully built a fully distributed cluster of 3 = nodes (1 master, 2 slaves) as a proof of concept. I have some questions = below.

 

Is there a dashboard to monitor the progress of a = mapreduce computation?

1.       = I’m looking to ensure the computation gets = allocated and uses the correct number of computation = nodes

2.       = Monitor computation on the nodes = (up/down/in-progress/completed)

3.       If = possible direct computation to specific group of nodes (depending on the = computation priority).

 

Similarly = for HDFS

1.       = Ensure data file gets replicated to the correct = number of nodes

2.       If = possible prioritize data replication (i.e. replicate data files that are = accessed frequently to nodes that have better hardware, so some sort of = load balancing distribution)

 

Many Thanks, = Caesar.

------=_NextPart_000_040B_01D09E22.52451480--