Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 734D19AFA for ; Wed, 1 Feb 2012 22:10:46 +0000 (UTC) Received: (qmail 73658 invoked by uid 500); 1 Feb 2012 22:10:45 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 73616 invoked by uid 500); 1 Feb 2012 22:10:44 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 73608 invoked by uid 99); 1 Feb 2012 22:10:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Feb 2012 22:10:44 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.91.2.13] (HELO smtp-outbound-2.vmware.com) (208.91.2.13) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Feb 2012 22:10:36 +0000 Received: from sc9-mailhost1.vmware.com (sc9-mailhost1.vmware.com [10.113.161.71]) by smtp-outbound-2.vmware.com (Postfix) with ESMTP id 7CD2C2852B for ; Wed, 1 Feb 2012 14:10:14 -0800 (PST) Received: from sc9-exht06.vmware.com (sc9-exht06.vmware.com [10.113.190.36]) by sc9-mailhost1.vmware.com (Postfix) with ESMTP id 78F0C182BD for ; Wed, 1 Feb 2012 14:10:14 -0800 (PST) Received: from exch-mbx-112.vmware.com ([10.113.190.112]) by sc9-exht06.vmware.com ([10.113.190.36]) with mapi; Wed, 1 Feb 2012 14:10:14 -0800 From: Jeffrey Buell To: "hdfs-user@hadoop.apache.org" Date: Wed, 1 Feb 2012 14:10:14 -0800 Subject: RE: Testing rack awareness Thread-Topic: Testing rack awareness Thread-Index: AczhK69EvhZyBjnbSbioiLzPvXXR/QAAY0EA Message-ID: <71707C6AD2C02B4087F9E1BCA9EC816513AF11F92D@exch-mbx-112.vmware.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_71707C6AD2C02B4087F9E1BCA9EC816513AF11F92Dexchmbx112vmw_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_71707C6AD2C02B4087F9E1BCA9EC816513AF11F92Dexchmbx112vmw_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable "rack awareness" is a bit of a misnomer. Hadoop is not aware of racks in t= he sense that it can figure out your network topology. Instead, you need t= o write a script that tells Hadoop which machines belong to which racks. F= or example: http://wiki.apache.org/hadoop/topology_rack_awareness_scripts.= You can tell if your script is working when "default-rack" is replaced by= one of your rack names. Jeff From: Jain, Prem [mailto:Premanshu.Jain@netapp.com] Sent: Wednesday, February 01, 2012 1:55 PM To: hdfs-user@hadoop.apache.org Subject: Testing rack awareness Is there a way to check if the rack awareness is working ? I seem to configure it right, but when I run a fsck on my files written, i= t still shows "Number of racks: 1" What I am looking for is a simple command which can tell me if rack awarene= ss is working or not ? Thanks in advance ---------------------------------------------------------------------------= --------------------------------------- root@namenode]# hadoop fsck /user/file -files -blocks -locations -racks FSCK started by root from /10.61.97.224 for path /user/file at Wed Feb 01 = 14:36:42 EST 2012 /user/file bytes, 1 block(s): OK 0. blk_-2693119624313554881_407726 len=3D18483674 repl=3D2 [/default-rack/1= 0.61.97.214:50010, /default-rack/10.61.97.217:50010] Status: HEALTHY Total size: 18483674 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 18483674 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 2 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 8 Number of racks: 1 FSCK ended at Wed Feb 01 14:36:42 EST 2012 in 0 milliseconds The filesystem under path '/user/file ' is HEALTHY --_000_71707C6AD2C02B4087F9E1BCA9EC816513AF11F92Dexchmbx112vmw_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

“rack awareness” is a bit of a misnomer.  Ha= doop is not aware of racks in the sense that it can figure out your network= topology.  Instead, you need to write a script that tells Hadoop whic= h machines belong to which racks.  For example:  http://wiki.apach= e.org/hadoop/topology_rack_awareness_scripts.  You can tell if you= r script is working when “default-rack” is replaced by one of y= our rack names.

 

Jeff

 

From: Jain, Prem [mailto:Premanshu.Jain@netapp.com]
Se= nt: Wednesday, February 01, 2012 1:55 PM
To: hdfs-user@hadoop= .apache.org
Subject: Testing rack awareness

=

 

Is there a way= to check if the rack awareness is working ?

I seem to configure it right, but  when I run a fsck on my files = written, it still shows  “Number o= f racks:           &= nbsp;   1”

What I am looking for is a simple command which can= tell me if rack awareness is working or not ?

 

 

Thanks in advance<= /o:p>

&nb= sp;

----= ---------------------------------------------------------------------------= -----------------------------------

 

root@namenode]# hadoop fsck /user/file=  -files -blocks -locations -racks

FSCK started by root from /10.61.97.22= 4 for path /user/file  at Wed Feb 01 14:36:42 EST 2012

/user/file  b= ytes, 1 block(s):  OK

0. blk_-2693119624313554881_407726 len=3D18483674 r= epl=3D2 [/default-rack/10.61.97.214:50010, /default-rack/10.6= 1.97.217:50010]

 

Status: HEALTHY

Total size:    18483674 B

Total= dirs:    0

Total files:   1

Total blocks (validated):&n= bsp;     1 (avg. block size 18483674 B)

Minimally replicat= ed blocks:   1 (100.0 %)

Over-replicated blocks:   &n= bsp;    0 (0.0 %)

= Under-replicated blocks:   &nb= sp;   0 (0.0 %)

Mis-replicated blocks:     =     0 (0.0 %)

Default replication factor:    2

Aver= age block replication:     2.0

Corrupt blocks:  =             &nb= sp; 0

Missing replicas:         &= nbsp;    0 (0.0 %)

Number of data-nodes:    = ;      8

Number of racks:    &nbs= p;          1

FSCK ended at Wed F= eb 01 14:36:42 EST 2012 in 0 milliseconds

 

 

The filesystem under path '/= user/file  ' is HEALTHY

 <= /p>

 

 

= --_000_71707C6AD2C02B4087F9E1BCA9EC816513AF11F92Dexchmbx112vmw_--