Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of ronymin@dgist.ac.kr designates
 114.71.99.27 as permitted sender)
Message-ID: <7993692.1426927997462.JavaMail.root@webmail1>
Date: Sat, 21 Mar 2015 17:53:17 +0900 (KST)
From: =?UTF-8?B?64Ko7Jyk66+8?= <ronymin@dgist.ac.kr>
Reply-To: =?UTF-8?B?64Ko7Jyk66+8?= <ronymin@dgist.ac.kr>
To: hdfs-dev@hadoop.apache.org
Subject: Does "copyToLocal" not consider the block locality?
Mime-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_Part_30981_1205995330.1426927997463"

------=_Part_30981_1205995330.1426927997463
Content-Type: multipart/alternative;
	boundary="----=_Part_30982_1212850924.1426927997463"

------=_Part_30982_1212850924.1426927997463
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

 Hello everyone.

I have experienced a very strange situation
about HDFS operation.

 
I have a 1 master and 10 slaves cluster
environment.

 
When I put a file A into HDFS with
dfs.replication=10, I can see every block of the file A is replicated in every
node.

So, it is reasonable to think that HDFS
file reader can operate as local block reader when I want to read that file A.

 
However, when I execute hdfs dfs –copyToLocal
A /to/my/localDir, the file reading time is same as the case of
dfs.replication=1.

 
So, I moniter the network resources
especially read and write data.

Both two cases that dfs.replication={1, 10}
fully exploit network resources.. 

This means reading that file does not
consider the block location..

 
Is it reasonable operation of HDFS?

Then, what is the true meaning of data
locality in HDFS? (We all know about the data locality of map task..)

 
I want to know the reason of the same performance
between both two “copyToLocal” cases.

 
Thanks!Yoonmin 


// Yoonmin Nam


------=_Part_30982_1212850924.1426927997463
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<style type='text/css'>.TerraceMsg { font-size: 12px; font-family:Dotum, Arial, Verdana, Sans-Serif;} .TerraceMsg p { margin: 0; padding: 0;	line-height: 1.5;} .Bold { font-weight: bold; }</style><div class='TerraceMsg'><p>&nbsp;</p><p class="MsoNormal"><span lang="EN-US">Hello everyone.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">I have experienced a very strange situation
about HDFS operation.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">I have a 1 master and 10 slaves cluster
environment.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">When I put a file A into HDFS with
dfs.replication=10, I can see every block of the file A is replicated in every
node.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">So, it is reasonable to think that HDFS
file reader can operate as local block reader when I want to read that file A.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">However, when I execute hdfs dfs –copyToLocal
A /to/my/localDir, the file reading time is same as the case of
dfs.replication=1.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">So, I moniter the network resources
especially read and write data.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">Both two cases that dfs.replication={1, 10}
fully exploit network resources.. <o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">This means reading that file does not
consider the block location..<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">Is it reasonable operation of HDFS?<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">Then, what is the true meaning of data
locality in HDFS? (We all know about the data locality of map task..)<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">I want to know the reason of the same performance
between both two “copyToLocal” cases.<o:p></o:p></span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">Thanks!<o:p></o:p></span></p><p class="MsoNormal">Yoonmin</p><p>&nbsp;</p></div><br><br/><br/><table border=0 cellpadding=0 cellspacing=0 ><tr><td style='vertical-align: top;font-size:9pt;'>// <strong><em>Yoonmin Nam</em></strong><br /><br /></td></tr></table><br>
<div id='TMSMDN' style="background:url('https://mail.dgist.ac.kr:443/mail/receiveMDN.do?mdnData=Ge6zlVtu1CpzU%2FKMyIi6Odw4DEI%2FmkTaibT5ME8C6banKm7lZYOpj2%2FIWLtXPPdHsBR6MitaXY8c%0A1LJbLKI7aBsANp16Mhrft0ZdQJmuutu0j6q247%2BIRPm76EDuC173eUREwx1gPNEFU6Qtmn9dFVP4%0AO2NKRCpBnzcdl1eu2G8%3D%0A')"></div>
------=_Part_30982_1212850924.1426927997463--

------=_Part_30981_1205995330.1426927997463--