Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: 98.139.253.105 is neither permitted
 nor denied by domain of daryn@yahoo-inc.com)
From: Daryn Sharp <daryn@yahoo-inc.com>
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Subject: Re: HDFS Federation address performance issue
Thread-Topic: HDFS Federation address performance issue
Thread-Index: AQHPHFOCUbF5JVgeWkCTwotH4Ms0GZqbAMYA
Date: Tue, 28 Jan 2014 18:53:06 +0000
Message-ID: <7DFE9BF1-B2F9-4C35-9A07-941DAA05D100@yahoo-inc.com>
References: 
 <CAKSPWFAkS_MGngOVqV+SBmhmhYGUoAmy-qkVw-hB8Mb+UvBwqQ@mail.gmail.com>
In-Reply-To: 
 <CAKSPWFAkS_MGngOVqV+SBmhmhYGUoAmy-qkVw-hB8Mb+UvBwqQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_7DFE9BF1B2F94C359A07941DAA05D100yahooinccom_"
MIME-Version: 1.0

--_000_7DFE9BF1B2F94C359A07941DAA05D100yahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi Anfernee,

You will achieve improved performance with federation only if you stripe fi=
les across the multiple NNs.  Federation basically shares DN storage with m=
ultiple NNs with the expectation the namespace load will be distributed acr=
oss the multiple NNs.  If everything writes to the exact same parent direct=
ory then no benefit is achieved over a single NN.  You will need to partiti=
on your jobs so some write to one NN, other jobs write to the other NN(s).

I hope this helps!

Daryn

On Jan 28, 2014, at 12:04 PM, Anfernee Xu <anfernee.xu@gmail.com<mailto:anf=
ernee.xu@gmail.com>>
 wrote:

Hi,

Based on http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hd=
fs/Federation.html#Key_Benefits, the overall performance can be improved by=
 federation, but I'm not sure federation address my usercase, could someone=
 elaborate it?

My usercase is I have one single NM and several DN, and I have bunch of con=
current MR jobs which will create new files(plan files and sub-directory) u=
nder the same parent directory, the questions are:

1) Will these concurrent writes(new file, plan files and sub-directory unde=
r the same parent directory) run in sequential because WRITE-once control g=
ovened by single NM?

I need this answer to estimate the necessity of moving to HDFS federation.

Thanks

--
--Anfernee


--_000_7DFE9BF1B2F94C359A07941DAA05D100yahooinccom_
Content-Type: text/html; charset="iso-8859-1"
Content-ID: <C62EEE0C00080748A5E63E2A86EE7DE2@yforest.corp.yahoo.com>
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; ">
<div>Hi Anfernee,</div>
<div><br>
</div>
<div>You will achieve improved performance with federation only if you stri=
pe files across the multiple NNs. &nbsp;Federation basically shares DN stor=
age with multiple NNs with the expectation the namespace load will be distr=
ibuted across the multiple NNs. &nbsp;If everything
 writes to the exact same parent directory then no benefit is achieved over=
 a single NN. &nbsp;You will need to partition your jobs so some write to o=
ne NN, other jobs write to the other NN(s).</div>
<div><br>
</div>
<div>I hope this helps!</div>
<div><br>
</div>
<div>Daryn</div>
<div><br>
<div>
<div>On Jan 28, 2014, at 12:04 PM, Anfernee Xu &lt;<a href=3D"mailto:anfern=
ee.xu@gmail.com">anfernee.xu@gmail.com</a>&gt;</div>
<div>&nbsp;wrote:</div>
<br class=3D"Apple-interchange-newline">
<blockquote type=3D"cite">
<div dir=3D"ltr">
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
Based on <a href=3D"http://hadoop.apache.org/docs/stable/hadoop-project-dis=
t/hadoop-hdfs/Federation.html#Key_Benefits">
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/Federa=
tion.html#Key_Benefits</a>, the overall performance can be improved by fede=
ration, but I'm not sure federation address my usercase, could someone elab=
orate it?<br>
<br>
</div>
My usercase is I have one single NM and several DN, and I have bunch of con=
current MR jobs which will create new files(plan files and sub-directory) u=
nder the same parent directory, the questions are:<br>
<br>
1) Will these concurrent writes(new file, plan files and sub-directory unde=
r the same parent directory) run in sequential because WRITE-once control g=
ovened by single NM?<br>
<br>
</div>
I need this answer to estimate the necessity of moving to HDFS federation.<=
br>
<br>
</div>
Thanks<br>
<div><br>
-- <br>
--Anfernee </div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>

--_000_7DFE9BF1B2F94C359A07941DAA05D100yahooinccom_--