Return-Path: 
 <user-return-5825-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E82DD022
	for <apmail-hadoop-user-archive@minotaur.apache.org>;
 Thu,  7 Mar 2013 16:31:01 +0000 (UTC)
Received: (qmail 86957 invoked by uid 500); 7 Mar 2013 16:30:56 -0000
Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org
Received: (qmail 86650 invoked by uid 500); 7 Mar 2013 16:30:56 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 86621 invoked by uid 99); 7 Mar 2013 16:30:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Mar 2013 16:30:55 +0000
X-ASF-Spam-Status: No, hits=3.2 required=5.0
	tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_REMOTE_IMAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of j_mezazap@hotmail.com
 designates 65.55.111.93 as permitted sender)
Received: from [65.55.111.93] (HELO blu0-omc2-s18.blu0.hotmail.com)
 (65.55.111.93)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Mar 2013 16:30:48 +0000
Received: from BLU002-W66 ([65.55.111.71]) by blu0-omc2-s18.blu0.hotmail.com
 with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 7 Mar 2013 08:30:28 -0800
X-EIP: [3Ln/woeiebf0ka7WHM9Uw/0shVYFb0sA]
X-Originating-Email: [j_mezazap@hotmail.com]
Message-ID: <BLU002-W6629C58C272F1DB764189382E50@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_7a379ee4-62cb-42d4-8bf3-f7c03e444960_"
From: John Meza <j_mezazap@hotmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: RE: dfs.datanode.du.reserved
Date: Thu, 7 Mar 2013 08:30:27 -0800
Importance: Normal
In-Reply-To: 
 <CAOcnVr1PQVLJUJQF=AXrV9RUU1=wEOe4KrDKQa84tQksJds3zQ@mail.gmail.com>
References: <BLU002-W23066D43C50F1506B7192F282FA0@phx.gbl>
 <CAMNKze0iAkRHUs2OqVJQ3vjQ4FHhRx4BHBG2-b_RZn5RBsEnLg@mail.gmail.com>,<BLU002-W1474348863ED8179B14602682FA0@phx.gbl>
 <CAO6W-2cCYn4tC-fCHtWu6_aD3L38dvZ4c8h57ypdRsJM=2soJQ@mail.gmail.com>,<BLU002-W918ECB3F4DF45FC90AAFB982E40@phx.gbl>,<CAOcnVr1PQVLJUJQF=AXrV9RUU1=wEOe4KrDKQa84tQksJds3zQ@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 07 Mar 2013 16:30:28.0543 (UTC)
 FILETIME=[1431C4F0:01CE1B51]
X-Virus-Checked: Checked by ClamAV on apache.org

--_7a379ee4-62cb-42d4-8bf3-f7c03e444960_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Ok. I'll add my ideas and comments.

From: harsh@cloudera.com
Date: Thu=2C 7 Mar 2013 06:37:20 +0530
Subject: Re: dfs.datanode.du.reserved
To: user@hadoop.apache.org

Hey John=2C
Ideas=2C comments and patches are welcome on https://issues.apache.org/jira=
/browse/HDFS-1564 for achieving this!=0A=
=0A=


On Wed=2C Mar 6=2C 2013 at 9:56 PM=2C John Meza <j_mezazap@hotmail.com> wro=
te:
=0A=
=0A=
=0A=
=0A=
=0A=
Thanks for the reply. This sounds like it has potential=2C but also seems t=
o be a rather duct-tape type of work around. It would be nice if there was =
a mod to dfs.datanode.du.reserved that worked within Hadoop=2C so that woul=
d imply that hadoop was a little more certain to adhere to it. =0A=
=0A=

I understand that dfs.datanode.du.reserved defines reservd storage on each =
volume. I would like to give each volume a different reserved value.
An example: <name>dfs.datanode.du.reserved</name>=0A=
   <value>///hstore1/dfs/dn:161061273600=2C///hstore2/dfs/dn:53687091200</v=
alue>=0A=
=0A=
Or something similiar. thanks=20
John
Date: Wed=2C 6 Mar 2013 10:25:17 +0100
Subject: Re: dfs.datanode.du.reserved
From: dechouxb@gmail.com
=0A=
=0A=
To: user@hadoop.apache.org

Not that I know. If so you should be able to identify each volume and as of=
 now this isn't the case.
=0A=
=0A=
BUT it can be done without Hadoop knowing about it=2C at the OS level=2C us=
ing different partitions/mounts for datanode and jobtracker stuff. That sho=
uld solve your problem.
=0A=

Regards

Bertrand

On Mon=2C Mar 4=2C 2013 at 10:26 PM=2C John Meza <j_mezazap@hotmail.com> wr=
ote:
=0A=
=0A=
=0A=
=0A=
I'm probably not being clear.this seems to describe it: dfs.datanode.du.res=
erved  configured per-volume.=0A=
=0A=
=0A=
https://issues.apache.org/jira/browse/HDFS-1564

=0A=
=0A=
=0A=
thanksJohn
From: outlawdba@gmail.com
Date: Mon=2C 4 Mar 2013 15:37:36 -0500
Subject: Re: dfs.datanode.du.reserved
To: user@hadoop.apache.org=0A=
=0A=
=0A=


Possible to reserve 0 from various testing I have done yet that could cause=
 the obvious side effect of achieving zero disk space:) Have only tested in=
 development environment=2C however. Yet there are various tuning white pap=
ers and other benchmarks where the very same has been tested.=20
=0A=
=0A=
=0A=
=0A=
=0A=

Thanks.=20

On Mon=2C Mar 4=2C 2013 at 2:00 PM=2C John Meza <j_mezazap@hotmail.com> wro=
te:
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
the parameter: dfs.datanode.du.reserved is used to reserve disk space PER d=
atanode. Is it possible to reserve a different amount of disk space per DIS=
K?=0A=
=0A=
=0A=
=0A=
=0A=

thanks=0A=
=0A=
=0A=
=0A=
=0A=
John 		 	   		  =0A=
=0A=
=0A=
=0A=
=0A=
=0A=


--=20
Ellis R. Miller937.829.2380

=0A=
=0A=
=0A=

Mundo Nulla Fides =0A=
=0A=



=0A=
=0A=
=0A=
=0A=

=0A=


 		 	   		  =0A=

 		 	   		  =0A=


--=20
Harsh J=0A=
 		 	   		  =

--_7a379ee4-62cb-42d4-8bf3-f7c03e444960_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style><!--
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 12pt=3B
font-family:Calibri
}
--></style></head>
<body class=3D'hmmessage'><div dir=3D'ltr'>Ok. I'll add my ideas and commen=
ts.<br><br><div><div id=3D"SkyDrivePlaceholder"></div><hr id=3D"stopSpellin=
g">From: harsh@cloudera.com<br>Date: Thu=2C 7 Mar 2013 06:37:20 +0530<br>Su=
bject: Re: dfs.datanode.du.reserved<br>To: user@hadoop.apache.org<br><br><d=
iv dir=3D"ltr">Hey John=2C<div><br></div><div>Ideas=2C comments and patches=
 are welcome on&nbsp=3B<a href=3D"https://issues.apache.org/jira/browse/HDF=
S-1564" target=3D"_blank">https://issues.apache.org/jira/browse/HDFS-1564</=
a>&nbsp=3Bfor achieving this!</div>=0A=
=0A=
</div><div class=3D"ecxgmail_extra"><br><br><div class=3D"ecxgmail_quote">O=
n Wed=2C Mar 6=2C 2013 at 9:56 PM=2C John Meza <span dir=3D"ltr">&lt=3B<a h=
ref=3D"mailto:j_mezazap@hotmail.com" target=3D"_blank">j_mezazap@hotmail.co=
m</a>&gt=3B</span> wrote:<br>=0A=
=0A=
<blockquote class=3D"ecxgmail_quote" style=3D"border-left:1px #ccc solid=3B=
padding-left:1ex=3B">=0A=
=0A=
=0A=
<div><div dir=3D"ltr">Thanks for the reply. This sounds like it has potenti=
al=2C but also seems to be a rather duct-tape type of work around. It would=
 be nice if there was a mod to dfs.datanode.du.reserved that worked within =
Hadoop=2C so that would imply that hadoop was a little more certain to adhe=
re to it.&nbsp=3B<div>=0A=
=0A=
<br></div><div>I understand that dfs.datanode.du.reserved defines reservd s=
torage on each volume. I would like to give each volume a different reserve=
d value.</div><div><br></div><div>An example:<div><pre style=3D"word-wrap:b=
reak-word=3B"><span style=3D"white-space:pre-wrap=3B"><font size=3D"3"> &lt=
=3Bname&gt=3Bdfs.datanode.du.reserved&lt=3B/name&gt=3B</font>=0A=
   <font size=3D"3">&lt=3Bvalue&gt=3B///hstore1/dfs/dn:161061273600=2C</fon=
t></span><font size=3D"3"><span style=3D"white-space:pre-wrap=3Bfont-family=
:Calibri=2Csans-serif=3B">///hstore2/dfs/dn:</span><font face=3D"Calibri=2C=
 sans-serif"><span style=3D"white-space:pre-wrap=3B">53687091200</span></fo=
nt><span style=3D"white-space:pre-wrap=3Bfont-family:Calibri=2Csans-serif=
=3B">&lt=3B/value&gt=3B</span></font></pre>=0A=
=0A=
Or something similiar.&nbsp=3B</div><div>thanks&nbsp=3B<br>John<br><div><di=
v></div><hr>Date: Wed=2C 6 Mar 2013 10:25:17 +0100<br>Subject: Re: dfs.data=
node.du.reserved<br>From: <a href=3D"mailto:dechouxb@gmail.com" target=3D"_=
blank">dechouxb@gmail.com</a><br>=0A=
=0A=
To: <a href=3D"mailto:user@hadoop.apache.org" target=3D"_blank">user@hadoop=
.apache.org</a><div><div class=3D"h5"><br><br>Not that I know. If so you sh=
ould be able to identify each volume and as of now this isn't the case.<br>=
=0A=
=0A=
BUT it can be done without Hadoop knowing about it=2C at the OS level=2C us=
ing different partitions/mounts for datanode and jobtracker stuff. That sho=
uld solve your problem.<br>=0A=
<br>Regards<br><br>Bertrand<br><br><div>On Mon=2C Mar 4=2C 2013 at 10:26 PM=
=2C John Meza <span dir=3D"ltr">&lt=3B<a href=3D"mailto:j_mezazap@hotmail.c=
om" target=3D"_blank">j_mezazap@hotmail.com</a>&gt=3B</span> wrote:<br>=0A=
<blockquote style=3D"border-left:1px #ccc solid=3Bpadding-left:1ex=3B">=0A=
=0A=
=0A=
<div><div dir=3D"ltr">I'm probably not being clear.<div>this seems to descr=
ibe it:&nbsp=3B<span style=3D"font-family:Arial=2CFreeSans=2CHelvetica=2Csa=
ns-serif=3Bfont-size:13px=3Bline-height:17px=3B">dfs.datanode.du.reserved &=
nbsp=3Bconfigured per-volume.</span></div>=0A=
=0A=
=0A=
<div><a href=3D"https://issues.apache.org/jira/browse/HDFS-1564" target=3D"=
_blank">https://issues.apache.org/jira/browse/HDFS-1564</a><font face=3D"Ar=
ial=2C FreeSans=2C Helvetica=2C sans-serif"><span style=3D"line-height:17px=
=3B"><br></span></font><br>=0A=
=0A=
=0A=
thanks</div><div>John<br><div><div></div><hr>From: <a href=3D"mailto:outlaw=
dba@gmail.com" target=3D"_blank">outlawdba@gmail.com</a><br>Date: Mon=2C 4 =
Mar 2013 15:37:36 -0500<br>Subject: Re: dfs.datanode.du.reserved<br>To: <a =
href=3D"mailto:user@hadoop.apache.org" target=3D"_blank">user@hadoop.apache=
.org</a><div>=0A=
=0A=
=0A=
<div><br><br>Possible to reserve 0 from various testing I have done yet tha=
t could cause the obvious side effect of achieving zero disk space:) Have o=
nly tested in development environment=2C however. Yet there are various tun=
ing white papers and other benchmarks where the very same has been tested. =
<br>=0A=
=0A=
=0A=
=0A=
=0A=
<br>Thanks. <br><br><div>On Mon=2C Mar 4=2C 2013 at 2:00 PM=2C John Meza <s=
pan dir=3D"ltr">&lt=3B<a href=3D"mailto:j_mezazap@hotmail.com" target=3D"_b=
lank">j_mezazap@hotmail.com</a>&gt=3B</span> wrote:<br><blockquote style=3D=
"border-left:1px #ccc solid=3Bpadding-left:1ex=3B">=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
<div><div dir=3D"ltr">the parameter:&nbsp=3B<span style=3D"background-color=
:rgb(243=2C248=2C252)=3Bcolor:rgb(51=2C51=2C51)=3Bfont-family:'courier new'=
=2Ccourier=2Cmonospaced=3Bfont-size:13px=3Bline-height:13px=3B">dfs.datanod=
e.du.reserved </span><span style=3D"line-height:13px=3Bcolor:rgb(51=2C51=2C=
51)=3Bfont-size:13px=3Bfont-family:'courier new'=2Ccourier=2Cmonospaced=3B"=
>is used to reserve disk space PER datanode. Is it possible to reserve a di=
fferent amount of disk space per DISK?</span><div>=0A=
=0A=
=0A=
=0A=
=0A=
<span style=3D"background-color:rgb(243=2C248=2C252)=3Bcolor:rgb(51=2C51=2C=
51)=3Bfont-family:'courier new'=2Ccourier=2Cmonospaced=3Bfont-size:13px=3Bl=
ine-height:13px=3B"><br></span></div><div><span style=3D"background-color:r=
gb(243=2C248=2C252)=3Bcolor:rgb(51=2C51=2C51)=3Bfont-family:'courier new'=
=2Ccourier=2Cmonospaced=3Bfont-size:13px=3Bline-height:13px=3B">thanks</spa=
n></div>=0A=
=0A=
=0A=
=0A=
=0A=
<span><font color=3D"#888888"><div><span style=3D"background-color:rgb(243=
=2C248=2C252)=3Bcolor:rgb(51=2C51=2C51)=3Bfont-family:'courier new'=2Ccouri=
er=2Cmonospaced=3Bfont-size:13px=3Bline-height:13px=3B">John</span></div> 	=
	 	   		  </font></span></div>=0A=
=0A=
=0A=
=0A=
=0A=
</div>=0A=
</blockquote></div><br><br clear=3D"all"><br>-- <br><span><div dir=3D"ltr">=
<div></div><div>Ellis R. Miller</div><div><a target=3D"_blank">937.829.2380=
</a><br></div><br><div dir=3D"ltr" style=3D"font-size:10px=3B">=0A=
<span style=3D"color:gray=3B"></span><a href=3D"http://my.wisestamp.com/lin=
k?u=3D2hxhdfd4p76bkhcm&amp=3Bsite=3Dwww.wisestamp.com/email-install" target=
=3D"_blank"><img height=3D"87" width=3D"96"></a></div>=0A=
=0A=
</div></span><span></span><span><div dir=3D"ltr"><div><span></span><br><fon=
t size=3D"1"><span><span style=3D"font-weight:bold=3B">Mundo Nulla Fides </=
span></span></font></div>=0A=
=0A=
<br><div dir=3D"ltr" style=3D"font-size:10px=3B"><span style=3D"color:gray=
=3B"></span><a href=3D"http://my.wisestamp.com/link?u=3Dgfbmwhzrwxzcrjqx&am=
p=3Bsite=3Dwww.wisestamp.com/email-install" target=3D"_blank"><br></a></div=
></div></span><br>=0A=
=0A=
=0A=
=0A=
<br>=0A=
<span></span><br><br><div><span style=3D"font-size:13.3px=3Bfont-family:Ver=
dana=2CArial=2CHelvetica=2Csans-serif=3B"></span></div></div></div></div></=
div> 		 	   		  </div></div>=0A=
</blockquote></div><br></div></div></div></div></div> 		 	   		  </div></di=
v>=0A=
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>Harsh J=0A=
</div></div> 		 	   		  </div></body>
</html>=

--_7a379ee4-62cb-42d4-8bf3-f7c03e444960_--