Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of eltonsky9404@gmail.com
 designates 74.125.82.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wM9oHLSIZ2RnB0PaLFtkTnBsFj9aLiFQ4ANp5Ia+1YrcQwHReDipIvHJeJApmzsOyn
         j9qQpKLAEi7Zaj9Nb+QdiGu7T44y6QLIIl8wG+2+Lwy2nVL5mYsp+Wj6tLEw+EPOPhKi
         4Wn/BSZlY4vrA1s3eJKvnriD+s4upKHzRd9ug=
MIME-Version: 1.0
In-Reply-To: <BANLkTinEvWYq_utvvgQQwjLb8O4VAZfMSg@mail.gmail.com>
References: <BANLkTi=AovzSAOQJVHOiyOts=AEoKqZaZg@mail.gmail.com>
	<BANLkTinEvWYq_utvvgQQwjLb8O4VAZfMSg@mail.gmail.com>
Date: Mon, 4 Apr 2011 21:21:07 +1000
Message-ID: <BANLkTin9+zSDk4DDUt89y+oJ1tNy019oJA@mail.gmail.com>
Subject: Re: Remove one directory from multiple dfs.data.dir, how?
From: elton sky <eltonsky9404@gmail.com>
To: hdfs-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151758a392b6628c04a015f453

--00151758a392b6628c04a015f453
Content-Type: text/plain; charset=ISO-8859-1

Thanks Harsh,

I will give it a go as you suggested.
But I feel it's not convenient in my case. Decommission is for taking down a
node. What I am doing here is taking out a dir. In my case, all I need to do
is copy files in the dir I want to remove to remaining dirs on the node,
isn't it?

Why not hadoop has this functionality?


On Mon, Apr 4, 2011 at 5:05 PM, Harsh Chouraria <harsh@cloudera.com> wrote:

> Hello Elton,
>
> On Mon, Apr 4, 2011 at 11:44 AM, elton sky <eltonsky9404@gmail.com> wrote:
> > Now I want to remove 1 disk from each node, say /data4/hdfs-data. What I
> > should do to keep data integrity?
> > -Elton
>
> This can be done using the reliable 'decommission' process, by
> recommissioning them after having reconfigured (multiple nodes may be
> taken down per decommission round this way, but be wary of your
> cluster's actual used data capacity, and your minimum replication
> factors). Read more about the decommission processes here:
>
> http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_user_guide.html#DFSAdmin+Command
> and http://developer.yahoo.com/hadoop/tutorial/module2.html#decommission
>
> You may also have to run a cluster-wide balancer of DNs after the
> entire process is done, to get rid of some skew in the distribution of
> data across them.
>
> (P.s. As an alternative solution, you may bring down one DataNode at a
> time, reconfigure it individually, and bring it up again; then repeat
> with the next one once NN's fsck reports a healthy situation again (no
> under-replicated blocks). But decommissioning is the guaranteed safe
> way and is easier to do for some bulk of nodes.)
>
> --
> Harsh J
> Support Engineer, Cloudera
>

--00151758a392b6628c04a015f453
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Harsh,<div><br></div><div>I will give it a go as you suggested.=A0</=
div><div>But I feel it&#39;s not convenient in my case. Decommission is for=
 taking down a node. What I am doing here is taking out a dir. In my case, =
all I need to do is copy files in the dir I want to remove to remaining dir=
s on the node, isn&#39;t it?</div>
<div><br></div><div>Why not hadoop has this functionality?</div><div><br></=
div><div><br><div class=3D"gmail_quote">On Mon, Apr 4, 2011 at 5:05 PM, Har=
sh Chouraria <span dir=3D"ltr">&lt;<a href=3D"mailto:harsh@cloudera.com">ha=
rsh@cloudera.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">Hello Elton,<br>
<div class=3D"im"><br>
On Mon, Apr 4, 2011 at 11:44 AM, elton sky &lt;<a href=3D"mailto:eltonsky94=
04@gmail.com">eltonsky9404@gmail.com</a>&gt; wrote:<br>
&gt; Now I want to remove 1 disk from each node, say /data4/hdfs-data. What=
 I<br>
&gt; should do to keep data integrity?<br>
&gt; -Elton<br>
<br>
</div>This can be done using the reliable &#39;decommission&#39; process, b=
y<br>
recommissioning them after having reconfigured (multiple nodes may be<br>
taken down per decommission round this way, but be wary of your<br>
cluster&#39;s actual used data capacity, and your minimum replication<br>
factors). Read more about the decommission processes here:<br>
<a href=3D"http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_user_guide.html#=
DFSAdmin+Command" target=3D"_blank">http://hadoop.apache.org/hdfs/docs/r0.2=
1.0/hdfs_user_guide.html#DFSAdmin+Command</a><br>
and <a href=3D"http://developer.yahoo.com/hadoop/tutorial/module2.html#deco=
mmission" target=3D"_blank">http://developer.yahoo.com/hadoop/tutorial/modu=
le2.html#decommission</a><br>
<br>
You may also have to run a cluster-wide balancer of DNs after the<br>
entire process is done, to get rid of some skew in the distribution of<br>
data across them.<br>
<br>
(P.s. As an alternative solution, you may bring down one DataNode at a<br>
time, reconfigure it individually, and bring it up again; then repeat<br>
with the next one once NN&#39;s fsck reports a healthy situation again (no<=
br>
under-replicated blocks). But decommissioning is the guaranteed safe<br>
way and is easier to do for some bulk of nodes.)<br>
<font color=3D"#888888"><br>
--<br>
Harsh J<br>
Support Engineer, Cloudera<br>
</font></blockquote></div><br></div>

--00151758a392b6628c04a015f453--