Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of felix.giguere@mate1inc.com
 designates 74.125.245.76 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <5154ABCE.3030107@buscapecompany.com>
References: 
 <CAECHK7uKfbOzuZckTvcp6k_CSk6Xbm_xwj453YSUBAfZUtdvow@mail.gmail.com>
 <5154ABCE.3030107@buscapecompany.com>
From: Felix GV <felix@mate1inc.com>
Date: Thu, 28 Mar 2013 17:23:16 -0400
Message-ID: 
 <CAECHK7s2P=+pX=Omx5GtbU2nWuRsmPogCAF1X4132L746RNpKA@mail.gmail.com>
Subject: Re: Why do some blocks refuse to replicate...?
To: user <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7bb049d0ca715904d902c445

--047d7bb049d0ca715904d902c445
Content-Type: text/plain; charset=ISO-8859-1

Yes, I didn't specify how I was testing my changes, but basically, here's
what I did:

My hdfs-site.xml file was modified to include a reference the a file
containing a list of all datanodes (via dfs.hosts) and a reference to a
file containing decommissioned nodes (via dfs.hosts.exclude). After that, I
just changed these files, not hdfs-site.xml.

I first added all my old nodes in the dfs.hosts.exclude file, did hdfs dfsadmin
-refreshNodes, and most of the data replicated correctly.

I then tried removing all old nodes from the dfs.hosts file, did hdfs dfsadmin
-refreshNodes, and I saw that I now had a coupe of corrupt and missing
blocks (60 of them).

I re-added all the old nodes in the dfs.hosts file, and removed them
gradually, each time doing the refreshNodes or restarting the NN, and I
narrowed it down to three datanodes in particular, which seem to be the
three nodes where all of those 60 blocks are located.

Is it possible, perhaps, that these three nodes are completely incapable of
replicating what they have (because they're corrupt or something), and so
every block was replicated from other nodes, but the blocks that happened
to be located on these three nodes are... doomed? I can see the data in
those blocks in the NN hdfs browser, so I guess it's not corrupted... I
also tried pinging the new nodes from those old ones and it works too, so I
guess there is no network partition...

I'm in the process of increasing replication factor above 3, but I don't
know if that's gonna do anything...

--
Felix


On Thu, Mar 28, 2013 at 4:45 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:

>  Felix,
>
> After changing hdfs-site.xml, did you run "hadoop dfsadmin -refreshNodes"?
> That should have been enough, but you can try increasing the replication
> factor of these files, wait for them to be replicated to the new nodes,
> then setting it back to its original value.
>
> Cheers,
> Marcos
>
>
> In 28-03-2013 17:00, Felix GV wrote:
>
>  Hello,
>
>  I've been running a virtualized CDH 4.2 cluster. I now want to migrate
> all my data to another (this time physical) set of slaves and then stop
> using the virtualized slaves.
>
>  I added the new physical slaves in the cluster, and marked all the old
> virtualized slaves as decommissioned using the dfs.hosts.exclude setting in
> hdfs-site.xml.
>
>  Almost all of the data replicated successfully to the new slaves, but
> when I bring down the old slaves, some blocks start showing up as missing
> or corrupt (according to the NN UI as well as fsck*). If I restart the old
> slaves, then there are no missing blocks reported by fsck.
>
>  I've tried shutting down the old slaves two by two, and for some of them
> I saw no problem, but then at some point I found two slaves which, when
> shut down, resulted in a couple of blocks being under-replicated (1 out of
> 3 replicas found). For example, fsck would report stuff like this:
>
>  /user/hive/warehouse/ads_destinations_hosts/part-m-00012:  Under
> replicated
> BP-1207449144-10.10.10.21-1356639087818:blk_6150201737015349469_121244.
> Target Replicas is 3 but found 1 replica(s).
>
>  The system then stayed in that state apparently forever. It never
> actually fixed the fact some blocks were under-replicated. Does that mean
> there's something wrong with some of the old datanodes...? Why do they keep
> block for themselves (even thought they're decommissioned) instead of
> replicating those blocks to the new (non-decommissioned) datanodes?
>
>  How do I force replication of under-replicated blocks?
>
>  *Actually, the NN UI and fsck report slightly different things. The NN
> UI always seems to report 60 under-replicated blocks, whereas fsck only
> reports those 60 under-replicated blocks when I shut down some of the old
> datanodes... When the old nodes are up, fsck reports 0 under-replicated
> blocks... This is very confusing!
>
>  Any help would be appreciated! Please don't hesitate to ask if I should
> provide some of my logs, settings, or the output of some commands...!
>
>  Thanks :) !
>
> --
> Felix
>
>
>

--047d7bb049d0ca715904d902c445
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Yes, I didn&#39;t specify how I was testing my changes, bu=
t basically, here&#39;s what I did:<div><br></div><div style>My hdfs-site.x=
ml file was modified to include a reference the a file containing a list of=
 all datanodes (via=A0dfs.hosts) and a reference to a file containing decom=
missioned nodes (via=A0dfs.hosts.exclude). After that, I just changed these=
 files, not hdfs-site.xml.</div>

<div style><br></div><div style>I first added all my old nodes in the=A0dfs=
.hosts.exclude=A0file, did hdfs=A0<span style=3D"font-family:arial,sans-ser=
if;font-size:12.800000190734863px">dfsadmin -refreshNodes,</span>=A0and mos=
t of the data replicated correctly.</div>

<div style><br></div><div style>I then tried removing all old nodes from th=
e dfs.hosts file, did hdfs=A0<span style=3D"font-size:12.800000190734863px;=
font-family:arial,sans-serif">dfsadmin -refreshNodes, and I saw that I now =
had a coupe of corrupt and missing blocks (60 of them).</span></div>

<div style><span style=3D"font-size:12.800000190734863px;font-family:arial,=
sans-serif"><br></span></div><div style><span style=3D"font-size:12.8000001=
90734863px;font-family:arial,sans-serif">I re-added all the old nodes in th=
e=A0</span>dfs.hosts file, and removed them gradually, each time doing the =
refreshNodes or restarting the NN, and I narrowed it down to three datanode=
s in particular, which seem to be the three nodes where all of those 60 blo=
cks are located.</div>

<div style><br></div><div style>Is it possible, perhaps, that these three n=
odes are completely incapable of replicating what they have (because they&#=
39;re corrupt or something), and so every block was replicated from other n=
odes, but the blocks that happened to be located on these three nodes are..=
. doomed? I can see the data in those blocks in the NN hdfs browser, so I g=
uess it&#39;s not corrupted... I also tried pinging the new nodes from thos=
e old ones and it works too, so I guess there is no network partition...</d=
iv>

<div style><br></div><div style>I&#39;m in the process of increasing replic=
ation factor above 3, but I don&#39;t know if that&#39;s gonna do anything.=
..</div></div><div class=3D"gmail_extra"><br clear=3D"all"><div>--<br>Felix=
<br>

</div>
<br><br><div class=3D"gmail_quote">On Thu, Mar 28, 2013 at 4:45 PM, MARCOS =
MEDRADO RUBINELLI <span dir=3D"ltr">&lt;<a href=3D"mailto:marcosm@buscapeco=
mpany.com" target=3D"_blank">marcosm@buscapecompany.com</a>&gt;</span> wrot=
e:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">


<div text=3D"#000000" bgcolor=3D"#FFFFFF">
<div>Felix,<br>
<br>
After changing hdfs-site.xml, did you run &quot;hadoop dfsadmin -refreshNod=
es&quot;? That should have been enough, but you can try increasing the repl=
ication factor of these files, wait for them to be replicated to the new no=
des, then setting it back to its original
 value.<br>
<br>
Cheers,<br>
Marcos<div><div class=3D"h5"><br>
<br>
In 28-03-2013 17:00, Felix GV wrote:<br>
</div></div></div><div><div class=3D"h5">
<blockquote type=3D"cite">
<div dir=3D"ltr">
<div>Hello,</div>
<div><br>
</div>
<div>I&#39;ve been running a virtualized CDH 4.2 cluster. I now want to mig=
rate all my data to another (this time physical) set of slaves and then sto=
p using the virtualized slaves.</div>
<div><br>
</div>
<div>I added the new physical slaves in the cluster, and marked all the old=
 virtualized slaves as decommissioned using the=A0dfs.hosts.exclude setting=
 in hdfs-site.xml.</div>
<div><br>
</div>
<div>Almost all of the data replicated successfully to the new slaves, but =
when I bring down the old slaves, some blocks start showing up as missing o=
r corrupt (according to the NN UI as well as fsck*). If I restart the old s=
laves, then there are no
 missing blocks reported by fsck.</div>
<div><br>
</div>
<div>I&#39;ve tried shutting down the old slaves two by two, and for some o=
f them I saw no problem, but then at some point I found two slaves which, w=
hen shut down, resulted in a couple of blocks being under-replicated (1 out=
 of 3 replicas found). For
 example, fsck would report stuff like this:</div>
<div><br>
</div>
<div>
<div>/user/hive/warehouse/ads_destinations_hosts/part-m-00012: =A0Under rep=
licated BP-1207449144-10.10.10.21-1356639087818:blk_6150201737015349469_121=
244. Target Replicas is 3 but found 1 replica(s).</div>
</div>
<div><br>
</div>
<div>The system then stayed in that state apparently forever. It never actu=
ally fixed the fact some blocks were under-replicated. Does that mean there=
&#39;s something wrong with some of the old datanodes...? Why do they keep =
block for themselves (even
 thought they&#39;re decommissioned) instead of replicating those blocks to=
 the new (non-decommissioned) datanodes?</div>
<div><br>
</div>
<div>How do I force replication of under-replicated blocks?</div>
<div><br>
</div>
<div>*Actually, the NN UI and fsck report slightly different things. The NN=
 UI always seems to report 60 under-replicated blocks, whereas fsck only re=
ports those 60 under-replicated blocks when I shut down some of the old dat=
anodes... When the old
 nodes are up, fsck reports 0 under-replicated blocks... This is very confu=
sing!</div>
<div><br>
</div>
<div>Any help would be appreciated! Please don&#39;t hesitate to ask if I s=
hould provide some of my logs, settings, or the output of some commands...!=
</div>
<div><br>
</div>
<div>Thanks :) !</div>
<br clear=3D"all">
<div>--<br>
Felix<br>
</div>
</div>
</blockquote>
<br>
</div></div></div>

</blockquote></div><br></div>

--047d7bb049d0ca715904d902c445--