Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of anilgupta84@gmail.com
 designates 209.85.160.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAL3=Dw1prBcYs7a3U8D6taSKQN1pVqDDPXGDYq0BYxP2SsLgHA@mail.gmail.com>
References: 
 <CAL3=Dw1prBcYs7a3U8D6taSKQN1pVqDDPXGDYq0BYxP2SsLgHA@mail.gmail.com>
From: anil gupta <anilgupta84@gmail.com>
Date: Wed, 8 Aug 2012 00:56:35 -0700
Message-ID: 
 <CAF1+Vs-E_sYu=2nB-W5HaQGm33nCw9OFquQdTx0hquYNxRZ0LA@mail.gmail.com>
Subject: Re: Data node error
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=14dae934086b6f411804c6bc74cc

--14dae934086b6f411804c6bc74cc
Content-Type: text/plain; charset=ISO-8859-1

Hi Prabhu,

Did you clean the data dir on DataNodes? Whenever Namenode is formated the
data directories of Datanodes needs to be cleaned up. As far as i remember
it's the directory which you mention in dfs.data.dir in hdfs-site.xml
file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the
exact property).

Thanks,
Anil

On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <prabhu.hadoop@gmail.com> wrote:

> Hi Users,
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.
>
> Data node(slave1):
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
>         at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
>         at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
>         at java.lang.Thread.run(Thread.java:662)
>
> *data node(slave2)*
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
>         at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> Please help me on this issue.
>
> Thanks,
> Prabhu.
>


-- 
Thanks & Regards,
Anil Gupta

--14dae934086b6f411804c6bc74cc
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Prabhu,<br><br>Did you clean the data dir on DataNodes? Whenever Namenod=
e is formated the data directories of Datanodes needs to be cleaned up. As =
far as i remember it&#39;s the directory which you mention in dfs.data.dir =
in hdfs-site.xml file.You can do a google search for the error and you can =
get more details.<br>

(Sorry, i dont have access to my cluster conf right now for telling you the=
 exact property). <br><br>Thanks,<br>Anil<br><br><div class=3D"gmail_quote"=
>On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <span dir=3D"ltr">&lt;<a href=3D=
"mailto:prabhu.hadoop@gmail.com" target=3D"_blank">prabhu.hadoop@gmail.com<=
/a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div>Hi Users,</div>
<div>=A0</div>
<div>I have formatted hadoop cluster, formatted successfully. after stop &a=
mp;start hadoop, i hit the jps command in master, getting fine, but in slav=
e machine am not getting data node, while see the data node log file, i am =
getting following error.</div>


<div>=A0</div>
<div><font style=3D"BACKGROUND-COLOR:#cc0000">Data node(slave1):</font></di=
v>
<div>2012-08-08 00:16:44,033 WARN org.apache.hadoop.hdfs.server.datanode.Da=
taBlockScanner: Second Verification failed for blk_-3831635302961953167_169=
0. Exception : <a href=3D"http://java.io" target=3D"_blank">java.io</a>.<br=
>

IOException: Block blk_-3831635302961953167_1690 is not valid.<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.FSDataset.g=
etBlockFile(FSDataset.java:1072)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.had=
oop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)<br>=A0=A0=
=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisi=
bleLength(FSDataset.java:1045)<br>


=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.BlockSender=
.&lt;init&gt;(BlockSender.java:94)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.h=
adoop.hdfs.server.datanode.BlockSender.&lt;init&gt;(BlockSender.java:81)<br=
>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataBlockS=
canner.verifyBlock(DataBlockScanner.java:453)<br>


=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataBlockSc=
anner.verifyFirstBlock(DataBlockScanner.java:519)<br>=A0=A0=A0=A0=A0=A0=A0 =
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockSca=
nner.java:617)<br>=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java=
:662)</div>


<div><br><font style=3D"BACKGROUND-COLOR:#cc0000"><b>data node(slave2)</b><=
/font></div>
<div>2012-08-08 13:03:50,195 ERROR org.apache.hadoop.hdfs.server.datanode.D=
ataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/=
dfs/data: namenode name<br>spaceID =3D 1434906924; datanode namespaceID =3D=
 474761520<br>


=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataStorage=
.doTransition(DataStorage.java:232)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.=
hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.j=
ava:147)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode=
.DataNode.startDataNode(DataNode.java:385)<br>


=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataNode.&l=
t;init&gt;(DataNode.java:299)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop=
.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)<br>=A0=A0=
=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataNode.instanti=
ateDataNode(DataNode.java:1521)<br>


=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataNode.cr=
eateDataNode(DataNode.java:1539)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.had=
oop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)<br>=A0=A0=
=A0=A0=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataNode.main(Dat=
aNode.java:1682)</div>


<div>=A0</div>
<div>Please help me on this issue.</div>
<div>=A0</div>
<div>Thanks,</div>
<div>Prabhu.</div>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Thanks &amp; Regards,<b=
r>Anil Gupta<br>

--14dae934086b6f411804c6bc74cc--