Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E88EDBE4 for ; Mon, 27 Aug 2012 05:50:23 +0000 (UTC) Received: (qmail 24101 invoked by uid 500); 27 Aug 2012 05:50:18 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 23724 invoked by uid 500); 27 Aug 2012 05:50:16 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 23708 invoked by uid 99); 27 Aug 2012 05:50:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Aug 2012 05:50:15 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of abhay.ratnaparkhi@gmail.com designates 74.125.83.48 as permitted sender) Received: from [74.125.83.48] (HELO mail-ee0-f48.google.com) (74.125.83.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Aug 2012 05:50:09 +0000 Received: by eekd41 with SMTP id d41so1447411eek.35 for ; Sun, 26 Aug 2012 22:49:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=OYmV3dIMTDxvfMLXq41c3wLDJM+oVOk9MwZ1jUR3iLk=; b=fHtk+K33GDKm4Qn42X1vk3o8x0waKpW3uqYFlb0RgQejHZsxV8bVd6Yc3MPZEJHW7q FlMU6C2krd851sPoiHX/2DFK5Tz0jltSV7m+/mT+448Xmst8g90EXZfm8e126CIfLcPJ Qjv+OZ/nG1pEGmXy5DFc1drBnZ1+RwmUnwOZRxq9qSK2JH3CEm/SE7150YLmeFlgfRxQ tthQHP5s0YAu0/pA+L8SE7az31O2KTFQkPb3l6BCLbfBPPOoX3arsaIfVRiNtDY0fL5+ iX3xV6biIM/LAMbDqoX/trKMhJtQ+NSyoKxJ16LMviLFvibg9kf1tZBOoQoSVCjgRNaY pM1Q== MIME-Version: 1.0 Received: by 10.14.215.197 with SMTP id e45mr16084200eep.36.1346046589167; Sun, 26 Aug 2012 22:49:49 -0700 (PDT) Received: by 10.14.223.136 with HTTP; Sun, 26 Aug 2012 22:49:49 -0700 (PDT) In-Reply-To: References: <858038922-1345793514-cardhu_decombobulator_blackberry.rim.net-342541276-@b25.c15.bise7.blackberry> Date: Mon, 27 Aug 2012 11:19:49 +0530 Message-ID: Subject: Re: namenode not starting From: Abhay Ratnaparkhi To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b603d18cc67a904c838e451 --047d7b603d18cc67a904c838e451 Content-Type: text/plain; charset=ISO-8859-1 Thank you Harsh, I have set "dfs.name.dir" explicitly. Still don't know why the data loss has happened. dfs.name.dir /wsadfs/${host.name}/name Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. The secondary namenode was same as namenode. Does this affect anyway since path of "dfs.name.dir" were same? I have now configured another machine as secondary namenode. I have now formatted the filesystem since not seen any way of recovering. I have some questions. 1. Apart from setting secondary namenode what are the other techniques used for namenode directory backups? 2. Is there any way or tools to recover some of the data even if namenode crashes. Regards, Abhay On Sat, Aug 25, 2012 at 7:45 PM, Harsh J wrote: > Abhay, > > I suspect that if you haven't set your dfs.name.dir explicitly, then > you haven't set fs.checkpoint.dir either, and since both use > hadoop.tmp.dir paths, you may have lost your data completely and there > is no recovery possible now. > > On Fri, Aug 24, 2012 at 1:10 PM, Abhay Ratnaparkhi > wrote: > > Hello, > > > > I was using cluster for long time and not formatted the namenode. > > I ran bin/stop-all.sh and bin/start-all.sh scripts only. > > > > I am using NFS for dfs.name.dir. > > hadoop.tmp.dir is a /tmp directory. I've not restarted the OS. Any way > to > > recover the data? > > > > Thanks, > > Abhay > > > > > > On Fri, Aug 24, 2012 at 1:01 PM, Bejoy KS > wrote: > >> > >> Hi Abhay > >> > >> What is the value for hadoop.tmp.dir or dfs.name.dir . If it was set to > >> /tmp the contents would be deleted on a OS restart. You need to change > this > >> location before you start your NN. > >> Regards > >> Bejoy KS > >> > >> Sent from handheld, please excuse typos. > >> ________________________________ > >> From: Abhay Ratnaparkhi > >> Date: Fri, 24 Aug 2012 12:58:41 +0530 > >> To: > >> ReplyTo: user@hadoop.apache.org > >> Subject: namenode not starting > >> > >> Hello, > >> > >> I had a running hadoop cluster. > >> I restarted it and after that namenode is unable to start. I am getting > >> error saying that it's not formatted. :( > >> Is it possible to recover the data on HDFS? > >> > >> 2012-08-24 03:17:55,378 ERROR > >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > >> initialization failed. > >> java.io.IOException: NameNode is not formatted. > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:270) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:433) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) > >> at > >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) > >> 2012-08-24 03:17:55,380 ERROR > >> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: > >> NameNode is not formatted. > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:270) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:433) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421) > >> at > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) > >> at > >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) > >> > >> Regards, > >> Abhay > >> > >> > > > > > > -- > Harsh J > --047d7b603d18cc67a904c838e451 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thank you Harsh,

I have set "dfs.name.dir&quo= t; explicitly. Still don't know why the data loss has happened.=A0
=
<property>
=A0 <name>dfs.name.dir= </name>
=A0 <value>/wsadfs/${host.name}= /name</value>
=A0 <description>Determines where on th= e local filesystem the DFS name node
=A0 =A0 =A0 should store the= name table. =A0If this is a comma-delimited list
=A0 =A0 =A0 of directories then the name table is replicated in all of= the
=A0 =A0 =A0 directories, for redundancy. </description>= ;
</property>

The secondary nameno= de was same as namenode. Does this affect =A0anyway since path of "dfs= .name.dir" were same?
I have now configured another machine as secondary namenode.
I have now =A0formatted the filesystem since not seen any way of recoverin= g.=A0

I have some questions.

<= div> 1. Apart from setting secondary namenode what are the other techniques used= for namenode directory backups?
2. Is there any way or tools to = recover some of the data even if namenode crashes.=A0

Regards,
Abhay



On Sat, Aug 25, 2012 at 7:45 PM, Har= sh J <harsh@cloudera.com> wrote:
Abhay,

I suspect that if you haven't set your dfs.name.dir explicitly, then you haven't set fs.checkpoint.dir either, and since both use
hadoop.tmp.dir paths, you may have lost your data completely and there
is no recovery possible now.

On Fri, Aug 24, 2012 at 1:10 PM, Abhay Ratnaparkhi
<abhay.ratnaparkhi@gmail.= com> wrote:
> Hello,
>
> I was using cluster for long time and not formatted the namenode.
> I ran bin/stop-all.sh and bin/start-all.sh scripts only.
>
> I am using NFS for dfs.name.dir.
> hadoop.tmp.dir is a /tmp directory. I've not restarted the OS. =A0= Any way to
> recover the data?
>
> Thanks,
> Abhay
>
>
> On Fri, Aug 24, 2012 at 1:01 PM, Bejoy KS <bejoy.hadoop@gmail.com> wrote:
>>
>> Hi Abhay
>>
>> What is the value for hadoop.tmp.dir or dfs.name.dir . If it was s= et to
>> /tmp the contents would be deleted on a OS restart. You need to ch= ange this
>> location before you start your NN.
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ________________________________
>> From: Abhay Ratnaparkhi <abhay.ratnaparkhi@gmail.com>
>> Date: Fri, 24 Aug 2012 12:58:41 +0530
>> To: <user@hadoop.apac= he.org>
>> ReplyTo: user@hadoop.apa= che.org
>> Subject: namenode not starting
>>
>> Hello,
>>
>> I had a running hadoop cluster.
>> I restarted it and after that namenode is unable to start. I am ge= tting
>> error saying that it's not formatted. :(
>> Is it possible to recover the data on HDFS?
>>
>> 2012-08-24 03:17:55,378 ERROR
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem<= br> >> initialization failed.
>> java.io.IOException: NameNode is not formatted.
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRe= ad(FSImage.java:434)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSD= irectory.java:110)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSN= amesystem.java:291)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(F= SNamesystem.java:270)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(Nam= eNode.java:271)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNod= e.java:303)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameN= ode.java:433)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameN= ode.java:421)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(Nam= eNode.java:1359)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java= :1368)
>> 2012-08-24 03:17:55,380 ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOExcepti= on:
>> NameNode is not formatted.
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRe= ad(FSImage.java:434)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSD= irectory.java:110)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSN= amesystem.java:291)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(F= SNamesystem.java:270)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(Nam= eNode.java:271)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNod= e.java:303)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameN= ode.java:433)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameN= ode.java:421)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(Nam= eNode.java:1359)
>> =A0 =A0 =A0 =A0 at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java= :1368)
>>
>> Regards,
>> Abhay
>>
>>
>



--
Harsh J

--047d7b603d18cc67a904c838e451--