hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fatih Haltas <fatih.hal...@nyu.edu>
Subject Re: I am about to lose all my data please help
Date Tue, 25 Mar 2014 05:22:18 GMT
Ok, Thanks to you all, I just removed version information of all datanodes
and namenodes, then restart, it is working fine now


On Mon, Mar 24, 2014 at 5:52 PM, praveenesh kumar <praveenesh@gmail.com>wrote:

> Can you also make sure your hostname and IP address are still mapped
> correctly. Because what I am guessing is when you restart your machine,
> your /etc/hosts entries might get restored (it happens in some
> distributions, based on how you installed it). So when you are trying to
> restart your namenode, it might be pointing to some different IP/machine
> (in general localhost).
>
> I can't think of any reason how  it can happen just by restarting the
> machine.
>
>
> On Mon, Mar 24, 2014 at 5:42 AM, Stanley Shi <sshi@gopivotal.com> wrote:
>
>> Can you confirm that you namenode image and fseditlog are still there? if
>> not, then your data IS lost.
>>
>> Regards,
>> *Stanley Shi,*
>>
>>
>>
>> On Sun, Mar 23, 2014 at 6:24 PM, Fatih Haltas <fatih.haltas@nyu.edu>wrote:
>>
>>> No, not ofcourse I blinded it.
>>>
>>>
>>> On Wed, Mar 19, 2014 at 5:09 PM, praveenesh kumar <praveenesh@gmail.com>wrote:
>>>
>>>> Is this property correct ?
>>>>
>>>>
>>>> <property>
>>>>     <name>fs.default.name</name>
>>>>     <value>-BLANKED</value>
>>>>   </property>
>>>>
>>>> Regards
>>>> Prav
>>>>
>>>>
>>>> On Wed, Mar 19, 2014 at 12:58 PM, Fatih Haltas <fatih.haltas@nyu.edu>wrote:
>>>>
>>>>> Thanks for you helps, but still could not solve my problem.
>>>>>
>>>>>
>>>>> On Tue, Mar 18, 2014 at 10:13 AM, Stanley Shi <sshi@gopivotal.com>wrote:
>>>>>
>>>>>> Ah yes, I overlooked this. Then please check the file are there or
>>>>>> not: "ls /home/hadoop/project/hadoop-data/dfs/name"?
>>>>>>
>>>>>> Regards,
>>>>>> *Stanley Shi,*
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 18, 2014 at 2:06 PM, Azuryy Yu <azuryyyu@gmail.com>wrote:
>>>>>>
>>>>>>> I don't think this is the case, because there is;
>>>>>>>   <property>
>>>>>>>     <name>hadoop.tmp.dir</name>
>>>>>>>     <value>/home/hadoop/project/hadoop-data</value>
>>>>>>>   </property>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 18, 2014 at 1:55 PM, Stanley Shi <sshi@gopivotal.com>wrote:
>>>>>>>
>>>>>>>> one possible reason is that you didn't set the namenode working
>>>>>>>> directory, by default it's in "/tmp" folder; and the "/tmp"
folder might
>>>>>>>> get deleted by the OS without any notification. If this is
the case, I am
>>>>>>>> afraid you have lost all your namenode data.
>>>>>>>>
>>>>>>>> *<property>
>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>   <value>${hadoop.tmp.dir}/dfs/name</value>
>>>>>>>>   <description>Determines where on the local filesystem
the DFS name node
>>>>>>>>       should store the name table(fsimage).  If this is a
comma-delimited list
>>>>>>>>       of directories then the name table is replicated in
all of the
>>>>>>>>       directories, for redundancy. </description>
>>>>>>>> </property>*
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> *Stanley Shi,*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 16, 2014 at 5:29 PM, Mirko Kämpf <
>>>>>>>> mirko.kaempf@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> what is the location of the namenodes fsimage and editlogs?
>>>>>>>>> And how much memory has the NameNode.
>>>>>>>>>
>>>>>>>>> Did you work with a Secondary NameNode or a Standby NameNode
for
>>>>>>>>> checkpointing?
>>>>>>>>>
>>>>>>>>> Where are your HDFS blocks located, are those still safe?
>>>>>>>>>
>>>>>>>>> With this information at hand, one might be able to fix
your
>>>>>>>>> setup, but do not format the old namenode before
>>>>>>>>> all is working with a fresh one.
>>>>>>>>>
>>>>>>>>> Grab a copy of the maintainance guide:
>>>>>>>>> http://shop.oreilly.com/product/0636920025085.do?sortby=publicationDate
>>>>>>>>> which helps solving such type of problems as well.
>>>>>>>>>
>>>>>>>>> Best wishes
>>>>>>>>> Mirko
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2014-03-16 9:07 GMT+00:00 Fatih Haltas <fatih.haltas@nyu.edu>:
>>>>>>>>>
>>>>>>>>> Dear All,
>>>>>>>>>>
>>>>>>>>>> I have just restarted machines of my hadoop clusters.
Now, I am
>>>>>>>>>> trying to restart hadoop clusters again, but getting
error on namenode
>>>>>>>>>> restart. I am afraid of loosing my data as it was
properly running for more
>>>>>>>>>> than 3 months. Currently, I believe if I do namenode
formatting, it will
>>>>>>>>>> work again, however, data will be lost. Is there
anyway to solve this
>>>>>>>>>> without losing the data.
>>>>>>>>>>
>>>>>>>>>> I will really appreciate any help.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> =====================
>>>>>>>>>> Here is the logs;
>>>>>>>>>> ====================
>>>>>>>>>> 2014-02-26 16:02:39,698 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode:
STARTUP_MSG:
>>>>>>>>>> /************************************************************
>>>>>>>>>> STARTUP_MSG: Starting NameNode
>>>>>>>>>> STARTUP_MSG:   host = ADUAE042-LAP-V/127.0.0.1
>>>>>>>>>> STARTUP_MSG:   args = []
>>>>>>>>>> STARTUP_MSG:   version = 1.0.4
>>>>>>>>>> STARTUP_MSG:   build =
>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0-r
1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012
>>>>>>>>>> ************************************************************/
>>>>>>>>>> 2014-02-26 16:02:40,005 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from
>>>>>>>>>> hadoop-metrics2.properties
>>>>>>>>>> 2014-02-26 16:02:40,019 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter:
MBean for source
>>>>>>>>>> MetricsSystem,sub=Stats registered.
>>>>>>>>>> 2014-02-26 16:02:40,021 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot
>>>>>>>>>> period at 10 second(s).
>>>>>>>>>> 2014-02-26 16:02:40,021 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
NameNode metrics system
>>>>>>>>>> started
>>>>>>>>>> 2014-02-26 16:02:40,169 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter:
MBean for source ugi
>>>>>>>>>> registered.
>>>>>>>>>> 2014-02-26 16:02:40,193 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter:
MBean for source jvm
>>>>>>>>>> registered.
>>>>>>>>>> 2014-02-26 16:02:40,194 INFO
>>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter:
MBean for source
>>>>>>>>>> NameNode registered.
>>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
VM
>>>>>>>>>> type       = 64-bit
>>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
2%
>>>>>>>>>> max memory = 17.77875 MB
>>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
>>>>>>>>>> capacity      = 2^21 = 2097152 entries
>>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
>>>>>>>>>> recommended=2097152, actual=2097152
>>>>>>>>>> 2014-02-26 16:02:40,273 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=hadoop
>>>>>>>>>> 2014-02-26 16:02:40,273 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=supergroup
>>>>>>>>>> 2014-02-26 16:02:40,274 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>>> isPermissionEnabled=true
>>>>>>>>>> 2014-02-26 16:02:40,279 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>>> dfs.block.invalidate.limit=100
>>>>>>>>>> 2014-02-26 16:02:40,279 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>>> isAccessTokenEnabled=false accessKeyUpdateInterval=0
min(s),
>>>>>>>>>> accessTokenLifetime=0 min(s)
>>>>>>>>>> 2014-02-26 16:02:40,724 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Registered
>>>>>>>>>> FSNamesystemStateMBean and NameNodeMXBean
>>>>>>>>>> 2014-02-26 16:02:40,749 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode:
Caching file names
>>>>>>>>>> occuring more than 10 times
>>>>>>>>>> 2014-02-26 16:02:40,780 ERROR
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
FSNamesystem
>>>>>>>>>> initialization failed.
>>>>>>>>>> java.io.IOException: NameNode is not formatted.
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>>>>>>>>>> 2014-02-26 16:02:40,781 ERROR
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode:
java.io.IOException:
>>>>>>>>>> NameNode is not formatted.
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>>>>>>>>>>
>>>>>>>>>> 2014-02-26 16:02:40,781 INFO
>>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
>>>>>>>>>> /************************************************************
>>>>>>>>>> SHUTDOWN_MSG: Shutting down NameNode at ADUAE042-LAP-V/127.0.0.1
>>>>>>>>>> ************************************************************/
>>>>>>>>>>
>>>>>>>>>> ===========================
>>>>>>>>>> Here is the core-site.xml
>>>>>>>>>> ===========================
>>>>>>>>>>  <?xml version="1.0"?>
>>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>>>>>
>>>>>>>>>> <!-- Put site-specific property overrides in this
file. -->
>>>>>>>>>>
>>>>>>>>>> <configuration>
>>>>>>>>>> <property>
>>>>>>>>>>     <name>fs.default.name</name>
>>>>>>>>>>     <value>-BLANKED</value>
>>>>>>>>>>   </property>
>>>>>>>>>>   <property>
>>>>>>>>>>     <name>hadoop.tmp.dir</name>
>>>>>>>>>>     <value>/home/hadoop/project/hadoop-data</value>
>>>>>>>>>>   </property>
>>>>>>>>>> </configuration>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message