Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.160.46 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMhkWdBBHncym+vMB0FcB5j4v3A+E9FL3KvBPtVmGAU2AaL=sA@mail.gmail.com>
References: 
 <CAMhkWdBBHncym+vMB0FcB5j4v3A+E9FL3KvBPtVmGAU2AaL=sA@mail.gmail.com>
Date: Sat, 7 Dec 2013 15:26:01 +0800
Message-ID: 
 <CALte62zyAVyALBFas70_Maz4jn8tdX5+jpDijyLFnL7OmgLCOw@mail.gmail.com>
Subject: Re: Namenode won't start after unclean shutdown of RS
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=bcaec5314e7dbcaded04ececac97

--bcaec5314e7dbcaded04ececac97
Content-Type: text/plain; charset=ISO-8859-1

What version of Hadoop / HBase are you using ?

Cheers


On Sat, Dec 7, 2013 at 9:38 AM, iain wright <iainwrig@gmail.com> wrote:

> Hi folks,
>
> One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted
> about reports failing, and i saw wierd logs in the jobtracker for "pending
> shutdown" etc.
>
> Proceeded to stop jobtracker/tt nodes, hbase, and hdfs.
>
> On attempting to turn back on the NN + DN's our namenode is failing to
> start, complaining about being unable to move one of our hbase region files
> because source file does exist.
>
> Has anyone come across this/have suggestions??
>
> 2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 149098688 loaded in 14 seconds.
> 2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> FSDirectory.unprotectedRenameTo: failed to rename
>
> /hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> to
>
> /hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> because source does not exist
> 2013-12-07 01:20:43,348 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>
> 2013-12-07 01:20:43,349 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at
> namenode0.hadoop.compe.tv/10.10.11.10
>
> Thank you,
>
> --
> iain
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>

--bcaec5314e7dbcaded04ececac97--