Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of himanish@gmail.com designates
 209.85.212.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DV8q9YRuFqwp25HLs4HaXX+I+PKSNb4Sl4T+HfotBSBv51STn+A2+P3+lc/5/pL+U3
         ZXzUAiRd2LHP/jXpycEGnoYv5AUFiIGVFKwmhplyAUKoduc5XdQFCE7oUmtjuv4TBEJW
         lClnQSa5gn5npH3qP2JpLut/plk2Xw35SwQU4=
MIME-Version: 1.0
In-Reply-To: <BANLkTi=YBToo860qToOGp_RTa2Kh==e3ng@mail.gmail.com>
References: <BANLkTi=rv7kLnZ8Szv+=7WSqfTCDponAMg@mail.gmail.com>
	<BANLkTinmyQRzWxBd=mNAEHZHiUpwQDOb8w@mail.gmail.com>
	<BANLkTin2gy78npTKCAtiz73R2eH2AaQGMw@mail.gmail.com>
	<BANLkTimGo6sy=5CRsAPqgfDZ-O-m_cJxOg@mail.gmail.com>
	<BANLkTi=YBToo860qToOGp_RTa2Kh==e3ng@mail.gmail.com>
Date: Tue, 24 May 2011 10:19:23 -0400
Message-ID: <BANLkTi=6RP2rKvbNCNa3QuG+4u-K943uqQ@mail.gmail.com>
Subject: Re: HBase Not Starting after improper shutdown
From: Himanish Kushary <himanish@gmail.com>
To: user@hbase.apache.org, billgraham@gmail.com
Content-Type: multipart/alternative; boundary=20cf304346d45a2abb04a40646dd

--20cf304346d45a2abb04a40646dd
Content-Type: text/plain; charset=ISO-8859-1

The Region Server logs also shows the same -ROOT- Region not online error.

On Mon, May 23, 2011 at 1:10 PM, Bill Graham <billgraham@gmail.com> wrote:

> Is there anything meaningful in the RS logs? I've seen situations like this
> where a RS is failing to start due to issues reading the WAL. If this is
> the
> case it would list which WAL is problematic, which is zero-length in my
> experience, so I delete it from HDFS and things start up.
>
>
> On Mon, May 23, 2011 at 9:16 AM, Himanish Kushary <himanish@gmail.com
> >wrote:
>
> > Both the Master and hbck command prints
> >
> > org.apache.hadoop.hbase.NotServingRegionException:
> > org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
> > -ROOT-,,0
> >
> > After the master thread exits due to the Heap Space error the hbck
> command
> > throws:
> >
> > org.apache.hadoop.hbase.MasterNotRunningException
> >
> > Is there anyway to fix this kind of issue.We are keeping the datanodes up
> > to
> > see whether the under replicated blocks may be recovered.Does improper
> > shutdown of the hadoop/hbase services cause this kind of issues? What
> > happens in case of disaster recovery situation, how are those situaltions
> > handled ?
> >
> > Thanks
> >
> >
> > On Mon, May 23, 2011 at 11:36 AM, Stack <stack@duboce.net> wrote:
> >
> > > What does hbase hbck say?  (http://hbase.apache.org/book.html#hbck).
> > >
> > > What does the master log have in it?  Anything of interest.
> > >
> > > St.Ack
> > >
> > > On Mon, May 23, 2011 at 7:53 AM, Himanish Kushary <himanish@gmail.com>
> > > wrote:
> > > > Pressed the send button too soon...
> > > >
> > > > Also here is the output from hadoop fsck
> > > >
> > > > *Status: HEALTHY*
> > > > * Total size: 37678848280 B*
> > > > * Total dirs: 941*
> > > > * Total files: 902 (Files currently being written: 1)*
> > > > * Total blocks (validated): 1141 (avg. block size 33022654 B) (Total
> > open
> > > > file blocks (not validated): 1)*
> > > > * Minimally replicated blocks: 1141 (100.0 %)*
> > > > * Over-replicated blocks: 0 (0.0 %)*
> > > > * Under-replicated blocks: 906 (79.40403 %)*
> > > > * Mis-replicated blocks: 0 (0.0 %)*
> > > > * Default replication factor: 2*
> > > > * Average block replication: 2.0*
> > > > * Corrupt blocks: 0*
> > > > * Missing replicas: 1886 (82.646805 %)*
> > > > * Number of data-nodes: 2*
> > > > * Number of racks: 1*
> > > > *FSCK ended at Mon May 23 10:51:13 EDT 2011 in 257 milliseconds*
> > > > *
> > > > *
> > > > *
> > > > *
> > > > *The filesystem under path '/' is HEALTHY*
> > > >
> > > >
> > > > Could anybody please help on how to recover from this scenario .
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Mon, May 23, 2011 at 10:50 AM, Himanish Kushary <
> himanish@gmail.com
> > > >wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Our hbase/hadoop servers machines were shutdown without bringing the
> > > hadoop
> > > >> and hbase services down properly.Now when we try to bring up hbase
> we
> > > get
> > > >> the following error in the master log:
> > > >>
> > > >> org.apache.hadoop.hbase.NotServingRegionException: Region is not
> > online:
> > > >> -ROOT-,,0
> > > >>
> > > >> Hadoop services (namenode,jobtracker,datanode etc) have come up
> > properly
> > > >> and we are able to see the files in HDFS. But HBase Master keeps on
> > > throwing
> > > >> this exception and then finally throws a Java Heap Space error.
> > > >>
> > > >> Note: We have two datanodes, replication set to 2 and around 900
> > blocks
> > > are
> > > >> shown as under-replicated.
> > > >>
> > > >> ---------------------------------
> > > >> Thanks & Regards
> > > >> Himanish
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards
> > > > Himanish
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards
> > Himanish
> >
>


-- 
Thanks & Regards
Himanish

--20cf304346d45a2abb04a40646dd--