Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 74.125.44.29 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=QFoBUZaOlPyzG+ebuTZPx60g81ufpRRewuIGaIKmbpCXr8fv6Z25caOsIX3vXlpNkB
         6bwaazIXfFIihQiMjrFNtniEo9GDhFHJxoqVkBaLtRZc+Y8RknjYEF/yZAsGWmr0ljtz
         zU7duT7Z+T3hd4N20i8bt1K6+QNRfYvi5G2H4=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <49761F90.7060502@duboce.net>
References: <9683564c0901201030m6cac505w1093f72ee257e8fe@mail.gmail.com>
	 <49761F90.7060502@duboce.net>
Date: Wed, 21 Jan 2009 21:43:22 -0800
Message-ID: <7c962aed0901212143r2c213b19j794f676f3a7fa349@mail.gmail.com>
Subject: Re: Probably lack-of-HADOOP-1700 causing DATA LOSS
From: stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e644cb7a20394f04610bc0e5

--0016e644cb7a20394f04610bc0e5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Any luck with this Genady?
St.Ack

On Tue, Jan 20, 2009 at 11:01 AM, stack <stack@duboce.net> wrote:

> My guess is that those files are empty.  Can you check them using
> './bin/hadoop fs -ls
> hdfs://master:54310/hbase/step0/980646640/oldlogfile.log'.  They may have
> been removed though.
>
> They are written by master splitting logs of a crashed regionserver so can
> you back up and look see why regionserver crashed in first place?
>
> HADOOP-1700 has been closed but hbase still needs some HDFS loving
> minimizing dataloss.  See HADOOP-4379 (Hopefully fixed in hadoop 0.19.1).
>
> St.Ack
>
>
>
>
> Genady Gillin wrote:
>
>> Hi,
>>
>> We're trying to use Hadoop0.19/Hbase 0.19 RC with four nodes, region
>> server
>> already has about 150 regions, each few successful runs one of region
>> servers keep to close all regions and fail the running task with the log
>> errors are listed below, complaining about Hadoop-1700 issue, which
>> already
>> fixed in Hadoop 19, is there is some workaround to prevent this behaviour
>> ?
>>
>> Thanks,
>> Gennady
>>
>>
>> *2009-01-20 10:46:53,049 WARN org.apache.hadoop.hbase.regionserver.HStore:
>> Exception processing reconstruction log
>> hdfs://master:54310/hbase/step0/980646640/oldlogfile.log opening
>> [B@940592-- continuing.  Probably lack-of-HADOOP-1700 causing DATA
>> LOSS!
>> java.io.EOFException
>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HStore.doReconstructionLog(HStore.java:342)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HStore.runReconstructionLog(HStore.java:297)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:237)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
>>        at java.lang.Thread.run(Thread.java:619)
>> 2009-01-20 10:46:53,303 WARN org.apache.hadoop.hbase.regionserver.HStore:
>> Exception processing reconstruction log
>> hdfs://master:54310/hbase/step0/980646640/oldlogfile.log opening
>> [B@980af4-- continuing.  Probably lack-of-HADOOP-1700 causing DATA
>> LOSS!
>> java.io.EOFException
>>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>>        at
>>
>> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HStore.doReconstructionLog(HStore.java:342)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HStore.runReconstructionLog(HStore.java:297)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:237)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
>>        at java.lang.Thread.run(Thread.java:619)
>> *
>>
>>
>>
>
>

--0016e644cb7a20394f04610bc0e5--