Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of ndimiduk@gmail.com designates
 74.125.82.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <0D4EC5F0-CF9C-4A7D-9B4C-A3378BDE8548@segel.com>
References: 
 <CAHg+FCLNmDK0naWz++WJzvBkWjmkT6xFhtx6f1w1TfPWZ0KLNg@mail.gmail.com>
 <0D4EC5F0-CF9C-4A7D-9B4C-A3378BDE8548@segel.com>
From: Nick Dimiduk <ndimiduk@gmail.com>
Date: Mon, 23 Feb 2015 12:34:25 -0800
Message-ID: 
 <CANZa=Gszv_N5530D2A4wTppfA7X=B-rTx3AKbNPPVvyONgQB+g@mail.gmail.com>
Subject: Re: HBase Region always in transition + corrupt HDFS
To: hbase-user <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=bcaec53f3583381214050fc75577

--bcaec53f3583381214050fc75577
Content-Type: text/plain; charset=UTF-8

HBase/HDFS are maintaining block checksums, so presumably a corrupted block
would fail checksum validation. Increasing the number of replicas increases
the odds that you'll still have a valid block. I'm not an HDFS expert, but
I would be very surprised if HDFS is validating a "questionable block" via
byte-wise comparison over the network amongst the replica peers.

On Mon, Feb 23, 2015 at 12:25 PM, Michael Segel <msegel@segel.com> wrote:

>
> On Feb 23, 2015, at 1:47 AM, Arinto Murdopo <arinto@gmail.com> wrote:
>
> We're running HBase (0.94.15-cdh4.6.0) on top of HDFS (Hadoop
> 2.0.0-cdh4.6.0).
> For all of our tables, we set the replication factor to 1 (dfs.replication
> = 1 in hbase-site.xml). We set to 1 because we want to minimize the HDFS
> usage (now we realize we should set this value to at least 2, because
> "failure is a norm" in distributed systems).
>
>
>
> Sorry, but you really want this to be a replication value of at least 3
> and not 2.
>
> Suppose you have corruption but not a lost block. Which copy of the two is
> right?
> With 3, you can compare the three and hopefully 2 of the 3 will match.
>
>

--bcaec53f3583381214050fc75577--