Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 209.85.212.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=n5bVP2sgglqThSBRMde+Ccxi86V1L4X6o+/HkAQ1Bci2fANOWpC7DXfwl0+BA6CZoJ
         dI16SbOLVtEQHZVq3SHqX6D2m08Cuk70+etxNxPycmFXOh1uXHFU0xxPBnSwpHMRF0Vl
         0doaj69qwp8xJ+PQvPzFrVmInqks/AjsMPeu4=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15BBEE3130@exchange2007.carrieriq.com>
References: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15BBEE3130@exchange2007.carrieriq.com>
Date: Thu, 22 Jul 2010 16:38:18 -0700
Message-ID: <AANLkTinFg4seEU_3JCLpo7zVm-wMoUYc8uaQx2DSTzep@mail.gmail.com>
Subject: Re: Data disappears and re-appears again after HBase cluster restart
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I would guess clock skew, all the machines have approx the same time?
A few seconds is acceptable, but not more.

J-D

On Thu, Jul 22, 2010 at 4:34 PM, Vladimir Rodionov
<vrodionov@carrieriq.com> wrote:
> Have anybody encountered this particular bug before?
> We have been having this intermittently in our QA small cluster.
>
> We run a flow =A0which is basically custom ETL process over data stored i=
n hdfs. Yes it is a bunch of M/R jobs.
> One of the jobs stores data into HBase (0.20.3), the next one loads data =
from HBase (using scan) performs additional transformations
> and stores data finally into RDBMS.
>
> Flow works fine (most of the time). It means that new HBase tables are cr=
eated, data is loaded and can be read after that during the next M/R job
>
> After flow finishes , data from tables (but not tables itself), sometimes=
, mysteriously disappear. This is not deterministic and to get data back we=
 need to RESTART HBase cluster.
> So HBase restart fixes the problem.
>
> Cluster is small (3 servers). RAM is limited - 8GB. Only 2 CPU cores per =
server but input data size is small as well and the average size of disappe=
aring tables is several 1000s rows-
> they are small. Hadoop is from CHD2. I can not get you any additional hel=
pful information at the time (no log files), but may be somebody has encoun=
tered this
> before and has idea how to fix it.
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>