Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of asaf.mesika@gmail.com
 designates 209.85.219.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAFLnt_po2E8NGcJ4NgXKeQOo9_5DDYAkPONZpFrx9WLqS7EmLw@mail.gmail.com>
References: 
 <CA+r7Yvmt-5OR6d4_dWBUZ9BLngO1h7tmeCN-0x0PnpX2yeOyrQ@mail.gmail.com>
	<CALte62wSH8+e5=1nC9xneCn+Dni4FJkBEFAy7HtEGavLaRGqhg@mail.gmail.com>
	<E3F55F00-333B-4D92-8037-EE7FE1F491F2@gmail.com>
	<CALte62yjrmDJeAUe1jkNxvuvk1z+4xWaT2k+7jgF+b+zNKW3RQ@mail.gmail.com>
	<CAFLnt_po2E8NGcJ4NgXKeQOo9_5DDYAkPONZpFrx9WLqS7EmLw@mail.gmail.com>
Date: Mon, 3 Jun 2013 08:18:43 +0300
Message-ID: 
 <CA+r7Yvn6sdKYWrHv9egnnuEoNuPWxGXvp7pmnhiEiAvj7yZZnA@mail.gmail.com>
Subject: Re: Weird Replication exception
From: Asaf Mesika <asaf.mesika@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=089e0115f3963261c804de391945

--089e0115f3963261c804de391945
Content-Type: text/plain; charset=UTF-8

No, this was brand new with 0 length thus the peculiar message of too old
was strange to me.

On Monday, June 3, 2013, Himanshu Vashishtha wrote:

> Hey Asaf,
>
> It looks like you only need 7122. Either upgrade, or you could also patch
> it up.
>
> Syncing up the master and slave cluster is also advised, but that stands
> good in case you are using master-master replication.
>
>
> bq. 172.25.98.74,60020,
> 1369903540894/172.25.98.74
> %2C60020%2C1369903540894.1369925171871
>
> bq.    Meaning I lost data
>
> Did the log whose znode you deleted had any data?
> You can do a cat to see if there is any data on it. You could copy-table
> for that time range, (or a hacky way is to re-create a znode for that log
> under a regionserver noticing the format of its current log znodes, and let
> the replicationSource pick it up in its normal run).
>
> Thanks,
> Himanshu
>
>
>
> On Sun, Jun 2, 2013 at 12:38 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > bq. Is 0.94.8 production ready?
> >
> > I think so. Lars released 0.94.8 Friday evening.
> >
> > On Sun, Jun 2, 2013 at 12:26 PM, Asaf Mesika <asaf.mesika@gmail.com>
> > wrote:
> >
> > > I use 0.94.7.
> > > Is 0.94.8 production ready?
> > >
> > > So in summary I have two issues:
> > > 1. Clocks are out of sync
> > > 2. I need to upgrade to 0.94.8 to avoid seeing this WARN messages?
> > >
> > > On Jun 2, 2013, at 5:46 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > What is the HBase version you're using ?
> > > >
> > > > In another thread, I mentioned this:
> > > >
> > > > There was a recently integrated JIRA (0.94.8):
> > > > HBASE-7122 Proper warning message when opening a log file with no
> > entries
> > > > (idle cluster)
> > > >
> > > > Does the HBase you're using contain HBASE-7122 ?
> > > >
> > > > Cheers
> > > >
> > > > On Sat, Jun 1, 2013 at 10:20 PM, Asaf Mesika <asaf.mesika@gmail.com>
> > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I have a weird error in a cluster I'm checking Replication with.
> > > >>
> > > >> I have two clusters set up, each on its own DC (different
> continents).
> > > Each
> > > >> has 1 master, and 3 RS.
> > > >>
> > > >> I've done all required setup, started replication and pushed in some
> > > data
> > > >> into the master. I had an issue where the slave (peer) cluster went
> > dead
> > > >> (all RS failed contacting the master), thus replication couldn't
> work.
> > > This
> > > >> happened right before the weekend, so it was out for 3 days.
> > > >>
> > > >> Now I'm back in the office - got slave cluster back up (just the
> RS),
> > > and I
> > > >> got some nasty exception in one of the RS of the master cluster:
> > > >>
> > > >> 2013-06-02 04:40:45,903 INFO
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Getting
> > > >> 0 rs from peer cluster # c
> > > >> 2013-06-02 04:40:45,903 INFO
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Slave
> > > >> cluster looks down: c has 0 region servers
> > > >> 2013-06-02 04:40:46,903 DEBUG
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Since
> > > >> we are unable to replicate, sleeping 1000 times 10
> > > >> 2013-06-02 04:40:57,019 INFO
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Getting
> > > >> 0 rs from peer cluster # c
> > > >> 2013-06-02 04:40:57,019 INFO
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Slave
> > > >> cluster looks down: c has 0 region servers
> > > >> 2013-06-02 04:40:58,019 DEBUG
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Since
> > > >> we are unable to replicate, sleeping 1000 times 10
> > > >> 2013-06-02 04:41:08,134 INFO
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > Getting
> > > >> 1 rs from peer cluster # c
> > > >> 2013-06-02 04:41:08,134 INFO
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > >> Choosing peer a72-246-95-86,60020,1370147274693
> > > >> 2013-06-02 04:41:08,672 DEBUG
> > > >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > > >> Replicating 1
> > > >> 2013-06-02 04:41:08,971 INFO
> > > >>
> > >
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceM

--089e0115f3963261c804de391945--