Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=pnNspLnXxsBoFsd3lpj/CSxYi8uyK0DCvgNQeMsd9EdvCUiBkd4Y5EZSKb47xRBUN6JKpVCBT0K2DszqqU5XLeG1Ngdp2o8S68vovFaHr//3gTQBbueOcL9w+VkCpL8G4rhflWcU7x0BSQKBYR1p5MsBFjyN2ZQiZsP12jo0X1w=;
References: 
 <CAFLnt_pNjBV=yFBF16f4OamyN1m3xg11MuzUqiwB=J8YgEYmmg@mail.gmail.com>
 <1334267417.71881.YahooMailNeo@web121704.mail.ne1.yahoo.com>
 <004640A3-EAF4-4C5D-ABF9-5936DFDAD762@gmail.com>
Message-ID: <1334361259.27978.YahooMailNeo@web121704.mail.ne1.yahoo.com>
Date: Fri, 13 Apr 2012 16:54:19 -0700 (PDT)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: HBase Replication use cases
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
In-Reply-To: <004640A3-EAF4-4C5D-ABF9-5936DFDAD762@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hey Lars,=0A=0Ain a DR scenario (i.e. a DC falls into the ocean) we SLAs th=
at allow for a certain amount of data loss.=0AThe main concern here would b=
e that "rows" could be in a state that does not correspond to the state at =
the end of any of the row transactions in the source system, right?=0A=0AOr=
 are you referring to even cross table consistency?=0A=0A=0A-- Lars=0A=0A=
=0A----- Original Message -----=0AFrom: Lars George <lars.george@gmail.com>=
=0ATo: dev@hbase.apache.org; lars hofhansl <lhofhansl@yahoo.com>=0ACc: =0AS=
ent: Thursday, April 12, 2012 11:13 PM=0ASubject: Re: HBase Replication use=
 cases=0A=0AHi Lars,=0A=0AI am really curious how you will handle the possi=
ble (or say likely) inconsistencies between regions of the same table in ca=
se of a DR situation. This seems to be solely applications layer logic but =
on the other hand a lot of people will need something here. So the question=
 is, could this be added to the code? The idea is, could we hint to the rep=
lication what schema we are using and it can therefore handle shipping the =
logs somewhat "transactional" on the receiving end? For example, it could r=
ecord sequence IDs or even timestamps and when the originating cluster fail=
s there is a mechanism on the receiving end that deletes all inconsistent c=
hanges, bringing it back to a well known checkpoint. The replication does s=
hip the WAL edits so, this might be all that is needed, and some ZooKeeper =
magic there to synchronize the checkpoint across the region servers?=0A=0AM=
aybe I am seeing this wrong here, but how else would you recover in the cas=
e of a DR situation?=0A=0ACheers,=0ALars=0A=0AOn Apr 12, 2012, at 11:50 PM,=
 lars hofhansl wrote:=0A=0A> Thanks Himanshu,=0A> =0A> we're planning to us=
e Replication for cross DC replication for DR (and we added a bunch of stuf=
f and fixed bugs in replication).=0A> =0A> =0A> We'll have it always on (an=
d only use stop/start_peer, which is new in 0.94+ to temporarily stop repli=
cation, rather than stop/start_replication)=0A> HBASE-2611 is a problem. We=
 did not have time recently to work on this.=0A> =0A> i) and ii) can be wor=
ked around by forcing a log roll on all region servers after replication wa=
s enabled. Replication would be considered started after the logs were=0A> =
rolled... But that is quite annoying.=0A> =0A> Is iii) still a problem in 0=
.92+? I thought we fixed that together with a).=0A> =0A> -- Lars=0A> =0A> _=
_______________________________=0A> From: Himanshu Vashishtha <hvashish@cs.=
ualberta.ca>=0A> To: dev@hbase.apache.org =0A> Sent: Thursday, April 12, 20=
12 12:11 PM=0A> Subject: HBase Replication use cases=0A> =0A> Hello All,=0A=
> =0A> I have been doing testing on the HBase replication (0.90.4, and 0.92=
 variants).=0A> =0A> Here are some of the findings:=0A> =0A> a) 0.90+ is no=
t that great in handling out znode changes; in an=0A> ongoing replication, =
if I delete a peer and a region server goes to=0A> the znode to update the =
log status, the region server aborts itself=0A> when it sees a missing znod=
e.=0A> =0A> Recoverable Zookeeper seems to have fix this in 0.92+?=0A> =0A>=
 0.92 has lot of new features (start/stop handle, master master, cyclic).=
=0A> =0A> But there are corner cases with the start/stop switches.=0A> i)=
=A0 A log is en-queue when the replication state is set to true. When we=0A=
> start the cluster, it is true and the starting region server takes the=0A=
> new log into the queue. If I do a stop_replication, and there is a log=0A=
> roll, and then I do a start_replication, the current log will not be=0A> =
replicated, as it has missed the opportunity of being added to the queue.=
=0A> =0A> ii) If I _start_ a region server when the replication state is se=
t to=0A> false, its log will not be added to the queue. Now, if I do a=0A> =
start_replication, its log will not be replicated.=0A> =0A> iii) Removing a=
 peer doesn't result in master region server abort, but=0A> in case of zk i=
s down and there is a log roll, it will abort. Not a=0A> serious one as zk =
is down so the cluster is not healthy anyway.=0A> =0A> I was looking for ji=
ras (including 2611), and stumbled upon 2223. I=0A> don't think there is an=
y thing like time based partition behavior (as=0A> mentioned in the jira de=
scription). Though. the patch has lot of other=0A> nice things which indeed=
 are in existing code. Please correct me if I=0A> miss=A0 anything.=0A> =0A=
> Having said that, I wonder about other folks out there use it.=0A> Their =
experience, common issues (minor + major) they come across.=0A> I did find =
a ppt by Jean Daniel at oscon mentioning about using it in=0A> SU productio=
n.=0A> =0A> I plan to file jiras for the above ones and will start digging =
in.=0A> =0A> Look forward for your responses.=0A> =0A> Thanks,=0A> Himanshu