hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: HBase Replication use cases
Date Thu, 12 Apr 2012 21:31:30 GMT
On Thu, Apr 12, 2012 at 12:11 PM, Himanshu Vashishtha
<hvashish@cs.ualberta.ca> wrote:
> Hello All,
> I have been doing testing on the HBase replication (0.90.4, and 0.92 variants).
> Here are some of the findings:
> a) 0.90+ is not that great in handling out znode changes; in an
> ongoing replication, if I delete a peer and a region server goes to
> the znode to update the log status, the region server aborts itself
> when it sees a missing znode.

I don't think I've encountered that, and I've deleted peers a few times. Log?

> Recoverable Zookeeper seems to have fix this in 0.92+?
> 0.92 has lot of new features (start/stop handle, master master, cyclic).
> But there are corner cases with the start/stop switches.
> i)  A log is en-queue when the replication state is set to true. When we
> start the cluster, it is true and the starting region server takes the
> new log into the queue. If I do a stop_replication, and there is a log
> roll, and then I do a start_replication, the current log will not be
> replicated, as it has missed the opportunity of being added to the queue.

stop_replication is a kill switch, it should only be used in that
context eg you want to kill replication. It's not supposed to do
something reliable outside of killing replication.

> ii) If I _start_ a region server when the replication state is set to
> false, its log will not be added to the queue. Now, if I do a
> start_replication, its log will not be replicated.

See above.

> iii) Removing a peer doesn't result in master region server abort, but
> in case of zk is down and there is a log roll, it will abort. Not a
> serious one as zk is down so the cluster is not healthy anyway.

If zk is down, HBase should be down.

> I was looking for jiras (including 2611), and stumbled upon 2223. I
> don't think there is any thing like time based partition behavior (as
> mentioned in the jira description). Though. the patch has lot of other
> nice things which indeed are in existing code. Please correct me if I
> miss  anything.

The jira was about being able to handle a network partition that
lasted more than 10 minutes, originally the replication code was
buffering in memory so eventually it was OOMEing.

> Having said that, I wonder about other folks out there use it.
> Their experience, common issues (minor + major) they come across.
> I did find a ppt by Jean Daniel at oscon mentioning about using it in
> SU production.

In 0.90 if there was a ZK hiccup it could kill a bunch of region
servers when they wanted to talk to ZK. In 0.92 RecoverableZookeeper
handles it.

The master is very sloppy when keeping track of the logs that can be
deleted, it was built like that so that it wouldn't hit ZK too hard,
but it seems to be retaining too many logs.


View raw message