zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fournier, Camille F." <Camille.Fourn...@gs.com>
Subject RE: Zookeeper on two clusters?
Date Fri, 26 Aug 2011 22:08:16 GMT
Perhaps what you want is more of a messaging solution... Have you looked into, say ActiveMQ
or RabbitMQ? You can use ZK for this sort of confirmation stuff but it feels a bit like overkill.

C

-----Original Message-----
From: Aleksey Yakovlev [mailto:aleksey.yakovlev@gmail.com] 
Sent: Friday, August 26, 2011 6:06 PM
To: user@zookeeper.apache.org
Subject: Re: Zookeeper on two clusters?

Ted:

So, if I find a way to talk to one cluster from another using let's say the
netcat - it'll be enough for ZK, right?

As for the architecture and motivation - there is a lot of people, who
designed them and are managing them, and I'm not one of them. In my opinion
these clusters are too loosely connected, so it's very difficult to build a
distributed software system (which I'm doing) on them.

A simple example: if one cluster sends a file to another, and this another
one gets this file corrupted and needs to ask the first one to resend - it
doesn't have a simple way to do that. Actually, it can send a short file as
a confirmation - but this transaction can also fail. Then the first one will
need to say "I'm waiting for confirmation - where is it?" So, we are talking
about some communication protocol - which is already in the ZK, right?
That's why I'm thinking about the ZK. The second cluster could send the
confirmation to the ZK, and the first one
could receive it from the ZK. And it'd be the ZK internal matter to
replicate the message between its servers.

BTW, what is the "observer" you mentioned?

Thanks
AY

On Fri, Aug 26, 2011 at 5:04 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Not so much mutual pingability, but mutual tcp-ability.
>
> But seriously, why not run ZK in one cluster and put an observer in the

> other cluster?  Or do that bi-directionally?
>
> Or put two machines in each cluster and a tie-breaker somewhere outside
> like
> Amazon's EC2?
>
> You haven't provided any real motivation for your original architecture.
>  What are you really trying to do? For instance, you say a "distributed
> systems works in both centers" and needs "fail-over".  What do you really
> mean?
>
> Do you want both data centers to continue to operate in the even of
> partition?  In that case you need two separate ZK clusters, possibly with
> some cross-datacenter ephemeral files.
>
> Do you want one data center to stop work if it loses contact with the
> other?
>  In that case, put a single ZK cluster in one data-center.
>
> Do you want to ignore most partition issues, but make sure that exactly one
> data-center is considered master at all times and that at least one is
> considered master as much as possible?  In that case, use the external
> tie-breaker architecture.
>
>
> On Fri, Aug 26, 2011 at 1:39 PM, Aleksey Yakovlev <
> aleksey.yakovlev@gmail.com> wrote:
>
> > Camille:
> >
> > Sorry to say, I can't ping one cluster from another...
> > You are saying, that all the servers mutual pingability is necessary and
> > sufficient for ZK to operate, right?
> >
> > Thanks
> > AY
> >
> > On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. <
> > Camille.Fournier@gs.com> wrote:
> >
> > > As long as the two clusters can ping each other, just set up a single
> ZK
> > > cluster spread across the two, 3 or 5 nodes (total, not per cluster).
> > Note
> > > that if these two clusters are used for business continuity purposes
> > (spread
> > > across 2 data centers), you still risk the outage of the zookeeper if
> you
> > > lose one of the data centers (the one containing the majority of the ZK
> > > nodes).
> > >
> > > There's no reason to set up 2 ZK quorums and merge them.
> > >
> > > C
> > >
> > > -----Original Message-----
> > > From: Aleksey Yakovlev [mailto:aleksey.yakovlev@gmail.com]
> > > Sent: Friday, August 26, 2011 10:09 AM
> > > To: user@zookeeper.apache.org
> > > Subject: Re: Zookeeper on two clusters?
> > >
> > > Mahadev:
> > >
> > > I need these two clusters to communicate somehow, because our
> distributed
> > > system works on both of them - so we need some coordination, for
> example,
> > > to
> > > recover after failures. I could create my own servers/protocols for
> that,
> > > but why - the ZooKeeper does just that, right?
> > >
> > > The number 5 isn't important here, configuration and communication is
> > >
> > > Thanks
> > > AY
> > >
> > > On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <
> mahadev@hortonworks.com
> > > >wrote:
> > >
> > > > Aleksey,
> > > >  I am a little confused. Whats the motivation behind merging the
> > > > quorums? Why do you want 5 on each cluster?
> > > >
> > > > thanks
> > > > mahadev
> > > >
> > > > On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
> > > > <aleksey.yakovlev@gmail.com> wrote:
> > > > > Hi,
> > > > >
> > > > > We have two clusters but I can't just ssh from one cluster to
> another
> > -
> > > I
> > > > > have to enter a password, use some authentication, go through a
> load
> > > > > balancer etc
> > > > > It would be very useful for us to use the Zookeeper as a common
> > storage
> > > > > place for small configuration data, accessible from both these
> > > clusters.
> > > > >
> > > > > So, I'm thinking about starting like 5 servers on each cluster and
> > > merge
> > > > > them into a single quorum - anybody did that?
> > > > > If yes, how to configure this double-side quorum?
> > > > >
> > > > > (I'm completely new to the Zookeeper)
> > > > >
> > > > > Thanks
> > > > > AY
> > > > >
> > > >
> > >
> >
>

Mime
View raw message