helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Self managed cluster
Date Sun, 24 Feb 2013 17:00:35 GMT
Adding Sunil( you might have to subscribe to reply), who developed a
similar functionality at LinkedIn for databus consumer group.
Only difference is he runs zookeeper on separate machines.
3. I remember he wanted getCluster or existsCluster feature, not sure how
he worked around it. Appreciate if you can file a jira and submit a patch
for this.
4. You dont need semaphore. Helix already has the feature where you can
start multiple controllers, only one will be active.


On Sat, Feb 23, 2013 at 9:44 PM, Ming Fang <mingfang@mac.com> wrote:

> I'm looking to avoid maintaining separate Zookeepers and Controller
> processes.
> I want each of my process in the cluster to be more or less a peer to
> peer, self maintained cluster.
> Each process will behave something like this:
> 1-start a ZooKeeper QuorumPeerMain
> 2-connect to ZooKeeper
> 3-if Helix cluster is not already defined then define it.
>     Unfortunately HelixAdmin does have a getCluster() call.
>     Also will probably have to hold a distributed lock to avoid another
> peer from trying to create cluster too.
> 4-Start the cluster Controller.
>    Should use a distributed semaphore to prevent more than one Controller.
> 5-Finally start the Participant.
> On Feb 23, 2013, at 4:21 PM, Kapil Surlaker <kapil.surlaker@gmail.com>
> wrote:
> Ming:
> Can you explain the problem you are trying to solve?
> It's certainly possible to have one of the nodes itself be the controller,
> which you can do using the leader/standby pattern. Using that same pattern,
> you can ensure that one of the nodes will be able to manipulate the ideal
> state of the cluster you want to setup.
> If you can provide more details about your usecase, that will be very
> helpful.
> Thanks,
> Kapil
> On Fri, Feb 22, 2013 at 6:07 PM, Ming Fang <mingfang@mac.com> wrote:
>> Hi
>> I am looking for feasibility advice on building a self managed cluster.
>> Basically I don't have to have to maintain a separate Zookeeper and/or
>> Controller processes.
>> If I have a Master and a Slave, each will embed a Zookeeper node, forming
>> a Zookeeper cluster.
>> Then one of them will setup the Helix cluster.
>> This can be done by first creating global Zookeeper lock before setting
>> up the cluster if it does already exist.
>> It doesn't look like Helix supports distribute locks, but it's not hard
>> to do.
>> The biggest problem I see in this is that with even number of nodes,
>> Zookeeper can no form a quorum and that can result in split brain.
>> Do you see any other problems with this design?
>> Thanks
>> --ming

View raw message