helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Self managed cluster
Date Sun, 24 Feb 2013 17:00:35 GMT
Adding Sunil( you might have to subscribe to reply), who developed a
similar functionality at LinkedIn for databus consumer group.
Only difference is he runs zookeeper on separate machines.
3. I remember he wanted getCluster or existsCluster feature, not sure how
he worked around it. Appreciate if you can file a jira and submit a patch
for this.
4. You dont need semaphore. Helix already has the feature where you can
start multiple controllers, only one will be active.

thanks
Kishore



On Sat, Feb 23, 2013 at 9:44 PM, Ming Fang <mingfang@mac.com> wrote:

> I'm looking to avoid maintaining separate Zookeepers and Controller
> processes.
> I want each of my process in the cluster to be more or less a peer to
> peer, self maintained cluster.
>
> Each process will behave something like this:
> 1-start a ZooKeeper QuorumPeerMain
> 2-connect to ZooKeeper
> 3-if Helix cluster is not already defined then define it.
>     Unfortunately HelixAdmin does have a getCluster() call.
>     Also will probably have to hold a distributed lock to avoid another
> peer from trying to create cluster too.
> 4-Start the cluster Controller.
>    Should use a distributed semaphore to prevent more than one Controller.
> 5-Finally start the Participant.
>
>
>
> On Feb 23, 2013, at 4:21 PM, Kapil Surlaker <kapil.surlaker@gmail.com>
> wrote:
>
> Ming:
>
> Can you explain the problem you are trying to solve?
>
> It's certainly possible to have one of the nodes itself be the controller,
> which you can do using the leader/standby pattern. Using that same pattern,
> you can ensure that one of the nodes will be able to manipulate the ideal
> state of the cluster you want to setup.
>
> If you can provide more details about your usecase, that will be very
> helpful.
>
> Thanks,
> Kapil
>
>
>
> On Fri, Feb 22, 2013 at 6:07 PM, Ming Fang <mingfang@mac.com> wrote:
>
>> Hi
>>
>> I am looking for feasibility advice on building a self managed cluster.
>> Basically I don't have to have to maintain a separate Zookeeper and/or
>> Controller processes.
>>
>> If I have a Master and a Slave, each will embed a Zookeeper node, forming
>> a Zookeeper cluster.
>> Then one of them will setup the Helix cluster.
>> This can be done by first creating global Zookeeper lock before setting
>> up the cluster if it does already exist.
>> It doesn't look like Helix supports distribute locks, but it's not hard
>> to do.
>>
>> The biggest problem I see in this is that with even number of nodes,
>> Zookeeper can no form a quorum and that can result in split brain.
>> Do you see any other problems with this design?
>>
>> Thanks
>> --ming
>>
>>
>
>

Mime
View raw message