curator-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Drew Kutcharian (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CURATOR-321) Recipe for Group Membership with Partition/Shard Assignment
Date Sat, 30 Apr 2016 18:44:12 GMT
Drew Kutcharian created CURATOR-321:
---------------------------------------

             Summary: Recipe for Group Membership with Partition/Shard Assignment
                 Key: CURATOR-321
                 URL: https://issues.apache.org/jira/browse/CURATOR-321
             Project: Apache Curator
          Issue Type: Wish
            Reporter: Drew Kutcharian
            Assignee: Jordan Zimmerman


It'd be great if Curator had a recipe for partition/shard assignment, similar to Kafka's consumer
rebalancing. It could be thought of as the generalization of what Kafka consumers do using
Zookeeper.

Use Case:
You have 5 worker machines that need to process data from 10 database shards, but you want
each worker to be pinned to a single database so you don't have to worry about race conditions.

Solution:
A more advanced group management recipe that:
1. Has a concept of slots (or paritions). You can create a group with say 1000 slots.
2. When a node joins the group, it gets assigned a range of the slots. If it's the first node,
then it will get all the slots. If there are already nodes registered in the group, then they
will split the slots equally between them.
3. When a node leaves the group, the rest of the nodes rebalance so they each end up with
a equal number of remaining slots.

I am aware that something like this can be done using Storm et al but I feel that this primitive
would be useful on its own in Curator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message