curator-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Drew Kutcharian (JIRA)" <>
Subject [jira] [Commented] (CURATOR-321) Recipe for Group Membership with Partition/Shard Assignment
Date Sun, 01 May 2016 03:00:16 GMT


Drew Kutcharian commented on CURATOR-321:

Thanks. I wasn't really expecting you to build it, just putting out there to see if it already
exists and if there is interest for it.
I'll try to dig through Kafka "old" consumer code and see if I can put something together.
Will appreciate your code review though.

> Recipe for Group Membership with Partition/Shard Assignment
> -----------------------------------------------------------
>                 Key: CURATOR-321
>                 URL:
>             Project: Apache Curator
>          Issue Type: Wish
>          Components: Recipes
>            Reporter: Drew Kutcharian
>            Assignee: Jordan Zimmerman
>            Priority: Minor
> It'd be great if Curator had a recipe for partition/shard assignment, similar to Kafka's
consumer rebalancing. It could be thought of as the generalization of what Kafka consumers
do using Zookeeper.
> Use Case:
> You have 5 worker machines that need to process data from 10 database shards, but you
want each worker to be pinned to a single database so you don't have to worry about race conditions.
> Solution:
> A more advanced group management recipe that:
> 1. Has a concept of slots (or paritions). You can create a group with say 1000 slots.
> 2. When a node joins the group, it gets assigned a range of the slots. If it's the first
node, then it will get all the slots. If there are already nodes registered in the group,
then they will split the slots equally between them.
> 3. When a node leaves the group, the rest of the nodes rebalance so they each end up
with a equal number of remaining slots.
> I am aware that something like this can be done using Storm et al but I feel that this
primitive would be useful on its own in Curator.

This message was sent by Atlassian JIRA

View raw message