curator-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Drew Kutcharian (JIRA)" <>
Subject [jira] [Commented] (CURATOR-321) Recipe for Group Membership with Partition/Shard Assignment
Date Sun, 01 May 2016 02:15:12 GMT


Drew Kutcharian commented on CURATOR-321:

[~randgalt], wouldn't that be against TN6 ;)

> Recipe for Group Membership with Partition/Shard Assignment
> -----------------------------------------------------------
>                 Key: CURATOR-321
>                 URL:
>             Project: Apache Curator
>          Issue Type: Wish
>          Components: Recipes
>            Reporter: Drew Kutcharian
>            Assignee: Jordan Zimmerman
>            Priority: Minor
> It'd be great if Curator had a recipe for partition/shard assignment, similar to Kafka's
consumer rebalancing. It could be thought of as the generalization of what Kafka consumers
do using Zookeeper.
> Use Case:
> You have 5 worker machines that need to process data from 10 database shards, but you
want each worker to be pinned to a single database so you don't have to worry about race conditions.
> Solution:
> A more advanced group management recipe that:
> 1. Has a concept of slots (or paritions). You can create a group with say 1000 slots.
> 2. When a node joins the group, it gets assigned a range of the slots. If it's the first
node, then it will get all the slots. If there are already nodes registered in the group,
then they will split the slots equally between them.
> 3. When a node leaves the group, the rest of the nodes rebalance so they each end up
with a equal number of remaining slots.
> I am aware that something like this can be done using Storm et al but I feel that this
primitive would be useful on its own in Curator.

This message was sent by Atlassian JIRA

View raw message