helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Dynamic partition management
Date Sat, 03 Jun 2017 04:51:33 GMT
+helix

On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" <
subramanian.raghunathan@integral.com> wrote:

> Hi kishore,
>
>
>
>                 Trying to manage a cluster with a resource that has
> Dynamic number of partitions please provide me your valuable thoughts on
> same.
>
>
>
> Version(s) :
>
>                                 Helix: 0.7.1
>
>                                 Zookeeper:3.3.4
>
>
>
> -Zookeeper quorum (3 nodes)
>
> - State Model: OnlineOffline
>
> - Controller (leader elected from one of the cluster nodes)
>
> - Single resources with partitions.
>
> - Custom rebalancer
>
>
>
> Few of the approaches that I could think of ...
>
>
>
> Approach 1: *Buffer Partitions*
>
>                 Let’s say the current number of partitions are N.
>
>                                 1 ) Define N+X (Where x could be the
> delta where it can expand maximum , expecting to be of few in number
> (1-10))
>
>                                 2 ) Use property store to identify which
> are all valid partition(s)
>
>                                 3 ) Don’t act upon invalid ones by
> looking up property store (or) Disable the unallocated partition(s)
>
>
>
>                 When a new partition to be added -
>
>    1. Update property store of new partitions (Delta merge)
>    2. Enable the partition(s)
>
>
>
> --- The balancing algorithm should be able to balance valid & extra
> partition(s) differently. Lots of customizations.
>
>
>
> Approach 2: *New Resource Addition*
>
>                 Let’s say the current number of partitions are N.
>
>
>
>                 Create Resource 1 with N partitions initially.
>
>                 When a new partition(s) to be added -
>
>    1. Drop the resource
>    2. Create new resource with N+X (newly added partitions )
>
>
>
> --- Caveat that I could see is Dropping the resource would stop the entire
> workflow (It’s a bigger price to pay for us since we are real time).
>
>
>
>
>
> Thanks & Regards,
>
> Subramanian.
>
>
>
> Tel: +1 (650) 424 4655 <(650)%20424-4655>
>
> Mob:+1(650) 656 6006 <(650)%20656-6006>
>
>
>
> 850 Hansen Way,
>
> Palo Alto, CA 94304
>
> www.integral.com
>
> [image: Logo_signature_block]
> <http://www.integral.com/fxcloud_features/risk_management.html#ym>
>
> NOTICE: This e-mail message and any attachments, which may contain
> confidential information, are to be viewed solely by the intended recipient
> of Integral Development Corp. For further information, please visit
> http://www.integral.com/about/disclaimer.html.
>
>
>
>
>
>
>

Mime
View raw message