helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subramanian Raghunathan <subramanian.raghunat...@integral.com>
Subject RE: Dynamic partition management
Date Wed, 07 Jun 2017 21:58:55 GMT
Hi,

                Trying to follow up if there are any thoughts/suggestions over the approaches.

                Trying to manage a cluster with a resource that has Dynamic number of partitions
please provide me your valuable thoughts on same.
 Version(s) :
                                Helix: 0.7.1
                                Zookeeper:3.3.4
 -Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer
 Few of the approaches that I could think of ...
 Approach 1: Buffer Partitions
                Let’s say the current number of partitions are N.
                                1 ) Define N+X (Where x could be the delta where it can expand
maximum , expecting to be of few in number (1-10))
                                2 ) Use property store to identify which are all valid partition(s)
                                3 ) Don’t act upon invalid ones by looking up property store
(or) Disable the unallocated partition(s)

                When a new partition to be added -
1.       Update property store of new partitions (Delta merge)
2.       Enable the partition(s)
 --- The balancing algorithm should be able to balance valid & extra partition(s) differently.
Lots of customizations.
 Approach 2: New Resource Addition
                Let’s say the current number of partitions are N.
                Create Resource 1 with N partitions initially.
                When a new partition(s) to be added -
1.       Drop the resource
2.       Create new resource with N+X (newly added partitions )
 --- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s
a bigger price to pay for us since we are real time).



Thanks & Regards,
Subramanian.

From: kishore g [mailto:g.kishore@gmail.com]
Sent: Friday, June 02, 2017 9:52 PM
To: Subramanian Raghunathan <subramanian.raghunathan@integral.com>; user@helix.apache.org
Subject: Re: Dynamic partition management


+helix

On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" <subramanian.raghunathan@integral.com<mailto:subramanian.raghunathan@integral.com>>
wrote:
Hi kishore,

                Trying to manage a cluster with a resource that has Dynamic number of partitions
please provide me your valuable thoughts on same.

Version(s) :
                                Helix: 0.7.1
                                Zookeeper:3.3.4

-Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer

Few of the approaches that I could think of ...

Approach 1: Buffer Partitions
                Let’s say the current number of partitions are N.
                                1 ) Define N+X (Where x could be the delta where it can expand
maximum , expecting to be of few in number (1-10))
                                2 ) Use property store to identify which are all valid partition(s)
                                3 ) Don’t act upon invalid ones by looking up property store
(or) Disable the unallocated partition(s)

                When a new partition to be added -
3.       Update property store of new partitions (Delta merge)
4.       Enable the partition(s)

--- The balancing algorithm should be able to balance valid & extra partition(s) differently.
Lots of customizations.

Approach 2: New Resource Addition
                Let’s say the current number of partitions are N.

                Create Resource 1 with N partitions initially.
                When a new partition(s) to be added -
3.       Drop the resource
4.       Create new resource with N+X (newly added partitions )

--- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s
a bigger price to pay for us since we are real time).


Thanks & Regards,
Subramanian.

Tel: +1 (650) 424 4655<tel:(650)%20424-4655>
Mob:+1(650) 656 6006<tel:(650)%20656-6006>

850 Hansen Way,
Palo Alto, CA 94304
www.integral.com<http://www.integral.com/>
[Logo_signature_block]<http://www.integral.com/fxcloud_features/risk_management.html#ym>

NOTICE: This e-mail message and any attachments, which may contain confidential information,
are to be viewed solely by the intended recipient of Integral Development Corp. For further
information, please visit http://www.integral.com/about/disclaimer.html.



Mime
View raw message