helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Mapping from a resource partition to a logical data partition
Date Thu, 04 Feb 2016 07:31:06 GMT
I found an issue: the numberOfPartitions can only be increased, and
couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
old partitions still exist in idea view and external view; So, how can
descrease or drop a partition?



2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:

> Nice, that you Kishore!
>
> 2016-02-04 10:35 GMT+08:00 kishore g <g.kishore@gmail.com>:
>
>> This is needed for another project I am working on. There is no reason
>> for Helix to depend on this convention. I will fix this.
>>
>>
>>
>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <shaofengshi@apache.org>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on a
>>> cluster. My scenario is, every 5 minutes a partition will be added; What
>>> I
>>> do now is, get the ideal state, +1 for the partition number and then
>>> update
>>> it with ZKHelixAdmin. With a rebalance action, the new partition will be
>>> created and assigned to an instance. What the instance can get from Helix
>>> is the resource name and partition id. To map the parition to my logical
>>> data, I maintained a mapping table in our datastore, which looks like:
>>>
>>> {
>>>   "resource_0": 201601010000_ 201601010005,
>>>   "resource_1": 201601010005_ 201601010010,
>>>   ...
>>>   "resource_11": 201601010055_ 201601010100
>>> }
>>>
>>> Here it has 12 partitions; Now I want to discard some old partitions, say
>>> the first 6 partitions;  It seems in Helix the partitions must start from
>>> 0, so with an update on the IdealState, set # of partitions to 6, the new
>>> partitions on the cluster would become to:
>>>
>>> resource_0,resource_1, ..., resource_5,
>>>
>>> To make sure the partitions wouldn't be wrongly mapped, I need update my
>>> mapping table before the rebalance. While that may not ensure the atomic
>>> between the two updates.
>>>
>>> So my question is, what's the suggested way to do the resource partition
>>> mapping? does Helix allow user to specify additional information on a
>>> partition (if have this, I don't need maintain the mapping outside)? Can
>>> we
>>> have some simple APIs like addPartition(String parititionName),
>>> dropParitition(String partitionName), just like that for resource? The
>>> numeric paritition id can be an internal ID and not exposed to user.
>>>
>>> I guess many entry users will have such questions. Just raise this for a
>>> broad discussion; Any comment and suggestion is welcomed. Thanks for your
>>> time!
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>


-- 
Best regards,

Shaofeng Shi

Mime
View raw message