helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Mapping from a resource partition to a logical data partition
Date Fri, 05 Feb 2016 02:49:56 GMT
I didn't; But after add the transitions between Dropped and Offline, it got
the same behavior.
I also changed my code to drop the resource and create again when its
partition number reaches a limit; But the external view wasn't deleted,
that makes Helix thought all the partitions have already been propery
assigned; Following is a state after the resoruce be recreated, you can see
although its "NUM_PARTITIONS" is 8, the partitions in external view is 20
(which is the up limit):

IdealState for Resource_Stream:
{
  "id" : "Resource_Stream",
  "mapFields" : {
    "Resource_Stream_0" : {
    },
    "Resource_Stream_1" : {
    },
    "Resource_Stream_10" : {
    },
    "Resource_Stream_11" : {
    },
    "Resource_Stream_12" : {
    },
    "Resource_Stream_13" : {
    },
    "Resource_Stream_14" : {
    },
    "Resource_Stream_15" : {
    },
    "Resource_Stream_16" : {
    },
    "Resource_Stream_17" : {
    },
    "Resource_Stream_18" : {
    },
    "Resource_Stream_19" : {
    },
    "Resource_Stream_2" : {
    },
    "Resource_Stream_3" : {
    },
    "Resource_Stream_4" : {
    },
    "Resource_Stream_5" : {
    },
    "Resource_Stream_6" : {
    },
    "Resource_Stream_7" : {
    },
    "Resource_Stream_8" : {
    },
    "Resource_Stream_9" : {
    }
  },
  "listFields" : {
    "Resource_Stream_0" : [ ],
    "Resource_Stream_1" : [ ],
    "Resource_Stream_10" : [ ],
    "Resource_Stream_11" : [ ],
    "Resource_Stream_12" : [ ],
    "Resource_Stream_13" : [ ],
    "Resource_Stream_14" : [ ],
    "Resource_Stream_15" : [ ],
    "Resource_Stream_16" : [ ],
    "Resource_Stream_17" : [ ],
    "Resource_Stream_18" : [ ],
    "Resource_Stream_19" : [ ],
    "Resource_Stream_2" : [ ],
    "Resource_Stream_3" : [ ],
    "Resource_Stream_4" : [ ],
    "Resource_Stream_5" : [ ],
    "Resource_Stream_6" : [ ],
    "Resource_Stream_7" : [ ],
    "Resource_Stream_8" : [ ],
    "Resource_Stream_9" : [ ]
  },
  "simpleFields" : {
    "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
    "INSTANCE_GROUP_TAG" : "Tag_StreamBuilder",
    "NUM_PARTITIONS" : "8",
    "REBALANCE_MODE" : "FULL_AUTO",
    "REPLICAS" : "2",
    "STATE_MODEL_DEF_REF" : "LeaderStandby",
    "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
  }
}

ExternalView for Resource_Stream:
{
  "id" : "Resource_Stream",
  "mapFields" : {
    "Resource_Stream_0" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_1" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_10" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_11" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_12" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_13" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_14" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_15" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_16" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_17" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_18" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_19" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_2" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_3" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_4" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_5" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_6" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_7" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    },
    "Resource_Stream_8" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "LEADER",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "STANDBY"
    },
    "Resource_Stream_9" : {
      "kylin-dev3-eaz-969995.phx01.eaz.ebayc3.com_8080" : "STANDBY",
      "kylin-dev4-eaz-687945.phx01.eaz.ebayc3.com_8080" : "LEADER"
    }
  },
  "listFields" : {
  },
  "simpleFields" : {
    "BUCKET_SIZE" : "0"
  }
}

2016-02-04 23:08 GMT+08:00 kishore g <g.kishore@gmail.com>:

> Ha ave you defined offline to dropped state transition in your state
> transition handler.
> On Feb 3, 2016 11:31 PM, "ShaoFeng Shi" <shaofengshi@apache.org> wrote:
>
>> I found an issue: the numberOfPartitions can only be increased, and
>> couldn't be decreased; Even if I set it to 0, and then do a rebalance, the
>> old partitions still exist in idea view and external view; So, how can
>> descrease or drop a partition?
>>
>>
>>
>> 2016-02-04 15:28 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:
>>
>>> Nice, that you Kishore!
>>>
>>> 2016-02-04 10:35 GMT+08:00 kishore g <g.kishore@gmail.com>:
>>>
>>>> This is needed for another project I am working on. There is no reason
>>>> for Helix to depend on this convention. I will fix this.
>>>>
>>>>
>>>>
>>>> On Wed, Feb 3, 2016 at 5:51 PM, ShaoFeng Shi <shaofengshi@apache.org>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to use Helix (0.7.1) to manage our resource partitions on
a
>>>>> cluster. My scenario is, every 5 minutes a partition will be added;
>>>>> What I
>>>>> do now is, get the ideal state, +1 for the partition number and then
>>>>> update
>>>>> it with ZKHelixAdmin. With a rebalance action, the new partition will
>>>>> be
>>>>> created and assigned to an instance. What the instance can get from
>>>>> Helix
>>>>> is the resource name and partition id. To map the parition to my
>>>>> logical
>>>>> data, I maintained a mapping table in our datastore, which looks like:
>>>>>
>>>>> {
>>>>>   "resource_0": 201601010000_ 201601010005,
>>>>>   "resource_1": 201601010005_ 201601010010,
>>>>>   ...
>>>>>   "resource_11": 201601010055_ 201601010100
>>>>> }
>>>>>
>>>>> Here it has 12 partitions; Now I want to discard some old partitions,
>>>>> say
>>>>> the first 6 partitions;  It seems in Helix the partitions must start
>>>>> from
>>>>> 0, so with an update on the IdealState, set # of partitions to 6, the
>>>>> new
>>>>> partitions on the cluster would become to:
>>>>>
>>>>> resource_0,resource_1, ..., resource_5,
>>>>>
>>>>> To make sure the partitions wouldn't be wrongly mapped, I need update
>>>>> my
>>>>> mapping table before the rebalance. While that may not ensure the
>>>>> atomic
>>>>> between the two updates.
>>>>>
>>>>> So my question is, what's the suggested way to do the resource
>>>>> partition
>>>>> mapping? does Helix allow user to specify additional information on a
>>>>> partition (if have this, I don't need maintain the mapping outside)?
>>>>> Can we
>>>>> have some simple APIs like addPartition(String parititionName),
>>>>> dropParitition(String partitionName), just like that for resource? The
>>>>> numeric paritition id can be an internal ID and not exposed to user.
>>>>>
>>>>> I guess many entry users will have such questions. Just raise this for
>>>>> a
>>>>> broad discussion; Any comment and suggestion is welcomed. Thanks for
>>>>> your
>>>>> time!
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>


-- 
Best regards,

Shaofeng Shi

Mime
View raw message