helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Sharma <va...@pinterest.com>
Subject Re: Bucketized resources not working in Helix 0.6.4
Date Tue, 24 Feb 2015 18:30:56 GMT
I see and is it okay to set the bucket size for a resource which previously
did not have buckets - would that also lead to bad behaviour (using
setResourceIdealState)

Thanks
Varun

On Tue, Feb 24, 2015 at 10:27 AM, Zhen Zhang <zzhang@linkedin.com> wrote:

>  drop ideal state means drop the resource.
>
>  yes. if you set a different bucket size, the old buckets will remains.
> since the bucket size is changed, helix controller will not be able to read
> the correct current state of each partition and considers them all be in
> initial state. This will lead to controller sending incorrect transitions
> to existing partitions.
>
>  ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Tuesday, February 24, 2015 12:37 AM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
>   Do I need to drop the resource in order to drop the ideal state - also
> wondering what the behaviour is if i set the bucket size using
> setResourceIdealState - would it just keep the previous bucketing or bad
> things could happen ?
>
> On Tue, Feb 24, 2015 at 12:20 AM, Zhen Zhang <nehzgnahz@gmail.com> wrote:
>
>> To get bucket size, get the ideal state and then getBucketSize() on it.
>> If you want to change the bucket size, you may first drop the ideal state
>> and re-create using the new bucket size.
>>
>> On Mon, Feb 23, 2015 at 3:10 PM, Varun Sharma <varun@pinterest.com>
>> wrote:
>>
>>> Also, would changing the bucketSize throw an exception or would it cause
>>> undefined behaviour or would the new bucket size simply be ignored ? This
>>> is assuming the number of partitions remain the same in the
>>> setResourceIdealState call ?
>>>
>>>  Thanks !
>>>  Varun
>>>
>>> On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <varun@pinterest.com>
>>> wrote:
>>>
>>>> Cool - thanks - for a pre existing resource, how can we retrieve the
>>>> bucket size - do we need to pull in the whole ideal state and then run a
>>>> getBucketSize() on it ?
>>>>
>>>> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zzhang@linkedin.com>
>>>> wrote:
>>>>
>>>>>  bucktSize is the size of each bucket, i.e. the number of partitions
>>>>> each bucket will hold. bucket size can't be changed dynamically.
>>>>>
>>>>>  ------------------------------
>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>> *Sent:* Monday, February 23, 2015 2:37 PM
>>>>>
>>>>> *To:* user@helix.apache.org
>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>>    OK - thanks. Btw, is bucket size the number of buckets or is it
>>>>> the size of one bucket ? Also can the bucket size be changed dynamically
>>>>> using setResourceIdealState ?
>>>>>
>>>>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zzhang@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>>  Hi Varun,
>>>>>>
>>>>>>  The partition number is mainly for calculating the buckets.
>>>>>> Partition name will remain the same. You can always get and build
Helix
>>>>>> source code from here:
>>>>>> https://github.com/apache/helix
>>>>>>
>>>>>>  Changing the partition naming is not hard, but we need to come up
>>>>>> with a general way to achieve this. We will think about this requirement.
>>>>>> If you have any good idea for this, you can always submit a pull
request on
>>>>>> github.
>>>>>>
>>>>>>  Thanks,
>>>>>> Jason
>>>>>>  ------------------------------
>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>> *Sent:* Monday, February 23, 2015 12:04 PM
>>>>>>
>>>>>> *To:* user@helix.apache.org
>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>>    How hard is this to change to '$' - its difficult for us to now
>>>>>> change the partition names now. Is there a place to get the helix
0.6.4
>>>>>> source code, make changes and built it ?
>>>>>>
>>>>>>  Another related question - is the partition number mainly needed
>>>>>> for hashing into the buckets - the partition name would still continue
to
>>>>>> be "resource_name_partition_num", right ?
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zzhang@linkedin.com>
>>>>>> wrote:
>>>>>>
>>>>>>>  The bucketize logic is expecting partition name following the
>>>>>>> convention {resource_name}_{partition_number}. In your case,
you are using
>>>>>>> $ ?
>>>>>>>  ------------------------------
>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>>>>>
>>>>>>> *To:* user@helix.apache.org
>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>
>>>>>>>    Note that we had to prefix each partition with the resource
name
>>>>>>> to make the partitions unique.
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <varun@pinterest.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here is the latest error I am getting:
>>>>>>>>
>>>>>>>>  2015-02-23 19:44:48,153 [controller-pool-0]
>>>>>>>> (TerrapinControllerServiceImpl.java:129) WARN  Resource creation
failed for
>>>>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>>>>>
>>>>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>>>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>>>>>
>>>>>>>> *at
>>>>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>>>>>
>>>>>>>> *at
>>>>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zzhang@linkedin.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>  Not really. The bucketize logic just creates child znode
under
>>>>>>>>> the original ideal state, and each of the child znode
holds a subset of the
>>>>>>>>> total partitions.
>>>>>>>>>  ------------------------------
>>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>>>>>
>>>>>>>>> *To:* user@helix.apache.org
>>>>>>>>> *Subject:* Re: Bucketized resources not working in Helix
0.6.4
>>>>>>>>>
>>>>>>>>>    One other question - would bucketizing add anything
to the
>>>>>>>>> partition name - just worried about any special symbols
(we use _ and $)
>>>>>>>>> causing issues there ?
>>>>>>>>>
>>>>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zzhang@linkedin.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>  Hi Varun,
>>>>>>>>>>
>>>>>>>>>>  HelixAdmin#addResource was initially used for adding
an empty
>>>>>>>>>> resource only, followed by HelixAdmin#rebalance,
which bucketize the ideal
>>>>>>>>>> state, so the bucketize logic is missing from addResource.
We should
>>>>>>>>>> definitely fix this. We are also considering merging
addResource and
>>>>>>>>>> rebalance, since separating them are confusing. For
now you may use
>>>>>>>>>> setResourceIdealState as a workaround.
>>>>>>>>>>
>>>>>>>>>>  Thanks,
>>>>>>>>>> Zhen
>>>>>>>>>>
>>>>>>>>>>  ------------------------------
>>>>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>>>>>> *To:* user@helix.apache.org
>>>>>>>>>> *Subject:* Re: Bucketized resources not working in
Helix 0.6.4
>>>>>>>>>>
>>>>>>>>>>    Thanks, Zhen.
>>>>>>>>>>
>>>>>>>>>>  What about just calling addResource with an idealState
with the
>>>>>>>>>> bucket size - is that expected to not work on its
own ?
>>>>>>>>>>
>>>>>>>>>>  Varun
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zzhang@linkedin.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Varun,
>>>>>>>>>>>
>>>>>>>>>>> Did you do the following?
>>>>>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket
size
>>>>>>>>>>>
>>>>>>>>>>> When you set the ideal state, you still need
set the bucket size
>>>>>>>>>>> in the ideal state. This value will be used to
bucketize the ideal state.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jason
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>>>>>> To: user@helix.apache.org
>>>>>>>>>>> Subject: Bucketized resources not working in
Helix 0.6.4
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to bucketize a resource on Helix
0.6.4 - however, am
>>>>>>>>>>> unable to do so. I am setting the bucketSize
in the idealState and using
>>>>>>>>>>> addResource() and setResourceIdealState and it
ends up complaining that ZK
>>>>>>>>>>> size of 1M has been exceeded.
>>>>>>>>>>>
>>>>>>>>>>> I also tried using the addResource() api which
creates an empty
>>>>>>>>>>> ideal state with a valid bucket size. However,
when I tried to set the
>>>>>>>>>>> ideal state on that bucket (this ideal state
did not have the bucket size
>>>>>>>>>>> set), it failed as well.
>>>>>>>>>>>
>>>>>>>>>>> Whats the right way to get bucketizing to work
?
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>> Varun
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message