helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Sharma <va...@pinterest.com>
Subject Re: Bucketized resources not working in Helix 0.6.4
Date Mon, 23 Feb 2015 23:10:15 GMT
Also, would changing the bucketSize throw an exception or would it cause
undefined behaviour or would the new bucket size simply be ignored ? This
is assuming the number of partitions remain the same in the
setResourceIdealState call ?

Thanks !
Varun

On Mon, Feb 23, 2015 at 2:45 PM, Varun Sharma <varun@pinterest.com> wrote:

> Cool - thanks - for a pre existing resource, how can we retrieve the
> bucket size - do we need to pull in the whole ideal state and then run a
> getBucketSize() on it ?
>
> On Mon, Feb 23, 2015 at 2:42 PM, Zhen Zhang <zzhang@linkedin.com> wrote:
>
>>  bucktSize is the size of each bucket, i.e. the number of partitions
>> each bucket will hold. bucket size can't be changed dynamically.
>>
>>  ------------------------------
>> *From:* Varun Sharma [varun@pinterest.com]
>> *Sent:* Monday, February 23, 2015 2:37 PM
>>
>> *To:* user@helix.apache.org
>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>
>>   OK - thanks. Btw, is bucket size the number of buckets or is it the
>> size of one bucket ? Also can the bucket size be changed dynamically using
>> setResourceIdealState ?
>>
>> On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zzhang@linkedin.com> wrote:
>>
>>>  Hi Varun,
>>>
>>>  The partition number is mainly for calculating the buckets. Partition
>>> name will remain the same. You can always get and build Helix source code
>>> from here:
>>> https://github.com/apache/helix
>>>
>>>  Changing the partition naming is not hard, but we need to come up with
>>> a general way to achieve this. We will think about this requirement. If you
>>> have any good idea for this, you can always submit a pull request on github.
>>>
>>>  Thanks,
>>> Jason
>>>  ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 12:04 PM
>>>
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>>    How hard is this to change to '$' - its difficult for us to now
>>> change the partition names now. Is there a place to get the helix 0.6.4
>>> source code, make changes and built it ?
>>>
>>>  Another related question - is the partition number mainly needed for
>>> hashing into the buckets - the partition name would still continue to be
>>> "resource_name_partition_num", right ?
>>>
>>> On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zzhang@linkedin.com>
>>> wrote:
>>>
>>>>  The bucketize logic is expecting partition name following the
>>>> convention {resource_name}_{partition_number}. In your case, you are using
>>>> $ ?
>>>>  ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 11:48 AM
>>>>
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>>    Note that we had to prefix each partition with the resource name to
>>>> make the partitions unique.
>>>>
>>>> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <varun@pinterest.com>
>>>> wrote:
>>>>
>>>>> Here is the latest error I am getting:
>>>>>
>>>>>  2015-02-23 19:44:48,153 [controller-pool-0]
>>>>> (TerrapinControllerServiceImpl.java:129) WARN  Resource creation failed
for
>>>>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>>>>
>>>>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>>>>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>>>>
>>>>> *at
>>>>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>>>>
>>>>> *at
>>>>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>>>>
>>>>> at
>>>>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>>>>
>>>>> at
>>>>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>>>>
>>>>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zzhang@linkedin.com>
>>>>> wrote:
>>>>>
>>>>>>  Not really. The bucketize logic just creates child znode under the
>>>>>> original ideal state, and each of the child znode holds a subset
of the
>>>>>> total partitions.
>>>>>>  ------------------------------
>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>>>>
>>>>>> *To:* user@helix.apache.org
>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>
>>>>>>    One other question - would bucketizing add anything to the
>>>>>> partition name - just worried about any special symbols (we use _
and $)
>>>>>> causing issues there ?
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zzhang@linkedin.com>
>>>>>> wrote:
>>>>>>
>>>>>>>  Hi Varun,
>>>>>>>
>>>>>>>  HelixAdmin#addResource was initially used for adding an empty
>>>>>>> resource only, followed by HelixAdmin#rebalance, which bucketize
the ideal
>>>>>>> state, so the bucketize logic is missing from addResource. We
should
>>>>>>> definitely fix this. We are also considering merging addResource
and
>>>>>>> rebalance, since separating them are confusing. For now you may
use
>>>>>>> setResourceIdealState as a workaround.
>>>>>>>
>>>>>>>  Thanks,
>>>>>>> Zhen
>>>>>>>
>>>>>>>  ------------------------------
>>>>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>>>>> *To:* user@helix.apache.org
>>>>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>>>>
>>>>>>>    Thanks, Zhen.
>>>>>>>
>>>>>>>  What about just calling addResource with an idealState with
the
>>>>>>> bucket size - is that expected to not work on its own ?
>>>>>>>
>>>>>>>  Varun
>>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zzhang@linkedin.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Varun,
>>>>>>>>
>>>>>>>> Did you do the following?
>>>>>>>> HelixAdmin#addResource() // set bucket size
>>>>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>>>>
>>>>>>>> When you set the ideal state, you still need set the bucket
size in
>>>>>>>> the ideal state. This value will be used to bucketize the
ideal state.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jason
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>>>>> To: user@helix.apache.org
>>>>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however,
am
>>>>>>>> unable to do so. I am setting the bucketSize in the idealState
and using
>>>>>>>> addResource() and setResourceIdealState and it ends up complaining
that ZK
>>>>>>>> size of 1M has been exceeded.
>>>>>>>>
>>>>>>>> I also tried using the addResource() api which creates an
empty
>>>>>>>> ideal state with a valid bucket size. However, when I tried
to set the
>>>>>>>> ideal state on that bucket (this ideal state did not have
the bucket size
>>>>>>>> set), it failed as well.
>>>>>>>>
>>>>>>>> Whats the right way to get bucketizing to work ?
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>> Varun
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message