helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Sharma <va...@pinterest.com>
Subject Re: Bucketized resources not working in Helix 0.6.4
Date Mon, 23 Feb 2015 20:04:52 GMT
How hard is this to change to '$' - its difficult for us to now change the
partition names now. Is there a place to get the helix 0.6.4 source code,
make changes and built it ?

Another related question - is the partition number mainly needed for
hashing into the buckets - the partition name would still continue to be
"resource_name_partition_num", right ?

On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zzhang@linkedin.com> wrote:

>  The bucketize logic is expecting partition name following the convention
> {resource_name}_{partition_number}. In your case, you are using $ ?
>  ------------------------------
> *From:* Varun Sharma [varun@pinterest.com]
> *Sent:* Monday, February 23, 2015 11:48 AM
>
> *To:* user@helix.apache.org
> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>
>   Note that we had to prefix each partition with the resource name to
> make the partitions unique.
>
> On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <varun@pinterest.com>
> wrote:
>
>> Here is the latest error I am getting:
>>
>>  2015-02-23 19:44:48,153 [controller-pool-0]
>> (TerrapinControllerServiceImpl.java:129) WARN  Resource creation failed for
>> $terrapin$data$pinpin_board_join$1424377884722, rolling back.
>>
>> *java.lang.IllegalArgumentException: Could NOT parse partition#
>> (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0*
>>
>> *at
>> org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)*
>>
>> *at
>> org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)*
>>
>> at
>> org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)
>>
>> at
>> org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)
>>
>> On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zzhang@linkedin.com> wrote:
>>
>>>  Not really. The bucketize logic just creates child znode under the
>>> original ideal state, and each of the child znode holds a subset of the
>>> total partitions.
>>>  ------------------------------
>>> *From:* Varun Sharma [varun@pinterest.com]
>>> *Sent:* Monday, February 23, 2015 10:37 AM
>>>
>>> *To:* user@helix.apache.org
>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>
>>>    One other question - would bucketizing add anything to the partition
>>> name - just worried about any special symbols (we use _ and $) causing
>>> issues there ?
>>>
>>> On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zzhang@linkedin.com>
>>> wrote:
>>>
>>>>  Hi Varun,
>>>>
>>>>  HelixAdmin#addResource was initially used for adding an empty
>>>> resource only, followed by HelixAdmin#rebalance, which bucketize the ideal
>>>> state, so the bucketize logic is missing from addResource. We should
>>>> definitely fix this. We are also considering merging addResource and
>>>> rebalance, since separating them are confusing. For now you may use
>>>> setResourceIdealState as a workaround.
>>>>
>>>>  Thanks,
>>>> Zhen
>>>>
>>>>  ------------------------------
>>>> *From:* Varun Sharma [varun@pinterest.com]
>>>> *Sent:* Monday, February 23, 2015 10:20 AM
>>>> *To:* user@helix.apache.org
>>>> *Subject:* Re: Bucketized resources not working in Helix 0.6.4
>>>>
>>>>    Thanks, Zhen.
>>>>
>>>>  What about just calling addResource with an idealState with the
>>>> bucket size - is that expected to not work on its own ?
>>>>
>>>>  Varun
>>>>
>>>> On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zzhang@linkedin.com>
>>>> wrote:
>>>>
>>>>> Hi Varun,
>>>>>
>>>>> Did you do the following?
>>>>> HelixAdmin#addResource() // set bucket size
>>>>> HelixAdmin#setResourceIdealState() // no bucket size
>>>>>
>>>>> When you set the ideal state, you still need set the bucket size in
>>>>> the ideal state. This value will be used to bucketize the ideal state.
>>>>>
>>>>> Thanks,
>>>>> Jason
>>>>>
>>>>> ________________________________________
>>>>> From: Varun Sharma [varun@pinterest.com]
>>>>> Sent: Monday, February 23, 2015 1:51 AM
>>>>> To: user@helix.apache.org
>>>>> Subject: Bucketized resources not working in Helix 0.6.4
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to bucketize a resource on Helix 0.6.4 - however, am
>>>>> unable to do so. I am setting the bucketSize in the idealState and using
>>>>> addResource() and setResourceIdealState and it ends up complaining that
ZK
>>>>> size of 1M has been exceeded.
>>>>>
>>>>> I also tried using the addResource() api which creates an empty ideal
>>>>> state with a valid bucket size. However, when I tried to set the ideal
>>>>> state on that bucket (this ideal state did not have the bucket size set),
>>>>> it failed as well.
>>>>>
>>>>> Whats the right way to get bucketizing to work ?
>>>>>
>>>>> Thanks !
>>>>> Varun
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message