helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhen Zhang <zzh...@linkedin.com>
Subject RE: Bucketized resources not working in Helix 0.6.4
Date Mon, 23 Feb 2015 22:42:56 GMT
bucktSize is the size of each bucket, i.e. the number of partitions each bucket will hold.
bucket size can't be changed dynamically.

________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 2:37 PM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4

OK - thanks. Btw, is bucket size the number of buckets or is it the size of one bucket ? Also
can the bucket size be changed dynamically using setResourceIdealState ?

On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zzhang@linkedin.com<mailto:zzhang@linkedin.com>>
wrote:
Hi Varun,

The partition number is mainly for calculating the buckets. Partition name will remain the
same. You can always get and build Helix source code from here:
https://github.com/apache/helix

Changing the partition naming is not hard, but we need to come up with a general way to achieve
this. We will think about this requirement. If you have any good idea for this, you can always
submit a pull request on github.

Thanks,
Jason
________________________________
From: Varun Sharma [varun@pinterest.com<mailto:varun@pinterest.com>]
Sent: Monday, February 23, 2015 12:04 PM

To: user@helix.apache.org<mailto:user@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4

How hard is this to change to '$' - its difficult for us to now change the partition names
now. Is there a place to get the helix 0.6.4 source code, make changes and built it ?

Another related question - is the partition number mainly needed for hashing into the buckets
- the partition name would still continue to be "resource_name_partition_num", right ?

On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zzhang@linkedin.com<mailto:zzhang@linkedin.com>>
wrote:
The bucketize logic is expecting partition name following the convention {resource_name}_{partition_number}.
In your case, you are using $ ?
________________________________
From: Varun Sharma [varun@pinterest.com<mailto:varun@pinterest.com>]
Sent: Monday, February 23, 2015 11:48 AM

To: user@helix.apache.org<mailto:user@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4

Note that we had to prefix each partition with the resource name to make the partitions unique.

On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <varun@pinterest.com<mailto:varun@pinterest.com>>
wrote:
Here is the latest error I am getting:


2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.java:129) WARN
 Resource creation failed for $terrapin$data$pinpin_board_join$1424377884722, rolling back.

java.lang.IllegalArgumentException: Could NOT parse partition# (join$1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0

at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.java:72)

at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:105)

at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataAccessor.java:110)

at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAdmin.java:686)

On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zzhang@linkedin.com<mailto:zzhang@linkedin.com>>
wrote:
Not really. The bucketize logic just creates child znode under the original ideal state, and
each of the child znode holds a subset of the total partitions.
________________________________
From: Varun Sharma [varun@pinterest.com<mailto:varun@pinterest.com>]
Sent: Monday, February 23, 2015 10:37 AM

To: user@helix.apache.org<mailto:user@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4

One other question - would bucketizing add anything to the partition name - just worried about
any special symbols (we use _ and $) causing issues there ?

On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zzhang@linkedin.com<mailto:zzhang@linkedin.com>>
wrote:
Hi Varun,

HelixAdmin#addResource was initially used for adding an empty resource only, followed by HelixAdmin#rebalance,
which bucketize the ideal state, so the bucketize logic is missing from addResource. We should
definitely fix this. We are also considering merging addResource and rebalance, since separating
them are confusing. For now you may use setResourceIdealState as a workaround.

Thanks,
Zhen

________________________________
From: Varun Sharma [varun@pinterest.com<mailto:varun@pinterest.com>]
Sent: Monday, February 23, 2015 10:20 AM
To: user@helix.apache.org<mailto:user@helix.apache.org>
Subject: Re: Bucketized resources not working in Helix 0.6.4

Thanks, Zhen.

What about just calling addResource with an idealState with the bucket size - is that expected
to not work on its own ?

Varun

On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zzhang@linkedin.com<mailto:zzhang@linkedin.com>>
wrote:
Hi Varun,

Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size

When you set the ideal state, you still need set the bucket size in the ideal state. This
value will be used to bucketize the ideal state.

Thanks,
Jason

________________________________________
From: Varun Sharma [varun@pinterest.com<mailto:varun@pinterest.com>]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.apache.org<mailto:user@helix.apache.org>
Subject: Bucketized resources not working in Helix 0.6.4

Hi,

I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to do so. I am setting
the bucketSize in the idealState and using addResource() and setResourceIdealState and it
ends up complaining that ZK size of 1M has been exceeded.

I also tried using the addResource() api which creates an empty ideal state with a valid bucket
size. However, when I tried to set the ideal state on that bucket (this ideal state did not
have the bucket size set), it failed as well.

Whats the right way to get bucketizing to work ?

Thanks !
Varun







Mime
View raw message