helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhen Zhang <zzh...@linkedin.com>
Subject RE: Specifying Bucket size when adding resource with IDEAL_STATE
Date Mon, 24 Nov 2014 18:30:56 GMT
Hi Varun, it's IdealState#setBatchMessageMode(true).


From: Varun Sharma [varun@pinterest.com]
Sent: Monday, November 24, 2014 9:28 AM
To: user@helix.apache.org
Subject: Re: Specifying Bucket size when adding resource with IDEAL_STATE

How can I enable the group message feature ?


On Sat, Nov 22, 2014 at 1:34 PM, kishore g <g.kishore@gmail.com<mailto:g.kishore@gmail.com>>

We should add some docs for large number of partitions.
Varun, you might also want to enable group message feature to for faster state transitions.

On Nov 22, 2014 1:28 PM, "Varun Sharma" <varun@pinterest.com<mailto:varun@pinterest.com>>
I got hold of the EXTERNALVIEW znode for 500 partitions and it worked out to 80kB. So, I am
using 1 bucket per 1000 partitions ~ 160Kb. There are a handful of cases where we need upto
4K partitions - hence I wanna use 4 buckets for that usecase.

On Sat, Nov 22, 2014 at 12:59 PM, Zhen Zhang <nehzgnahz@gmail.com<mailto:nehzgnahz@gmail.com>>
Hi Varun,

You can refer to this example:

The accessor handles bucket so HelixAdmin#setResourceIdealState should do the work also.
May I know how many partitions you have and how large is your ideal state?


On Sat, Nov 22, 2014 at 11:53 AM, kishore g <g.kishore@gmail.com<mailto:g.kishore@gmail.com>>
I dint realize that you had buckets. Let me check again

On Sat, Nov 22, 2014 at 11:52 AM, kishore g <g.kishore@gmail.com<mailto:g.kishore@gmail.com>>
public abstract void setResourceIdealState (String<http://download.oracle.com/javase/6/docs/api/index.html?java/lang/String.html>
clusterName, String<http://download.oracle.com/javase/6/docs/api/index.html?java/lang/String.html>
resourceName, IdealState<http://helix.apache.org/javadocs/0.6.3/reference/org/apache/helix/model/IdealState.html>

Set ideal state for a resource.

That should work.

On Sat, Nov 22, 2014 at 11:26 AM, Varun Sharma <varun@pinterest.com<mailto:varun@pinterest.com>>
How do I add a resource with an ideal state and buckets since my resource has extremely large
number of partitions ? I can't find an addResource call in HelixAdmin to do the same ?


On Fri, Nov 21, 2014 at 8:07 PM, Kanak Biscuitwala <kanak.b@hotmail.com<mailto:kanak.b@hotmail.com>>
Hi Varun,

Adding a resource is equivalent to adding an ideal state. So if you just create your own initial
ideal state and set it, you've effectively created a resource with the properties that you


> Date: Fri, 21 Nov 2014 17:08:36 -0800
> Subject: Re: Specifying Bucket size when adding resource with IDEAL_STATE
> From: varun@pinterest.com<mailto:varun@pinterest.com>
> To: user@helix.apache.org<mailto:user@helix.apache.org>
> Or should I break this up into 2 calls - one to add the resource and
> then another to set the ideal states ?
> On Fri, Nov 21, 2014 at 5:03 PM, Varun Sharma
> <varun@pinterest.com<mailto:varun@pinterest.com><mailto:varun@pinterest.com<mailto:varun@pinterest.com>>>
> Looking at the HelixAdmin API - I do not see an API for adding a
> resource with a particular IdealState and the number of buckets to use
> (znodes).
> Thanks
> Varun

View raw message