Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D374717C9B for ; Mon, 23 Feb 2015 22:43:24 +0000 (UTC) Received: (qmail 26528 invoked by uid 500); 23 Feb 2015 22:43:24 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 26486 invoked by uid 500); 23 Feb 2015 22:43:24 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 26476 invoked by uid 99); 23 Feb 2015 22:43:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Feb 2015 22:43:24 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,LOTS_OF_MONEY,RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of prvs=4892b2b66=zzhang@linkedin.com designates 69.28.149.80 as permitted sender) Received: from [69.28.149.80] (HELO esv4-mav04.corp.linkedin.com) (69.28.149.80) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Feb 2015 22:42:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linkedin.com; i=@linkedin.com; q=dns/txt; s=proddkim1024; t=1424731397; x=1456267397; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=YERCH8ijyPmPyINMqt6xCTCQZgGBMB7f3HIrVLyAxqc=; b=cIw4cfPfdsKhQ7qHXWp0QYxYLAlnsEY3Q6Y+B95mAz+B0J2uRZUSSRBB nK+GFjAEl5zlu/iuNBUS5aPvovLZn45EeSEy/NG/PqXTWWjosbpOBg4sx Ww6GoJpQQNsQ/gGrD7iDB36APx0E2Bf0UwH/mL3DlIXRzA9hxTqm15c92 0=; X-IronPort-AV: E=Sophos;i="5.09,633,1418112000"; d="scan'208,217";a="180053857" Received: from ESV4-MB02.linkedin.biz ([fe80::8093:3d15:3c8e:a479]) by esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.03.0195.001; Mon, 23 Feb 2015 14:42:56 -0800 From: Zhen Zhang To: "user@helix.apache.org" Subject: RE: Bucketized resources not working in Helix 0.6.4 Thread-Topic: Bucketized resources not working in Helix 0.6.4 Thread-Index: AQHQT06LzCUTCsnZVUyb8KVF09rGE5z+ihOSgACISQD//3xNuIAAiHuA//96SfiAAJl4gIAAACAA//96drgAET+GAP//lKgGgACWFoD//3rVHQ== Date: Mon, 23 Feb 2015 22:42:56 +0000 Message-ID: <23CA11DC8830BA44A37C6B44B14D013CB9D1816C@ESV4-MB02.linkedin.biz> References: <23CA11DC8830BA44A37C6B44B14D013CB9D17BFD@ESV4-MB02.linkedin.biz> <23CA11DC8830BA44A37C6B44B14D013CB9D17C5C@ESV4-MB02.linkedin.biz> <23CA11DC8830BA44A37C6B44B14D013CB9D17C77@ESV4-MB02.linkedin.biz> <23CA11DC8830BA44A37C6B44B14D013CB9D17F4B@ESV4-MB02.linkedin.biz> <23CA11DC8830BA44A37C6B44B14D013CB9D1808D@ESV4-MB02.linkedin.biz>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.18.46.251] Content-Type: multipart/alternative; boundary="_000_23CA11DC8830BA44A37C6B44B14D013CB9D1816CESV4MB02linkedi_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_23CA11DC8830BA44A37C6B44B14D013CB9D1816CESV4MB02linkedi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable bucktSize is the size of each bucket, i.e. the number of partitions each bu= cket will hold. bucket size can't be changed dynamically. ________________________________ From: Varun Sharma [varun@pinterest.com] Sent: Monday, February 23, 2015 2:37 PM To: user@helix.apache.org Subject: Re: Bucketized resources not working in Helix 0.6.4 OK - thanks. Btw, is bucket size the number of buckets or is it the size of= one bucket ? Also can the bucket size be changed dynamically using setReso= urceIdealState ? On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang > wrote: Hi Varun, The partition number is mainly for calculating the buckets. Partition name = will remain the same. You can always get and build Helix source code from h= ere: https://github.com/apache/helix Changing the partition naming is not hard, but we need to come up with a ge= neral way to achieve this. We will think about this requirement. If you hav= e any good idea for this, you can always submit a pull request on github. Thanks, Jason ________________________________ From: Varun Sharma [varun@pinterest.com] Sent: Monday, February 23, 2015 12:04 PM To: user@helix.apache.org Subject: Re: Bucketized resources not working in Helix 0.6.4 How hard is this to change to '$' - its difficult for us to now change the = partition names now. Is there a place to get the helix 0.6.4 source code, m= ake changes and built it ? Another related question - is the partition number mainly needed for hashin= g into the buckets - the partition name would still continue to be "resourc= e_name_partition_num", right ? On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang > wrote: The bucketize logic is expecting partition name following the convention {r= esource_name}_{partition_number}. In your case, you are using $ ? ________________________________ From: Varun Sharma [varun@pinterest.com] Sent: Monday, February 23, 2015 11:48 AM To: user@helix.apache.org Subject: Re: Bucketized resources not working in Helix 0.6.4 Note that we had to prefix each partition with the resource name to make th= e partitions unique. On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma > wrote: Here is the latest error I am getting: 2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceImpl.= java:129) WARN Resource creation failed for $terrapin$data$pinpin_board_jo= in$1424377884722, rolling back. java.lang.IllegalArgumentException: Could NOT parse partition# (join$142437= 7884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0 at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNRecordBucketizer.jav= a:72) at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecordBucketizer.java:10= 5) at org.apache.helix.manager.zk.ZKHelixDataAccessor.setProperty(ZKHelixDataA= ccessor.java:110) at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdealState(ZKHelixAd= min.java:686) On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang > wrote: Not really. The bucketize logic just creates child znode under the original= ideal state, and each of the child znode holds a subset of the total parti= tions. ________________________________ From: Varun Sharma [varun@pinterest.com] Sent: Monday, February 23, 2015 10:37 AM To: user@helix.apache.org Subject: Re: Bucketized resources not working in Helix 0.6.4 One other question - would bucketizing add anything to the partition name -= just worried about any special symbols (we use _ and $) causing issues the= re ? On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang > wrote: Hi Varun, HelixAdmin#addResource was initially used for adding an empty resource only= , followed by HelixAdmin#rebalance, which bucketize the ideal state, so the= bucketize logic is missing from addResource. We should definitely fix this= . We are also considering merging addResource and rebalance, since separati= ng them are confusing. For now you may use setResourceIdealState as a worka= round. Thanks, Zhen ________________________________ From: Varun Sharma [varun@pinterest.com] Sent: Monday, February 23, 2015 10:20 AM To: user@helix.apache.org Subject: Re: Bucketized resources not working in Helix 0.6.4 Thanks, Zhen. What about just calling addResource with an idealState with the bucket size= - is that expected to not work on its own ? Varun On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang > wrote: Hi Varun, Did you do the following? HelixAdmin#addResource() // set bucket size HelixAdmin#setResourceIdealState() // no bucket size When you set the ideal state, you still need set the bucket size in the ide= al state. This value will be used to bucketize the ideal state. Thanks, Jason ________________________________________ From: Varun Sharma [varun@pinterest.com] Sent: Monday, February 23, 2015 1:51 AM To: user@helix.apache.org Subject: Bucketized resources not working in Helix 0.6.4 Hi, I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to = do so. I am setting the bucketSize in the idealState and using addResource(= ) and setResourceIdealState and it ends up complaining that ZK size of 1M h= as been exceeded. I also tried using the addResource() api which creates an empty ideal state= with a valid bucket size. However, when I tried to set the ideal state on = that bucket (this ideal state did not have the bucket size set), it failed = as well. Whats the right way to get bucketizing to work ? Thanks ! Varun --_000_23CA11DC8830BA44A37C6B44B14D013CB9D1816CESV4MB02linkedi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
bucktSize is the size of each bucket, i.e. the number of partitions = each bucket will hold. bucket size can't be changed dynamically.

From: Varun Sharma [varun@pinterest.com]<= br> Sent: Monday, February 23, 2015 2:37 PM
To: user@helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4

OK - thanks. Btw, is bucket size the number of buckets or = is it the size of one bucket ? Also can the bucket size be changed dynamica= lly using setResourceIdealState ?

On Mon, Feb 23, 2015 at 1:53 PM, Zhen Zhang <zzhang@linkedi= n.com> wrote:
Hi Varun, 

The partition number is mainly for calculating the buckets. Partition = name will remain the same. You can always get and build Helix source code f= rom here:

Changing the partition naming is not hard, but we need to come up with= a general way to achieve this. We will think about this requirement. If yo= u have any good idea for this, you can always submit a pull request on gith= ub.

Thanks,
Jason
Fro= m: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 12:04 PM

To: user@= helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4

How hard is this to change to '$' - its difficult for us t= o now change the partition names now. Is there a place to get the helix 0.6= .4 source code, make changes and built it ?

Another related question - is the partition number mainly needed for h= ashing into the buckets - the partition name would still continue to be &qu= ot;resource_name_partition_num", right ?

On Mon, Feb 23, 2015 at 11:52 AM, Zhen Zhang <zzhang@linkedi= n.com> wrote:
The bucketize logic is expecting partition name following the conventi= on {resource_name}_{partition_number}. In your case, you are using $ ?
Fro= m: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 11:48 AM

To: user@= helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4

Note that we had to prefix each partition with the resourc= e name to make the partitions unique.

On Mon, Feb 23, 2015 at 11:48 AM, Varun Sharma <= span dir=3D"ltr"> <varun@pinteres= t.com> wrote:
Here is the latest error I am getting:

2015-02-23 19:44:48,153 [controller-pool-0] (TerrapinControllerServiceIm= pl.java:129) WARN  Resource creation failed for $terrapin$data$pinpin_= board_join$1424377884722, rolling back.

java.lang.IllegalArgumentException: Could NOT parse partition# (join$= 1424377884722$0) in $terrapin$data$pinpin_board_join$1424377884722$0

at org.apache.helix.ZNRecordBucketizer.getBucketName(ZNR= ecordBucketizer.java:72)

at org.apache.helix.ZNRecordBucketizer.bucketize(ZNRecor= dBucketizer.java:105)

at org.apache.helix.manager.zk.ZKHelixDataAccessor.setPrope= rty(ZKHelixDataAccessor.java:110)

at org.apache.helix.manager.zk.ZKHelixAdmin.setResourceIdea= lState(ZKHelixAdmin.java:686)


On Mon, Feb 23, 2015 at 10:40 AM, Zhen Zhang <zzhang@linkedi= n.com> wrote:
Not really. The bucketize logic just creates child znode under the ori= ginal ideal state, and each of the child znode holds a subset of the total = partitions. 
Fro= m: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 10:37 AM

To: user@= helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4

One other question - would bucketizing add anything to the= partition name - just worried about any special symbols (we use _ and $) c= ausing issues there ?

On Mon, Feb 23, 2015 at 10:35 AM, Zhen Zhang <zzhang@linkedi= n.com> wrote:
Hi Varun,

HelixAdmin#addResource was initially used for adding an empty resource= only, followed by HelixAdmin#rebalance, which bucketize the ideal state, s= o the bucketize logic is missing from addResource. We should definitely fix= this. We are also considering merging addResource and rebalance, since separating them are confusing. For now yo= u may use setResourceIdealState as a workaround.

Thanks,
Zhen

Fro= m: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 10:20 AM
To: user@= helix.apache.org
Subject: Re: Bucketized resources not working in Helix 0.6.4

Thanks, Zhen.

What about just calling addResource with an idealState with the bucket= size - is that expected to not work on its own ?

Varun

On Mon, Feb 23, 2015 at 10:18 AM, Zhen Zhang <zzhang@linkedi= n.com> wrote:
Hi Varun,

Did you do the following?
HelixAdmin#addResource() // set bucket size
HelixAdmin#setResourceIdealState() // no bucket size

When you set the ideal state, you still need set the bucket size in the ide= al state. This value will be used to bucketize the ideal state.

Thanks,
Jason

________________________________________
From: Varun Sharma [varun@pinterest.com]
Sent: Monday, February 23, 2015 1:51 AM
To: user@helix.a= pache.org
Subject: Bucketized resources not working in Helix 0.6.4

Hi,

I am trying to bucketize a resource on Helix 0.6.4 - however, am unable to = do so. I am setting the bucketSize in the idealState and using addResource(= ) and setResourceIdealState and it ends up complaining that ZK size of 1M h= as been exceeded.

I also tried using the addResource() api which creates an empty ideal state= with a valid bucket size. However, when I tried to set the ideal state on = that bucket (this ideal state did not have the bucket size set), it failed = as well.

Whats the right way to get bucketizing to work ?

Thanks !
Varun






--_000_23CA11DC8830BA44A37C6B44B14D013CB9D1816CESV4MB02linkedi_--