Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3260F114AB for ; Wed, 18 Jun 2014 20:06:46 +0000 (UTC) Received: (qmail 11168 invoked by uid 500); 18 Jun 2014 20:06:45 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 11118 invoked by uid 500); 18 Jun 2014 20:06:45 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 11106 invoked by uid 99); 18 Jun 2014 20:06:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jun 2014 20:06:45 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kanak.b@hotmail.com designates 65.54.190.18 as permitted sender) Received: from [65.54.190.18] (HELO BAY004-OMC1S7.hotmail.com) (65.54.190.18) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jun 2014 20:06:41 +0000 Received: from BAY182-W81 ([65.54.190.61]) by BAY004-OMC1S7.hotmail.com with Microsoft SMTPSVC(7.5.7601.22712); Wed, 18 Jun 2014 13:06:16 -0700 X-TMN: [SMNrD38SAxSEA34/XTi6o1+o/geA4OjO] X-Originating-Email: [kanak.b@hotmail.com] Message-ID: From: Kanak Biscuitwala To: "user@helix.apache.org" , Jianqiu Lu CC: Gavin Li Subject: RE: question on a state change scenario Date: Wed, 18 Jun 2014 13:06:16 -0700 Importance: Normal In-Reply-To: References: ,,,, Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 18 Jun 2014 20:06:16.0878 (UTC) FILETIME=[C353E4E0:01CF8B30] X-Virus-Checked: Checked by ClamAV on apache.org The requested state feature is currently only supported in the task framewo= rk and otherwise ignored. Currently the only way for a participant to chang= e its own state is to update the ideal state itself.=0A= =0A= Kanak=0A= =0A= ________________________________=0A= > Date: Wed=2C 18 Jun 2014 13:02:43 -0700 =0A= > Subject: Re: question on a state change scenario =0A= > From: g.kishore@gmail.com =0A= > To: lvjq@yahoo-inc.com =0A= > CC: user@helix.apache.org=3B liyu@yahoo-inc.com =0A= > =0A= > Hi Jianqui=2C =0A= > =0A= > For updating the idealstate=2C yes race conditions are possible. There is= =0A= > a low level api that allows you to handle the race conditions. =0A= > =0A= > see HelixDataAccessor.update method: This method does a merge of =0A= > existing and new values. For example=2C for a participant to go into =0A= > read_only state for a partition. It can do =0A= > acce =0A= > is=3D new IdealState()=3B =0A= > is.set("participant_id"=2C "partition_id"=2C "read_only")=3B =0A= > helixdataaccessor.update(accessor.keyBuilder.idealState("resource_name")= =2Cis)=3B =0A= > =0A= > This should handle the race conditions. Jason/Kanak=2C please correct me = =0A= > if I am wrong. =0A= > =0A= > Which version of Helix are you using. I think documentation for #2 is =0A= > weak and its something used internally by Helix for Task Management. =0A= > Kanak=2C do we have any example for this ? =0A= > =0A= > thanks=2C =0A= > Kishore G =0A= > =0A= > =0A= > =0A= > =0A= > On Wed=2C Jun 18=2C 2014 at 11:30 AM=2C Jianqiu Lu =0A= > > wrote: =0A= > Hi Kishore=2CKanak=2C =0A= > Thanks for your reply. In my case it is a participant-level property=2C = =0A= > an instance can have different state for different partitions. Ex. =0A= > Partition 1-3 are on instance 1=2C and partition 1 is read =2C partition = 2 =0A= > is weak read =2Cpartition 3 is read. It does not have a strong constrain = =0A= > in my case=2C but updates ideal state may have some race conditions. From= =0A= > my understand=2C I need to pass the whole ideal state(all partitions) as = =0A= > a map to the method=2C but what if two servers update their ideal state = =0A= > at the same time? By the way how does #2 work? Is there any example? =0A= > For failure handling we have different replica consumes the same =0A= > queue=2C and register both of the two servers in different replica to =0A= > same partition=2C so unless both of them are down=2C we can always have = =0A= > some one reachable. =0A= > =0A= > Thanks =0A= > Jianqiu =0A= > =0A= > From: kishore g > =0A= > Date: Tuesday=2C June 17=2C 2014 at 10:22 PM =0A= > To: "user@helix.apache.org" =0A= > >=2C Jianqiu Lv =0A= > > =0A= > Cc: Gavin Li > =0A= > Subject: Re: question on a state change scenario =0A= > =0A= > Hi Jianqiu=2C =0A= > =0A= > Thank you for the kind words. I agree with what Kanak said but it =0A= > mostly applies to systems where state level constraint is important. =0A= > For example=2C if the state model is master-slave=2C we don't want the = =0A= > slave to automatically say that its the new master. =0A= > =0A= > Looks like in your systems=2C there are no strong constraints on the =0A= > state. In this case it is ok for the participant to control its state. = =0A= > You have two options =0A= > =0A= > #1. The participant can decide the state and update the idealstate to =0A= > reflect that state. This will be reflected in the ExternalView and the = =0A= > spectators can use the state to route requests. You can do this if you = =0A= > don't intend to have any hard constraints on the state. =0A= > =0A= > #2. Another option is to use the feature of requested state=2C the =0A= > participant can request the controller to go to a specific state. The =0A= > controller however can decide if it can honor that request. =0A= > =0A= > For the application you have described #1 can work. Let us know if #1 =0A= > is acceptable. =0A= > =0A= > Actually=2C I can see this as a requirement as a common pattern. We can = =0A= > allow the participants to override the state in certain scenarios. We =0A= > should be able to provide a much simpler api to accomplish this. =0A= > =0A= > Also what is your requirement for failure handling. =0A= > =0A= > thanks=2C =0A= > Kishore G =0A= > =0A= > =0A= > =0A= > On Tue=2C Jun 17=2C 2014 at 8:46 PM=2C Kanak Biscuitwala =0A= > > wrote: =0A= > Hi Jianqiu=2C =0A= > =0A= > In Helix we typically discourage participants from changing the states = =0A= > of partitions they serve because then we have multiple brains in the =0A= > system. =0A= > =0A= > It seems like the read property is a participant-level property rather = =0A= > than a partition-level state. Is this true? Can an instance have =0A= > different speeds per partition on the same instance? If it is purely =0A= > participant-level=2C then you can use HelixAdmin#setConfig to set this = =0A= > flag for the participant=2C and you can register an =0A= > InstanceConfigChangeListener with HelixManager in order to listen for =0A= > changes. =0A= > =0A= > It is certainly possible to use HelixDataAccessor#updateIdealState to =0A= > only update the partitions that currently belong to an instance=2C but = =0A= > that feels more like a hack=2C and it requires you to fully customize the= =0A= > ideal state calculation logic=2C rather than using FULL_AUTO or SEMI_AUTO= =0A= > mode. =0A= > =0A= > Kanak =0A= > =0A= > ________________________________ =0A= >> From: lvjq@yahoo-inc.com =0A= >> To: user@helix.apache.org =0A= >> CC: liyu@yahoo-inc.com =0A= >> Subject: question on a state change scenario =0A= >> Date: Wed=2C 18 Jun 2014 02:05:00 +0000 =0A= >> =0A= >> Hi team=2C =0A= >> First I want to thanks all for this great software. =0A= >> I am using helix for our storage system. In our system we have 3 =0A= >> states read =2C weak read and offline. The state transactions are =0A= >> read-> <- weak read-> <-offline=2C very similar to the master/slave =0A= >> example. But the difference is that we need participate to control its = =0A= >> status =2C which means a participate can decide what is itself's current= =0A= >> ideal state. If a participate find he can=92t follow write speed(we use = a =0A= >> queue to do write =2C participate consumes message from this queue)=2C t= hen =0A= >> it can change its state from read to weak read. Is there a proper way = =0A= >> to do this state change from participate side? I looked into Helix java = =0A= >> doc and found in HelixAdmin there is a method =93rebalance=94 seems can = =0A= >> achieve my goal=2C but my scenario is only need to change one server=92s= =0A= >> state=2C=93rebalance=94 seems too over. Or does this a Helix fit scenari= o? =0A= >> The other solution I thought about is using zookeeper to manage read =0A= >> and weak read states=2C and helix only maintain the online offline state= . =0A= >> Once I get an online server from helix =2C then I will look up into =0A= >> zookeeper to find out what is the current read state of this =0A= >> server(maybe push mode from zookeeper is more proper). In this =0A= >> solution we don=92t need to change helix ideal state from participate = =0A= >> side=2C but this seems like mix helix and zookeeper to maintain state = =0A= >> machine together. Could you please give me some advise on my thoughts? = =0A= >> =0A= >> Thanks =0A= >> Jianqiu =0A= > =0A= > =0A= > =0A= =