Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8A75A11A04 for ; Wed, 6 Aug 2014 23:08:00 +0000 (UTC) Received: (qmail 40951 invoked by uid 500); 6 Aug 2014 23:08:00 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 40901 invoked by uid 500); 6 Aug 2014 23:08:00 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 40878 invoked by uid 99); 6 Aug 2014 23:08:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2014 23:08:00 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kanak.b@hotmail.com designates 65.54.190.16 as permitted sender) Received: from [65.54.190.16] (HELO BAY004-OMC1S5.hotmail.com) (65.54.190.16) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2014 23:07:56 +0000 Received: from BAY182-W22 ([65.54.190.60]) by BAY004-OMC1S5.hotmail.com with Microsoft SMTPSVC(7.5.7601.22712); Wed, 6 Aug 2014 16:07:30 -0700 X-TMN: [QfyJKWFQg+taTWR37qzlRjQLgt0j9gKu] X-Originating-Email: [kanak.b@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_7b4eed80-763e-4506-97ff-a0432823507b_" From: Kanak Biscuitwala To: "user@helix.apache.org" Subject: RE: Questions about custom helix rebalancer/controller/agent Date: Wed, 6 Aug 2014 16:07:29 -0700 Importance: Normal In-Reply-To: References: ,,,,,,,,,,,,,,,,,, MIME-Version: 1.0 X-OriginalArrivalTime: 06 Aug 2014 23:07:30.0362 (UTC) FILETIME=[32ABA1A0:01CFB1CB] X-Virus-Checked: Checked by ClamAV on apache.org --_7b4eed80-763e-4506-97ff-a0432823507b_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Yeah=2C it's not possible to attach to a controller because the custom code= runner essentially treats each runner as a participant for a logical resou= rce corresponding to leadership of running that code. Rather than using HelixCustomCodeRunner=2C perhaps you can just use a basic= scheduled timer task on each controller=2C and when it is triggered=2C che= ck if you're the leader=2C and if so=2C run the ideal state update code. An= other thing you can do is helixManager#addControllerListener()=2C and on th= e callback check if you're leader=2C and schedule the timer if you are=2C a= nd otherwise cancel the timer. Here's how you can check if you're leader: HelixDataAccessor accessor =3D helixManager.getHelixDataAccessor()=3BLiveIn= stance leader =3D accessor.getProperty(accessor.keyBuilder().controllerLead= er())=3Bif (leader !=3D null && leader.getId().equals(myId)) { // I am the= leader} Date: Wed=2C 6 Aug 2014 15:56:35 -0700 Subject: Re: Questions about custom helix rebalancer/controller/agent From: varun@pinterest.com To: user@helix.apache.org I am attempting to attach the HelixCustomCodeRunner to a controller instanc= e - not really running a controller alongside each of my nodes. HelixCustom= CodeRunner.start() is failing as above with a nullpointer exception at line= 120. Is it not possible to attach the HelixCustomCodeRunner to a controlle= r instance ?=0A= Thanks !Varun On Wed=2C Aug 6=2C 2014 at 3:46 PM=2C Kanak Biscuitwala wrote: =0A= =0A= =0A= =0A= I would suggest maintaining 2 HelixManager connections: one for CONTROLLER= =2C one for PARTICIPANT (I'm assuming you're running a controller instance = alongside each of your nodes). It's wasteful=2C but you should just leave t= he controller one alone=2C and then attach the state model factory and cust= om code runner to the participant one.=0A= Kanak Date: Wed=2C 6 Aug 2014 15:43:16 -0700 Subject: Re: Questions about custom helix rebalancer/controller/agent From: varun@pinterest.com =0A= To: user@helix.apache.org Without this I was getting a null pointer exception in the CustomCodeRunner= - Helix 0.6.3 - Lines 120 and 121=0A= =0A= =0A= =0A= 120=0A= =0A= StateMachineEngine stateMach =3D _manager.getStateMachineEngine()=3B=0A= =0A= =0A= 121=0A= =0A= =0A= =0A= stateMach.registerStateModelFactory(LEADER_STANDBY=2C _stateModelFty=2C= _resourceName)=3B=0A= =0A= =0A= My code calls the HelixCustomCodeRunner in the following way:=0A= =0A= new HelixCustomCodeRunner(helixManager=2C zookeeperQuorum).=0A= on(HelixConstants.ChangeType.LIVE_INSTANCE).invoke(myCallback).=0A= usingLeaderStandbyModel("HDFS_rebalancer").start()=3B =0A= =0A= On Wed=2C Aug 6=2C 2014 at 3:08 PM=2C Kanak Biscuitwala wrote: =0A= =0A= =0A= =0A= Hi Varun=2C getStateMachineEngine is only supported for InstanceType.PARTICIPANT. May I= ask why you need your controller to have state transition callbacks? In future releases=2C we're creating separate classes for each role=2C so h= opefully that will resolve confusions like this moving forward.=0A= =0A= Kanak Date: Wed=2C 6 Aug 2014 15:04:36 -0700 Subject: Re: Questions about custom helix rebalancer/controller/agent From: varun@pinterest.com =0A= =0A= To: user@helix.apache.org I am getting a weird null pointer exception while instantiating the control= ler. Here is the error: =0A= =0A= this.helixManager =3D HelixManagerFactory.getZKHelixManager(this.clusterNam= e=2C=0A= InetAddress.getLocalHost().getHostName() + ":" + thriftPort=2C = InstanceType.CONTROLLER=2C zookeeperQuorum)=3BStateMachineEngine = machineEngine =3D helixManager.getStateMachineEngine()=3B=0A= =0A= =0A= machineEngine.registerStateModelFactory("HDFS_state_machine"=2C new OnlineOfflineStateModelFactory(1000))=3Bthis.helixManager.connect(= )=3B =0A= =0A= =0A= I get a NullPointerException at line #3 because getStateMachineEngine() ret= urns a null value. Is that supposed to happen ? ThanksVarun =0A= On Fri=2C Aug 1=2C 2014 at 11:23 AM=2C Zhen Zhang wro= te: =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= Hi Varun=2C =0A= =0A= =0A= The state transitions will be independent. Helix controller may send MASTER= ->OFFLINE to all three nodes=2C for example=2C and if node1 completes the M= ASTER->OFFLINE transition first=2C controller will send OFFLINE->DROPPED to= node1 first. Or if all three nodes=0A= completes MASTER->OFFLINE at the same time=2C controller may send OFFLINE-= >DROPPED to all three nodes together.=0A= =0A= =0A= Thanks=2C=0A= Jason=0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= From: Varun Sharma =0A= Reply-To: "user@helix.apache.org" =0A= =0A= =0A= =0A= Date: Friday=2C August 1=2C 2014 11:10 AM =0A= To: "user@helix.apache.org" =0A= =0A= =0A= =0A= Subject: Re: Questions about custom helix rebalancer/controller/agent =0A= =0A= =0A= =0A= =0A= =0A= Thanks a lot. Most of my questions are answered=2C except I have one follow= up question.=0A= =0A= =0A= Lets say I have a situation with 3 masters per partition. For partition X= =2C these are on node1=2C node2 and node3. Upon dropping the resource=2C wo= uld the partition X be offlined on all three nodes and then dropped or can = that be independent as in=2C node1 offlines=0A= and drops=2C followed by node2 and so on. Just want to check if we first w= ait for all the masters to offline and then initiate the offline->drop or t= he other way round.=0A= =0A= =0A= Thanks !=0A= Varun=0A= =0A= =0A= =0A= On Fri=2C Aug 1=2C 2014 at 10:33 AM=2C Kanak Biscuitwala =0A= wrote: =0A= =0A= =0A= =0A= Dropping a resource will cause the controller to first send MASTER --> OFFL= INE for all partitions=2C and then OFFLINE --> DROPPED.=0A= =0A= =0A= Kanak =0A= =0A= =0A= Date: Fri=2C 1 Aug 2014 10:30:54 -0700=0A= =0A= =0A= Subject: Re: Questions about custom helix rebalancer/controller/agent =0A= From: varun@pinterest.com =0A= To: user@helix.apache.org =0A= =0A= In my case=2C I will have many resources - like say upto a 100 resources. E= ach of them will have partitions in the range of 100-5K. So I guess=2C I d= o require the bucket size. 300K partitions is the sum of partitions across = all resources=2C rather=0A= than the # of partitions within a single resource.=0A= =0A= =0A= Another question=2C I had was regarding removing a resource in Helix. When = a removeResource is called from HelixAdmin=2C would it trigger the MASTER->= OFFLINE the respective partitions before the resource is removing ? To conc= retize my use case=2C we have many=0A= resources with a few thousand partitions being loaded every day. New versi= ons of the resources keep getting loaded as brand new resources into Helix = and the older versions are decommissioned/garbage collected. So we would be= issuing upto a 100 or so resource=0A= additions per day and upto a 100 or so resource deletions every day. Just = want to check that deleting a resource would also trigger the appropriate M= ASTER->OFFLINE transitions. =0A= =0A= =0A= Thanks=0A= Varun=0A= =0A= =0A= =0A= On Fri=2C Aug 1=2C 2014 at 10:18 AM=2C Kanak Biscuitwala wrote: =0A= =0A= =0A= a) By default=2C there is one znode per resource=2C which as you know is a = grouping of partitions. The biggest limitation is that ZK has a 1MB limit o= n znode sizing. To get around this=2C Helix has the concept of bucketizing= =2C where in your ideal state=2C=0A= you can set a bucket size=2C which will effectively create that many znode= s to fully represent all your partitions. I believe that you can have ~2k p= artitions before you start needing to bucketize.=0A= =0A= =0A= 300k may cause you separate issues=2C and you may want to consider doing th= ings like enabling batch message mode in your ideal state so that each mess= age we send to an instance contains transitions for all partitions hosted o= n that instance=2C rather than=0A= creating a znode per partition state change. However=2C in theory (we've n= ever played with this many in practice)=2C Helix should be able to function= correctly with that many partitions.=0A= =0A= =0A= b) Yes=2C if you have a hard limit of 1 master per partition=2C Helix will = transition the first node to OFFLINE before sending the MASTER transition t= o the new master.=0A= =0A= =0A= Kanak =0A= =0A= =0A= =0A= Date: Fri=2C 1 Aug 2014 10:09:24 -0700=0A= =0A= =0A= Subject: Re: Questions about custom helix rebalancer/controller/agent =0A= From: varun@pinterest.com =0A= To: user@helix.apache.org =0A= =0A= Sounds fine to me. I can work without the FINALIZE notification for now=2C = but I hope its going to come out soon. A few more questions:=0A= =0A= =0A= a) How well does Helix scale with partitions - is each partition a separate= znode inside helix ? If I have 300K partitions in Helix would that be an i= ssue ?=0A= b) If a partition which was assigned as a master on node1 is now assigned a= s a master on node2=2C will node1 get a callback execution for transition f= rom MASTER-->OFFLINE=0A= =0A= =0A= Thanks=0A= Varun=0A= =0A= =0A= =0A= On Thu=2C Jul 31=2C 2014 at 11:18 PM=2C Kanak Biscuitwala wrote: =0A= =0A= =0A= s/run/start/g -- sorry about that=2C fixed in javadocs for future releases =0A= =0A= You may need to register for a notification type=3B I believe HelixCustomCo= deRunner complains if you don't. However=2C you can simply ignore that noti= fication type=2C and just check for INIT and FINALIZE notification types in= your callback to to track whether or=0A= not you're the leader. On INIT=2C you start your 30 minute timer=2C and on= FINALIZE you stop it. You may need to wait for us to make a 0.6.4 release = (we will likely do this soon) to get the FINALIZE notification. =0A= =0A= Here is an example of a custom code runner usage:=0A= Registration: https://github.com/kishoreg/fullmatix/blob/master/mysql-clust= er/src/main/java/org/apache/fullmatix/mysql/MySQLAgent.java=0A= =0A= =0A= =0A= Callback: https://github.com/kishoreg/fullmatix/blob/master/mysql-cluster/s= rc/main/java/org/apache/fullmatix/mysql/MasterSlaveRebalancer.java =0A= =0A= =0A= =0A= =0A= Regarding setting up the Helix controller=2C you actually don't need to ins= tantiate a GenericHelixController. If you create a HelixManager with Instan= ceType.CONTROLLER=2C then ZKHelixManager automatically creates a GenericHel= ixController and sets it up with leader=0A= election. We really should update the documentation to clarify that. =0A= =0A= =0A= =0A= Date: Thu=2C 31 Jul 2014 23:00:13 -0700=0A= =0A= =0A= Subject: Re: Questions about custom helix rebalancer/controller/agent =0A= From: varun@pinterest.com =0A= To: user@helix.apache.org =0A= =0A= =0A= Thanks for the suggestions..=0A= =0A= =0A= Seems like the HelixCustomCodeRunner could do it. However=2C it seems like = the CustomCodeRunner only provides hooks for plugging into notifications. T= he documentation example in the above link suggests a run() method=2C which= does not seem to exist.=0A= =0A= =0A= =0A= =0A= =0A= However=2C this maybe sufficient for my case. I essentially hook in an empt= y CustomCodeRunner into my helix manager. Then I can instantiate my own thr= ead which would run above snippet and keep writing ideal states every 30 mi= nutes. I guess I would still=0A= need to attach the GenericHelixController with the following code snippet = to take action whenever the ideal state changes ??=0A= =0A= =0A= =0A= GenericHelixController controller =3D new GenericHelixController()=3B=0A= manager.addConfigChangeListener(controller)=3B=0A= manager.addLiveInstanceChangeListener(controller)=3B=0A= manager.addIdealStateChangeListener(controller)=3B=0A= manager.addExternalViewChangeListener(controller)=3B=0A= manager.addControllerListener(controller)=3B=0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= On Thu=2C Jul 31=2C 2014 at 6:01 PM=2C kishore g wrot= e: =0A= =0A= =0A= =0A= List resourceList =3D helixAdmin.getResourceList()=3B=0A= for each resource:=0A= Compute target ideal state=0A= =0A= helixAdmin.setIdealState(resource=2C targetIdealState)=3B=0A= =0A= =0A= Thread.sleep(30minutes)=3B =0A= =0A= =0A= This can work right. This code can be as part of CustomCodeRunner. =0A= http://helix.apache.org/javadocs/0.6.3/reference/org/apache/helix/participa= nt/HelixCustomCodeRunner.html. You can say you are interested in notificati= ons but can ignore that. =0A= =0A= =0A= thanks=2C =0A= Kishore G =0A= =0A= =0A= =0A= =0A= =0A= =0A= On Thu=2C Jul 31=2C 2014 at 5:45 PM=2C Kanak Biscuitwala wrote: =0A= =0A= =0A= i.e. helixAdmin.enableCluster(clusterName=2C false)=3B =0A= =0A= =0A= =0A= =0A= =0A= From: kanak.b@hotmail.com =0A= To: user@helix.apache.org =0A= Subject: RE: Questions about custom helix rebalancer/controller/agent =0A= Date: Thu=2C 31 Jul 2014 17:44:40 -0700=0A= =0A= =0A= =0A= Unfortunately HelixAdmin#rebalance is a misnomer=2C and it is a function of= all the configured instances and not the live instances. The closest you c= an get to that is to use the third option I listed related to CUSTOMIZED mo= de=2C where you write=0A= the mappings yourself based on what is live.=0A= =0A= =0A= Another thing you could do is pause the cluster controller and unpause it f= or a period every 30 minutes. That will essentially enforce that the contro= ller will not send transitions (or do anything else=2C really) during the t= ime it is paused. This sounds=0A= a little like a hack to me=2C but it may do what you want.=0A= =0A= =0A= Kanak =0A= =0A= =0A= =0A= Date: Thu=2C 31 Jul 2014 17:39:40 -0700 =0A= Subject: Re: Questions about custom helix rebalancer/controller/agent =0A= From: varun@pinterest.com =0A= To: user@helix.apache.org =0A= =0A= Thanks Kanak=2C for your detailed response and this is really very helpful.= I was wondering if its possible for me do something like the following:=0A= =0A= =0A= List resourceList =3D helixAdmin.getResourceList()=3B=0A= for each resource:=0A= Compute target ideal state=0A= helixAdmin.rebalance(resource)=3B=0A= =0A= =0A= Thread.sleep(30minutes)=3B=0A= =0A= =0A= So=2C the above happens inside a while loop thread and this is the only pla= ce where we do the rebalancing ?=0A= =0A= =0A= Thanks=0A= Varun=0A= =0A= =0A= =0A= On Thu=2C Jul 31=2C 2014 at 5:25 PM=2C Kanak Biscuitwala wrote: =0A= =0A= =0A= =0A= Hi Varun=2C=0A= =0A= =0A= Sorry for the delay. =0A= =0A= =0A= 1 and 3) There are a number of ways to do this=2C with various tradeoffs.= =0A= =0A= =0A= - You can write a user-defined rebalancer. In helix 0.6.x=2C it involves im= plementing the following interface:=0A= =0A= =0A= https://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/o= rg/apache/helix/controller/rebalancer/Rebalancer.java=0A= =0A= =0A= =0A= =0A= =0A= Essentially what it does is given an existing ideal state=2C compute a new = ideal state. For 0.6.x=2C this will read the preference lists in the output= ideal state and compute a state mapping based on them. If you need more co= ntrol=2C you can also implement:=0A= =0A= =0A= =0A= =0A= =0A= https://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/o= rg/apache/helix/controller/rebalancer/internal/MappingCalculator.java=0A= =0A= =0A= =0A= =0A= =0A= which will allow you to create a mapping from partition to map of participa= nt and state. In 0.7.x=2C we consolidated these into a single method.=0A= =0A= =0A= Here is a tutorial on the user-defined rebalancer: http://helix.apache.org/= 0.6.3-docs/tutorial_user_def_rebalancer.html=0A= =0A= =0A= =0A= =0A= =0A= Now=2C running this every 30 minutes is tricky because by default the contr= oller responds to all cluster events (and really it needs to because it agg= regates all participant current states into the external view -- unless you= don't care about that).=0A= =0A= =0A= =0A= =0A= =0A= - Combined with the user-defined rebalancer (or not)=2C you can have a Gene= ricHelixController that doesn't listen on any events=2C but calls startReba= lancingTimer()=2C into which you can pass 30 minutes. The problem with this= is that the instructions at http://helix.apache.org/0.6.3-docs/tutorial_co= ntroller.html won't=0A= work as described because of a known issue. The workaround is to connect H= elixManager as role ADMINISTRATOR instead of CONTROLLER.=0A= =0A= =0A= However=2C if you connect as ADMINISTRATOR=2C you have to set up leader ele= ction yourself (assuming you want a fault-tolerant controller). See https:/= /github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/org/apac= he/helix/manager/zk/DistributedLeaderElection.java for=0A= a controller change listener that can do leader election=2C but your versi= on will have to be different=2C as you actually don't want to add listeners= =2C but rather set up a timer.=0A= =0A= =0A= This also gives you the benefit of plugging in your own logic into the cont= roller pipeline. See https://github.com/apache/helix/blob/helix-0.6.x/helix= -core/src/main/java/org/apache/helix/controller/GenericHelixController.java= createDefaultRegistry()=0A= for how to create an appropriate PipelineRegistry.=0A= =0A= =0A= - You can take a completely different approach and put your ideal state in = CUSTOMIZED rebalance mode. Then you can have a meta-resource where one part= icipant is a leader and the others are followers (you can create an ideal s= tate in SEMI_AUTO mode=2C where=0A= the replica count and the replica count and preference list of resourceNam= e_0 is "ANY_LIVEINSTANCE". When one participant is told to become leader=2C= you can set a timer for 30 minutes and update and write the map fields of = the ideal state accordingly.=0A= =0A= =0A= =0A= =0A= =0A= 2) I'm not sure I understand the question. If you're in the JVM=2C you simp= ly need to connect as a PARTICIPANT for your callbacks=2C but that can just= be something you do at the beginning of your node startup. The rest of you= r code is more or less governed=0A= by your transitions=2C but if there are things you need to do on the side= =2C there is nothing in Helix preventing you from doing so. See http://heli= x.apache.org/0.6.3-docs/tutorial_participant.html for=0A= participant logic.=0A= =0A= =0A= 4) The current state is per-instance and is literally called CurrentState. = For a given participant=2C you can query a current state by doing something= like:=0A= =0A= =0A= HelixDataAccessor accessor =3D helixManager.getHelixDataAccessor()=3B=0A= CurrentState currentState =3D accessor.getProperty(accessor.keyBuilder().cu= rrentState(instanceName=2C sessionId=2C resourceName)=3B=0A= =0A= =0A= If you implement a user-defined rebalancer as above=2C we automatically agg= regate all these current states into a CurrentStateOutput object.=0A= =0A= =0A= 5) You can use a Helix spectator:=0A= =0A= =0A= http://helix.apache.org/0.6.3-docs/tutorial_spectator.html=0A= =0A= =0A= This basically gives you a live-updating routing table for the mappings of = the Helix-managed resource. However=2C it requires the external view to be = up to date=2C going back to my other point of perhaps separating the concep= t of changing mappings every 30=0A= minutes from the frequency at which the controller runs.=0A= =0A= =0A= Hopefully this helps.=0A= =0A= =0A= Kanak=0A= =0A= =0A= =0A= =0A= =0A= Date: Thu=2C 31 Jul 2014 12:13:27 -0700 =0A= Subject: Questions about custom helix rebalancer/controller/agent =0A= From: varun@pinterest.com =0A= To: user@helix.apache.org=0A= =0A= =0A= =0A= Hi=2C=0A= =0A= =0A= I am trying to write a customized rebalancing algorithm. I would like to ru= n the rebalancer every 30 minutes inside a single thread. I would also like= to completely disable Helix triggering=0A= the rebalancer.=0A= =0A= =0A= I have a few questions:=0A= 1) What's the best way to run the custom controller ? Can I simply instanti= ate a ZKHelixAdmin object and then keep running my rebalancer inside a thre= ad or do I need to do something more.=0A= =0A= =0A= =0A= =0A= =0A= Apart from rebalancing=2C I want to do other things inside the the controll= er=2C so it would be nice if I could simply fire up the controller through = code. I could not find this in the documentation.=0A= =0A= =0A= =0A= =0A= =0A= 2) Same question for the Helix agent. My Helix Agent is a JVM process which= does other things apart from exposing the callbacks for state transitions.= Is there a code sample for the same=0A= ?=0A= =0A= =0A= 3) How do I disable Helix triggered rebalancing once I am able to run the c= ustom controller ?=0A= =0A= =0A= 4) During my custom rebalance run=2C how I can get the current cluster stat= e - is it through ClusterDataCache.getIdealState() ?=0A= =0A= =0A= 5) For clients talking to the cluster=2C does helix provide an easy abstrac= tion to find the partition distribution for a helix resource ?=0A= =0A= =0A= Thanks=0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= =0A= = --_7b4eed80-763e-4506-97ff-a0432823507b_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Yeah=2C it's not possible to att= ach to a controller because the custom code runner essentially treats each = runner as a participant for a logical resource corresponding to leadership = of running that code.

Rather than using HelixCustomCodeR= unner=2C perhaps you can just use a basic scheduled timer task on each cont= roller=2C and when it is triggered=2C check if you're the leader=2C and if = so=2C run the ideal state update code. Another thing you can do is helixMan= ager#addControllerListener()=2C and on the callback check if you're leader= =2C and schedule the timer if you are=2C and otherwise cancel the timer.

Here's how you can check if you're leader:

HelixDataAccessor accessor =3D helixManager.getHelixDataAcc= essor()=3B
LiveInstance leader =3D accessor.getProperty(accessor.= keyBuilder().controllerLeader())=3B
if (leader !=3D null &=3B&= amp=3B leader.getId().equals(myId)) {
 =3B // I am the leader=
}

Date: Wed=2C 6 Aug 2014 15= :56:35 -0700
Subject: Re: Questions about custom helix rebalancer/contro= ller/agent
From: varun@pinterest.com
To: user@helix.apache.org
I am attempting to attach the HelixCustomCodeRunner to a = controller instance - not really running a controller alongside each of my = nodes. HelixCustomCodeRunner.start() is failing as above with a nullpointer= exception at line 120. Is it not possible to attach the HelixCustomCodeRun= ner to a controller instance ?
=0A=
Thanks !
Varun


On Wed=2C Aug 6=2C 2014 at 3:46= PM=2C Kanak Biscuitwala <=3Bkan= ak.b@hotmail.com>=3B wrote:
=0A=
=0A= =0A= =0A=
I would suggest maintaining 2 HelixManager connection= s: one for CONTROLLER=2C one for PARTICIPANT (I'm assuming you're running a= controller instance alongside each of your nodes). It's wasteful=2C but yo= u should just leave the controller one alone=2C and then attach the state m= odel factory and custom code runner to the participant one.
=0A=
Kanak

Date: Wed=2C 6 Aug 2014 15:43:16 -0700=

Subject: Re: Questions about custom helix rebala= ncer/controller/agent
From: varun@pinterest.com<= /a>
=0A= To: user@helix.apache.org

Without this I was getting a null pointer exception in the CustomC= odeRunner - Helix 0.6.3 - Lines 120 and 121
=0A=
=0A=
=0A= =0A= 
 =3B =3B =3B =3BStateMachineEng=
ine =3BstateMach =3B=3D =3B_manager.getStateMachineEngine()=3B
=0A= =0A=
=0A= 
121=0A=
=0A=
=0A= =0A= 
 =3B =3B =3B =3Bstat=
eMach.registerStateModelFactory(LEADER_STANDBY=2C =3B_stateModelFty=2C =3B=
_resourceName)=3B
=0A= =0A=

=0A= 
My code calls the HelixCustomCodeRunner in=
 the following way:
=0A= =0A= 

   new HelixCustomCodeRunner(helixManager=2C zookeeperQuorum).=0A=
        on(HelixConstants.ChangeType.LIVE_INSTANCE).invoke(myCallback).=0A=
        usingLeaderStandbyModel("HDFS_rebalancer").start()=3B
=0A= =0A=


On Wed=2C Aug 6=2C 2014 at 3:08 PM=2C Kan= ak Biscuitwala <=3Bkanak.b@hotma= il.com>=3B wrote:
=0A=
=0A= =0A= =0A=
Hi Varun=2C

getStateMachineEngine = is only supported for InstanceType.PARTICIPANT. May I ask why you need your= controller to have state transition callbacks?

In= future releases=2C we're creating separate classes for each role=2C so hop= efully that will resolve confusions like this moving forward.
=0A= =0A=

Kanak

Date: Wed=2C 6 Aug 2014 15:04:36 = -0700

Subject: Re: Questions about custom helix rebalancer/con= troller/agent
From: varun@pinterest.com
= =0A= =0A= To: user@helix.apache.org

I am getting a weird null pointer exception while instantiating th= e controller. Here is the error:

=0A=
=0A=
this.helixManager =3D HelixManagerFactory.getZKHelixManager(this.clust= erName=2C
=0A=  =3B  =3B  =3B  =3B InetAddress.getLocalHost().getHostName(= ) + ":" + thriftPort=2C
 =3B  =3B  =3B  =3B Insta= nceType.CONTROLLER=2C
 =3B  =3B  =3B  =3B zookeep= erQuorum)=3B
StateMachineEngine machineEngine =3D helixManager.ge= tStateMachineEngine()=3B
=0A= =0A= =0A=
machineEngine.registerStateModelFactory("HDFS_state_machine"=2C
=
 =3B  =3B  =3Bnew OnlineOfflineStateModelFact= ory(1000))=3B
this.helixManager.connect()=3B
=
=0A= =0A= =0A= I get a NullPointerException at line #3 because getStateMachineEngine() ret= urns a null value. Is that supposed to happen ?

Th= anks
Varun


=0A= On Fri=2C Aug 1=2C 2014 at 11:23 AM=2C Zhen Zhang <=3B<= a href=3D"mailto:zzhang@linkedin.com" target=3D"_blank" onclick=3D"window.o= pen('https://mail.google.com/mail/?view=3Dcm&=3Btf=3D1&=3Bto=3Dzzhang= @linkedin.com&=3Bcc=3D&=3Bbcc=3D&=3Bsu=3D&=3Bbody=3D'=2C'_blank= ')=3Breturn false=3B">zzhang@linkedin.com>=3B wrote:
=0A= =0A= =0A= =0A= =0A= =0A= =0A=
=0A=
Hi Varun=2C =3B
=0A=

=0A=
=0A=
The state transitions will be independent. Helix controller may send M= ASTER->=3BOFFLINE to all three nodes=2C for example=2C and if node1 compl= etes the MASTER->=3BOFFLINE transition first=2C controller will send OFFL= INE->=3BDROPPED to node1 first. Or if all three nodes=0A= completes MASTER->=3BOFFLINE at the same time=2C controller may send OFF= LINE->=3BDROPPED to all three nodes together.
=0A=

=0A=
=0A=
Thanks=2C
=0A=
Jason
=0A=

=0A=
=0A= =0A=
=0A= =0A= =0A= =0A= From: Varun Sharma <=3Bvarun@pinterest.com>=3B
=0A= Reply-To: "user@helix.apache.org" <=3Buser@he= lix.apache.org>=3B
=0A= =0A= =0A= =0A= Date: Friday=2C August 1=2C 2014= 11:10 AM
=0A= To: "= user@helix.apache.org" <=3Buser@helix.ap= ache.org>=3B
=0A= =0A= =0A=

=0A= Subject: Re: Questions about cus= tom helix rebalancer/controller/agent
=0A=
=0A=

=0A=
=0A=
=0A=
=0A=
Thanks a lot. Most of my questions are answered=2C except = I have one follow up question.=0A=

=0A=
=0A=
Lets say I have a situation with 3 masters per partition. For partitio= n X=2C these are on node1=2C node2 and node3. Upon dropping the resource=2C= would the partition X be offlined on all three nodes and then dropped or c= an that be independent as in=2C node1 offlines=0A= and drops=2C followed by node2 and so on. Just want to check if we first w= ait for all the masters to offline and then initiate the offline->=3Bdrop= or the other way round.
=0A=

=0A=
=0A=
Thanks !
=0A=
Varun
=0A=
=0A=

=0A=
=0A=
On Fri=2C Aug 1=2C 2014 at 10:33 AM=2C Kanak Biscuitwala =0A= <=3Bkanak.b@hotmail.com>=3B wrote:=
=0A=
=0A=
=0A=

=0A= Dropping a resource will cause the controller to first send MASTER -->=3B= OFFLINE for all partitions=2C and then OFFLINE -->=3B DROPPED.=0A=

=0A=
=0A=
Kanak
=0A=
=0A=
=0A= Date: Fri=2C 1 Aug 2014 10:30:54 -0700=0A=
=0A=

=0A= Subject: Re: Questions about custom helix rebalancer/controller/agent
= =0A= From: varun@pinterest.com
=0A= To: user@helix.apache.org
=0A=
=0A=
In my case=2C I will have many resources - like say upto a= 100 resources. Each of them will have partitions in the range of 100-5K. &= nbsp=3BSo I guess=2C I do require the bucket size. 300K partitions is the s= um of partitions across all resources=2C rather=0A= than the # of partitions within a single resource.=0A=

=0A=
=0A=
Another question=2C I had was regarding removing a resource in Helix. = When a removeResource is called from HelixAdmin=2C would it trigger the MAS= TER->=3BOFFLINE the respective partitions before the resource is removing= ? To concretize my use case=2C we have many=0A= resources with a few thousand partitions being loaded every day. New versi= ons of the resources keep getting loaded as brand new resources into Helix = and the older versions are decommissioned/garbage collected. So we would be= issuing upto a 100 or so resource=0A= additions per day and upto a 100 or so resource deletions every day. Just = want to check that deleting a resource would also trigger the appropriate M= ASTER->=3BOFFLINE transitions. =3B
=0A=

=0A=
=0A=
Thanks
=0A=
Varun
=0A=
=0A=

=0A=
=0A=
On Fri=2C Aug 1=2C 2014 at 10:18 AM=2C Kanak Biscuitwala <=3Bkanak.b@hotmail.com>=3B wr= ote:
=0A=
=0A=
=0A=
a) By default=2C there is one znode per resource=2C which = as you know is a grouping of partitions. The biggest limitation is that ZK = has a 1MB limit on znode sizing. To get around this=2C Helix has the concep= t of bucketizing=2C where in your ideal state=2C=0A= you can set a bucket size=2C which will effectively create that many znode= s to fully represent all your partitions. I believe that you can have ~2k p= artitions before you start needing to bucketize.=0A=

=0A=
=0A=
300k may cause you separate issues=2C and you may want to consider doi= ng things like enabling batch message mode in your ideal state so that each= message we send to an instance contains transitions for all partitions hos= ted on that instance=2C rather than=0A= creating a znode per partition state change. However=2C in theory (we've n= ever played with this many in practice)=2C Helix should be able to function= correctly with that many partitions.
=0A=

=0A=
=0A=
b) Yes=2C if you have a hard limit of 1 master per partition=2C Helix = will transition the first node to OFFLINE before sending the MASTER transit= ion to the new master.
=0A=

=0A=
=0A=
Kanak
=0A=
=0A=
=0A=
=0A= Date: Fri=2C 1 Aug 2014 10:09:24 -0700=0A=
=0A=

=0A= Subject: Re: Questions about custom helix rebalancer/controller/agent
= =0A= From: varun@pinterest.com
=0A= To: user@helix.apache.org
=0A=
=0A=
Sounds fine to me. I can work without the FINALIZE notific= ation for now=2C but I hope its going to come out soon. A few more question= s:=0A=

=0A=
=0A=
a) How well does Helix scale with partitions - is each partition a sep= arate znode inside helix ? If I have 300K partitions in Helix would that be= an issue ?
=0A=
b) If a partition which was assigned as a master on node1 is now assig= ned as a master on node2=2C will node1 get a callback execution for transit= ion from MASTER-->=3BOFFLINE
=0A=

=0A=
=0A=
Thanks
=0A=
Varun
=0A=
=0A=

=0A=
=0A=
On Thu=2C Jul 31=2C 2014 at 11:18 PM=2C Kanak Biscuitwala <=3Bkanak.b@hotmail.com>=3B wr= ote:
=0A=
=0A=
=0A=
s/run/start/g -- sorry about that=2C fixed in javadocs for= future releases
=0A=
=0A= You may need to register for a notification type=3B I believe HelixCustomCo= deRunner complains if you don't. However=2C you can simply ignore that noti= fication type=2C and just check for INIT and FINALIZE notification types in= your callback to to track whether or=0A= not you're the leader. On INIT=2C you start your 30 minute timer=2C and on= FINALIZE you stop it. You may need to wait for us to make a 0.6.4 release = (we will likely do this soon) to get the FINALIZE notification.
=0A=
=0A= Here is an example of a custom code runner usage:=0A= =0A= =0A= =0A= =0A=
Callback: =3Bhttps://github.com/kishoreg/fullmatix/blo= b/master/mysql-cluster/src/main/java/org/apache/fullmatix/mysql/MasterSlave= Rebalancer.java
=0A= =0A= =0A= =0A=
=0A= Regarding setting up the Helix controller=2C you actually don't need to ins= tantiate a GenericHelixController. If you create a HelixManager with Instan= ceType.CONTROLLER=2C then ZKHelixManager automatically creates a GenericHel= ixController and sets it up with leader=0A= election. We really should update the documentation to clarify that.
= =0A=
=0A=
=0A=
=0A= Date: Thu=2C 31 Jul 2014 23:00:13 -0700=0A=
=0A=

=0A= Subject: Re: Questions about custom helix rebalancer/controller/agent
= =0A= From: varun@pinterest.com
=0A= To: user@helix.apache.org
=0A=
=0A=
=0A=
Thanks for the suggestions..
=0A=

=0A=
=0A=
Seems like the HelixCustomCodeRunner could do it. However=2C it seems = like the CustomCodeRunner only provides hooks for plugging into notificatio= ns. The documentation example in the above link suggests a run() method=2C = which does not seem to exist.
=0A= =0A= =0A= =0A=

=0A=
=0A=
However=2C this maybe sufficient for my case. I essentially hook in an= empty CustomCodeRunner into my helix manager. Then I can instantiate my ow= n thread which would run above snippet and keep writing ideal states every = 30 minutes. I guess I would still=0A= need to attach the GenericHelixController with the following code snippet = to take action whenever the ideal state changes ??
=0A=

=0A=
=0A=
=0A= 
GenericHel=
ixController controller =3D new GenericHelixController()=3B=0A=
     manager.addConfigChangeListener(controller)=3B=0A=
     manager.addLiveInstanceChangeListener(controller)=3B=0A=
     manager.addIdealStateChangeListener(controller)=3B=0A=
     manager.addExternalViewChangeListener(controller)=3B=0A=
     manager.addControllerListener(controller)=3B
=0A=
=0A=

=0A=
=0A=

=0A=
=0A=
=0A=

=0A=
=0A=
On Thu=2C Jul 31=2C 2014 at 6:01 PM=2C kishore g <= =3Bg.kishore@gmail.com>=3B wrote:
= =0A=
=0A=
=0A=
=0A=
List resourceList =3D helixAdmin.getResourceList()=3B
=0A=
for each resource:
=0A=
 =3B  =3BCompute target ideal state
=0A=
=0A=
 =3B  =3BhelixAdmin.setIdealState(resource=2C targetIdealState= )=3B
=0A=

=0A=
=0A=
Thread.sleep(30minutes)=3B
=0A=
=0A=
=0A=
This can work right. This code can be as part of CustomCodeRunner. =0A= http://helix.apache.org/javadocs/0.6.3/reference/org/apache/helix/participa= nt/HelixCustomCodeRunner.html. You can say you are interested in notifi= cations but can ignore that.
=0A=
=0A=
=0A=
thanks=2C
=0A= Kishore G
=0A=
=0A=
=0A=
=0A=
=0A=

=0A=
=0A=
On Thu=2C Jul 31=2C 2014 at 5:45 PM=2C Kanak Biscuitwala <=3Bkanak.b@hotmail.com>=3B wr= ote:
=0A=
=0A=
=0A=
i.e. helixAdmin.enableCluster(clusterName=2C false)=3B
= =0A=

=0A=
=0A=
=0A=
=0A=
=0A= From: kanak.b@hotmail.com
=0A= To: user@helix.apache.org
=0A= Subject: RE: Questions about custom helix rebalancer/controller/agent
= =0A= Date: Thu=2C 31 Jul 2014 17:44:40 -0700
=0A=
=0A=

=0A=
=0A=
Unfortunately HelixAdmin#rebalance is a misnomer=2C and it= is a function of all the configured instances and not the live instances. = The closest you can get to that is to use the third option I listed related= to CUSTOMIZED mode=2C where you write=0A= the mappings yourself based on what is live.=0A=

=0A=
=0A=
Another thing you could do is pause the cluster controller and unpause= it for a period every 30 minutes. That will essentially enforce that the c= ontroller will not send transitions (or do anything else=2C really) during = the time it is paused. This sounds=0A= a little like a hack to me=2C but it may do what you want.
=0A=

=0A=
=0A=
Kanak
=0A=
=0A=
=0A=
=0A= Date: Thu=2C 31 Jul 2014 17:39:40 -0700
=0A= Subject: Re: Questions about custom helix rebalancer/controller/agent
= =0A= From: varun@pinterest.com
=0A= To: user@helix.apache.org
=0A=
=0A=
Thanks Kanak=2C for your detailed response and this is rea= lly very helpful. I was wondering if its possible for me do something like = the following:=0A=

=0A=
=0A=
List resourceList =3D helixAdmin.getResourceList()=3B
=0A=
for each resource:
=0A=
 =3B  =3BCompute target ideal state
=0A=
 =3B  =3BhelixAdmin.rebalance(resource)=3B
=0A=

=0A=
=0A=
Thread.sleep(30minutes)=3B
=0A=

=0A=
=0A=
So=2C the above happens inside a while loop thread and this is the onl= y place where we do the rebalancing ?
=0A=

=0A=
=0A=
Thanks
=0A=
Varun
=0A=
=0A=

=0A=
=0A=
On Thu=2C Jul 31=2C 2014 at 5:25 PM=2C Kanak Biscuitwala <=3Bkanak.b@hotmail.com>=3B wr= ote:
=0A=
=0A=
=0A=
=0A=
Hi Varun=2C
=0A=

=0A=
=0A=
Sorry for the delay.
=0A=

=0A=
=0A=
1 and 3) There are a number of ways to do this=2C with various tradeof= fs.
=0A=

=0A=
=0A=
- You can write a user-defined rebalancer. In helix 0.6.x=2C it involv= es implementing the following interface:
=0A=

=0A=
=0A=
https://github.com/apache/helix/blob/helix-0.6.x/helix-core/s= rc/main/java/org/apache/helix/controller/rebalancer/Rebalancer.java=0A= =0A= =0A= =0A=

=0A=
=0A=
Essentially what it does is given an existing ideal state=2C compute a= new ideal state. For 0.6.x=2C this will read the preference lists in the o= utput ideal state and compute a state mapping based on them. If you need mo= re control=2C you can also implement:
=0A= =0A= =0A= =0A=

=0A=
=0A= =0A= =0A= =0A= =0A=

=0A=
=0A=
which will allow you to create a mapping from partition to map of part= icipant and state. In 0.7.x=2C we consolidated these into a single method.<= /div>=0A=

=0A=
=0A=
Here is a tutorial on the user-defined rebalancer: =3Bhttp://helix.apache.org/0.6.3-docs= /tutorial_user_def_rebalancer.html
=0A= =0A= =0A= =0A=

=0A=
=0A=
Now=2C running this every 30 minutes is tricky because by default the = controller responds to all cluster events (and really it needs to because i= t aggregates all participant current states into the external view -- unles= s you don't care about that).
=0A= =0A= =0A= =0A=

=0A=
=0A=
- Combined with the user-defined rebalancer (or not)=2C you can have a= GenericHelixController that doesn't listen on any events=2C but calls star= tRebalancingTimer()=2C into which you can pass 30 minutes. The problem with= this is that the instructions at =3Bhttp://helix.apache.org/0.6.3-docs/tutorial_controller.html&nb= sp=3Bwon't=0A= work as described because of a known issue. The workaround is to connect H= elixManager as role ADMINISTRATOR instead of CONTROLLER.
=0A=

=0A=
=0A=
However=2C if you connect as ADMINISTRATOR=2C you have to set up leade= r election yourself (assuming you want a fault-tolerant controller). See&nb= sp=3Bhttps://github.com/apache/hel= ix/blob/helix-0.6.x/helix-core/src/main/java/org/apache/helix/manager/zk/Di= stributedLeaderElection.java =3Bfor=0A= a controller change listener that can do leader election=2C but your versi= on will have to be different=2C as you actually don't want to add listeners= =2C but rather set up a timer.
=0A=

=0A=
=0A=
This also gives you the benefit of plugging in your own logic into the= controller pipeline. See =3Bhttp= s://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/org/a= pache/helix/controller/GenericHelixController.java =3BcreateDefault= Registry()=0A= for how to create an appropriate PipelineRegistry.
=0A=

=0A=
=0A=
- You can take a completely different approach and put your ideal stat= e in CUSTOMIZED rebalance mode. Then you can have a meta-resource where one= participant is a leader and the others are followers (you can create an id= eal state in SEMI_AUTO mode=2C where=0A= the replica count and the replica count and preference list of resourceNam= e_0 is "ANY_LIVEINSTANCE". When one participant is told to become leader=2C= you can set a timer for 30 minutes and update and write the map fields of = the ideal state accordingly.
=0A= =0A= =0A= =0A=

=0A=
=0A=
2) I'm not sure I understand the question. If you're in the JVM=2C you= simply need to connect as a PARTICIPANT for your callbacks=2C but that can= just be something you do at the beginning of your node startup. The rest o= f your code is more or less governed=0A= by your transitions=2C but if there are things you need to do on the side= =2C there is nothing in Helix preventing you from doing so. See =3Bhttp://helix.apache.org/0.6.3-docs= /tutorial_participant.html =3Bfor=0A= participant logic.
=0A=

=0A=
=0A=
4) The current state is per-instance and is literally called CurrentSt= ate. For a given participant=2C you can query a current state by doing some= thing like:
=0A=

=0A=
=0A=
HelixDataAccessor accessor =3D helixManager.getHelixDataAccessor()=3B<= /div>=0A=
CurrentState currentState =3D accessor.getProperty(accessor.keyBuilder= ().currentState(instanceName=2C sessionId=2C resourceName)=3B
=0A=

=0A=
=0A=
If you implement a user-defined rebalancer as above=2C we automaticall= y aggregate all these current states into a CurrentStateOutput object.=0A=

=0A=
=0A=
5) You can use a Helix spectator:
=0A=

=0A=
=0A= =0A=

=0A=
=0A=
This basically gives you a live-updating routing table for the mapping= s of the Helix-managed resource. However=2C it requires the external view t= o be up to date=2C going back to my other point of perhaps separating the c= oncept of changing mappings every 30=0A= minutes from the frequency at which the controller runs.
=0A=

=0A=
=0A=
Hopefully this helps.
=0A=

=0A=
=0A=
Kanak
=0A=

=0A=
=0A=
=0A=
=0A=
=0A= Date: Thu=2C 31 Jul 2014 12:13:27 -0700
=0A= Subject: Questions about custom helix rebalancer/controller/agent
=0A= From: varun@pinterest.com
=0A= To: user@helix.apache.org=0A=
=0A=

=0A=
=0A=
Hi=2C=0A=

=0A=
=0A=
I am trying to write a customized rebalancing algorithm. I would lik= e to run the rebalancer every 30 minutes inside a single thread. I would al= so like to completely disable Helix triggering=0A= the rebalancer.
=0A=

=0A=
=0A=
I have a few questions:
=0A=
1) What's the best way to run the custom controller ? Can I simply i= nstantiate a ZKHelixAdmin object and then keep running my rebalancer inside= a thread or do I need to do something more.
=0A= =0A= =0A= =0A=

=0A=
=0A=
Apart from rebalancing=2C I want to do other things inside the the c= ontroller=2C so it would be nice if I could simply fire up the controller t= hrough code. I could not find this in the documentation.
=0A= =0A= =0A= =0A=

=0A=
=0A=
2) Same question for the Helix agent. My Helix Agent is a JVM proces= s which does other things apart from exposing the callbacks for state trans= itions. Is there a code sample for the same=0A= ?
=0A=

=0A=
=0A=
3) How do I disable Helix triggered rebalancing once I am able to ru= n the custom controller ?
=0A=

=0A=
=0A=
4) During my custom rebalance run=2C how I can get the current clust= er state - is it through ClusterDataCache.getIdealState() ?
=0A=

=0A=
=0A=
5) For clients talking to the cluster=2C does helix provide an easy = abstraction to find the partition distribution for a helix resource ?
= =0A=

=0A=
=0A=
Thanks
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A=
=0A= =0A=

=0A=

=0A=
= --_7b4eed80-763e-4506-97ff-a0432823507b_--