helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kanak Biscuitwala <kana...@hotmail.com>
Subject RE: helix provisioning
Date Sun, 26 Jan 2014 22:28:10 GMT
Looks good to me. It might also be nice to have support for key-value service-level configs.
Also, is it easy to externally change the upper and lower bounds for min/max containers?

Date: Sun, 26 Jan 2014 14:17:51 -0800
Subject: Re: helix provisioning
From: g.kishore@gmail.com
To: user@helix.apache.org
CC: dev@helix.apache.org

Here is a sample app spec. does it make sense ?

appName: test
configs:
    k1: v1
services:
  - name: myservice
    participantClass: org.apache.helix.myApp.SimpleWebserver 
    minContainers:3

    maxContainers:3              
    configs:
        - participantId: myservice_0         
          port: 9500
        - participantId: myservice_1        
          port: 9501 
        - participantId: myservice_2         

          port: 9502
resources:
  - name: distributedLock
    numPartitions: 6
    numReplicas: 2
    rebalanceMode: FULL_AUTO
    stateModel: OnlineOffline
    tag: myservice 
    configs: 
        k1: v1





On Sat, Jan 18, 2014 at 11:22 AM, kishore g <g.kishore@gmail.com> wrote:

Thats a good suggestion. 

We also need to think how the controller pipeline works.


That means the targetprovider will now be across multiple services. Currently we have ability
to plugin rebalancer and provisioners per resource. But sometimes this is too fine and might
need logic that spans across resources and services.



I started another thread on composite rebalancers, I think we might need a similar concept
in targetprovider, provisioners.






On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <kanak.b@hotmail.com> wrote:





Maybe it should be at both scopes? Sort of like putting bounds cross-app and cross-service
within an app. This allows full control over an organization's physical resources.

> Date: Sat, 18 Jan 2014 09:28:32 -0800


> Subject: Re: helix provisioning
> From: g.kishore@gmail.com
> To: dev@helix.apache.org; user@helix.apache.org


> 
> Thanks Swaroop for adding interfaces to specify the application spec.
> 
> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning


> 
> From what I understand
> AppName maps to cluster name
> Service maps to instances, there can be multiple service components in an
> application. For example, in distributed data system, there may be a


> storage layer and router layer.
> 
> The min/max containers seems to be at application level but it probably
> should be at a Service scope ?
> 
> what do you think ?
> 
> 


> 
> 
> 
> 
> 
> 
> 
> 
> On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g.kishore@gmail.com> wrote:
> 


> > Hi,
> >
> > Here is a prototype of supporting the concept of provisioning in Helix.
> > This was the work done by one of the interns at LinkedIn. You can find more
> > info here


> >
> > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn


> >
> > The implementation done by intern was outside of Helix. Kanak and I took a
> > stab at it to treat provisioning as a first class citizen of Helix.
> >
> > The work is in helix-provisioning branch


> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning


> >
> > Most of the code is here,
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning


> >
> > Here is a YARN specific implementation
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning


> >
> > Its still a WIP and main goal was to see if we can abstract Resource
> > Managers like YARN, Mesos, EC2 from the application.
> >
> > I am working on providing the steps to try out launching a Helix cluster


> > on YARN.
> >
> > Feedback/Thoughts? Any interest in contributing to make this production
> > ready ?
> >
> > thanks,
> > Kishore G
> >
> >


> >
> >
> >
> >
> >
> >
 		 	   		  



 		 	   		  
Mime
View raw message