helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: helix provisioning
Date Sun, 26 Jan 2014 22:17:51 GMT
Here is a sample app spec. does it make sense ?

appName: test
configs:
    k1: v1
services:
  - name: myservice
    participantClass: org.apache.helix.myApp.SimpleWebserver
    minContainers:3
    maxContainers:3
    configs:
        - participantId: myservice_0
          port: 9500
        - participantId: myservice_1
          port: 9501
        - participantId: myservice_2
          port: 9502
resources:
  - name: distributedLock
    numPartitions: 6
    numReplicas: 2
    rebalanceMode: FULL_AUTO
    stateModel: OnlineOffline
    tag: myservice
    configs:
        k1: v1




On Sat, Jan 18, 2014 at 11:22 AM, kishore g <g.kishore@gmail.com> wrote:

> Thats a good suggestion.
>
> We also need to think how the controller pipeline works.
>
> That means the targetprovider will now be across multiple services.
> Currently we have ability to plugin rebalancer and provisioners per
> resource. But sometimes this is too fine and might need logic that spans
> across resources and services.
>
> I started another thread on composite rebalancers, I think we might need a
> similar concept in targetprovider, provisioners.
>
>
>
>
> On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <kanak.b@hotmail.com>wrote:
>
>> Maybe it should be at both scopes? Sort of like putting bounds cross-app
>> and cross-service within an app. This allows full control over an
>> organization's physical resources.
>>
>> > Date: Sat, 18 Jan 2014 09:28:32 -0800
>> > Subject: Re: helix provisioning
>> > From: g.kishore@gmail.com
>> > To: dev@helix.apache.org; user@helix.apache.org
>>
>> >
>> > Thanks Swaroop for adding interfaces to specify the application spec.
>> >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>> >
>> > From what I understand
>> > AppName maps to cluster name
>> > Service maps to instances, there can be multiple service components in
>> an
>> > application. For example, in distributed data system, there may be a
>> > storage layer and router layer.
>> >
>> > The min/max containers seems to be at application level but it probably
>> > should be at a Service scope ?
>> >
>> > what do you think ?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <g.kishore@gmail.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > Here is a prototype of supporting the concept of provisioning in
>> Helix.
>> > > This was the work done by one of the interns at LinkedIn. You can
>> find more
>> > > info here
>> > >
>> > >
>> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
>> > >
>> > > The implementation done by intern was outside of Helix. Kanak and I
>> took a
>> > > stab at it to treat provisioning as a first class citizen of Helix.
>> > >
>> > > The work is in helix-provisioning branch
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
>> > >
>> > > Most of the code is here,
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
>> > >
>> > > Here is a YARN specific implementation
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
>> > >
>> > > Its still a WIP and main goal was to see if we can abstract Resource
>> > > Managers like YARN, Mesos, EC2 from the application.
>> > >
>> > > I am working on providing the steps to try out launching a Helix
>> cluster
>> > > on YARN.
>> > >
>> > > Feedback/Thoughts? Any interest in contributing to make this
>> production
>> > > ready ?
>> > >
>> > > thanks,
>> > > Kishore G
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>>
>
>

Mime
View raw message