Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F048410F73 for ; Sat, 18 Jan 2014 17:36:49 +0000 (UTC) Received: (qmail 87894 invoked by uid 500); 18 Jan 2014 17:36:48 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 87804 invoked by uid 500); 18 Jan 2014 17:36:48 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 87791 invoked by uid 99); 18 Jan 2014 17:36:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jan 2014 17:36:48 +0000 X-ASF-Spam-Status: No, hits=3.2 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kanak.b@hotmail.com designates 65.54.190.28 as permitted sender) Received: from [65.54.190.28] (HELO bay0-omc1-s17.bay0.hotmail.com) (65.54.190.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jan 2014 17:36:41 +0000 Received: from BAY173-W15 ([65.54.190.60]) by bay0-omc1-s17.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Sat, 18 Jan 2014 09:36:20 -0800 X-TMN: [d9oWbfaSTSzapZVZJXwV0UbgQRWQ4f1n] X-Originating-Email: [kanak.b@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_05231be6-b63a-45ee-9648-3c8ae4ab7ffa_" From: Kanak Biscuitwala To: , Subject: RE: helix provisioning Date: Sat, 18 Jan 2014 09:36:20 -0800 Importance: Normal In-Reply-To: References: , MIME-Version: 1.0 X-OriginalArrivalTime: 18 Jan 2014 17:36:20.0733 (UTC) FILETIME=[CCD4DAD0:01CF1473] X-Virus-Checked: Checked by ClamAV on apache.org --_05231be6-b63a-45ee-9648-3c8ae4ab7ffa_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Maybe it should be at both scopes? Sort of like putting bounds cross-app an= d cross-service within an app. This allows full control over an organizatio= n's physical resources. > Date: Sat=2C 18 Jan 2014 09:28:32 -0800 > Subject: Re: helix provisioning > From: g.kishore@gmail.com > To: dev@helix.apache.org=3B user@helix.apache.org >=20 > Thanks Swaroop for adding interfaces to specify the application spec. >=20 > https://git-wip-us.apache.org/repos/asf?p=3Dhelix.git=3Ba=3Dblob=3Bf=3Dhe= lix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/Applicati= onSpec.java=3Bh=3D16b23fa02ffbe40c18b9cff286eb5eb321947d84=3Bhb=3Dhelix-pro= visioning >=20 > From what I understand > AppName maps to cluster name > Service maps to instances=2C there can be multiple service components in = an > application. For example=2C in distributed data system=2C there may be a > storage layer and router layer. >=20 > The min/max containers seems to be at application level but it probably > should be at a Service scope ? >=20 > what do you think ? >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > On Tue=2C Jan 14=2C 2014 at 11:27 AM=2C kishore g w= rote: >=20 > > Hi=2C > > > > Here is a prototype of supporting the concept of provisioning in Helix. > > This was the work done by one of the interns at LinkedIn. You can find = more > > info here > > > > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-= helix-and-apache-yarn > > > > The implementation done by intern was outside of Helix. Kanak and I too= k a > > stab at it to treat provisioning as a first class citizen of Helix. > > > > The work is in helix-provisioning branch > > > > > > https://git-wip-us.apache.org/repos/asf?p=3Dhelix.git=3Ba=3Dtree=3Bh=3D= refs/heads/helix-provisioning=3Bhb=3Dhelix-provisioning > > > > Most of the code is here=2C > > > > > > https://git-wip-us.apache.org/repos/asf?p=3Dhelix.git=3Ba=3Dtree=3Bf=3D= helix-core/src/main/java/org/apache/helix/controller/provisioner=3Bh=3D2a26= 50b053206ac4c521bf71899dc286cd0ef961=3Bhb=3Dhelix-provisioning > > > > Here is a YARN specific implementation > > > > > > https://git-wip-us.apache.org/repos/asf?p=3Dhelix.git=3Ba=3Dtree=3Bf=3D= helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn=3Bh=3D9= 3d08917feb88e06140b506eff8f498fcb588fca=3Bhb=3Dhelix-provisioning > > > > Its still a WIP and main goal was to see if we can abstract Resource > > Managers like YARN=2C Mesos=2C EC2 from the application. > > > > I am working on providing the steps to try out launching a Helix cluste= r > > on YARN. > > > > Feedback/Thoughts? Any interest in contributing to make this production > > ready ? > > > > thanks=2C > > Kishore G > > > > > > > > > > > > > > > > = --_05231be6-b63a-45ee-9648-3c8ae4ab7ffa_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Maybe it should be at both scope= s? Sort of like putting bounds cross-app and cross-service within an app. T= his allows full control over an organization'=3Bs physical resources.
> Date:=3B Sat=2C 18 Jan 2014 09:=3B28:=3B32 -0800
> Subje= ct:=3B Re:=3B helix provisioning
> From:=3B g.kishore@=3Bgma= il.com
> To:=3B dev@=3Bhelix.apache.org;=3B user@=3Bhelix.ap= ache.org
>
> Thanks Swaroop for adding interfaces to specify the app= lication spec.
>
> https:=3B//git-wip-us.apache.org/repos/asf?= =3Bp==3Bhelix.git;=3Ba==3Bblob;=3Bf==3Bhelix-provisioning/sr= c/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;=3Bh= ==3B16b23fa02ffbe40c18b9cff286eb5eb321947d84;=3Bhb==3Bhelix-provis= ioning
>
> From what I understand
> AppName maps to cluster name<= br>> Service maps to instances=2C there can be multiple service components = in an
> application. For example=2C in distributed data system=2C there = may be a
> storage layer and router layer.
>
> The min/max contai= ners seems to be at application level but it probably
> should be at a S= ervice scope ?=3B
>
> what do you think ?=3B
>
>
> =
>
>
>
>
>
>
>
> On Tue=2C Jan 14=2C 2014 at= 11:=3B27 AM=2C kishore g <=3Bg.kishore@=3Bgmail.com>=3B wrote&= #58=3B
>
> >=3B Hi=2C
> >=3B
> >=3B Here is a prototy= pe of supporting the concept of provisioning in Helix.
> >=3B This wa= s the work done by one of the interns at LinkedIn. You can find more
> &= #62=3B info here
> >=3B
> >=3B http:=3B//engineering.linkedi= n.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
> = 2=3B
> >=3B The implementation done by intern was outside of Helix. K= anak and I took a
> >=3B stab at it to treat provisioning as a first = class citizen of Helix.
> >=3B
> >=3B The work is in helix-prov= isioning branch
> >=3B
> >=3B
> >=3B https:=3B//git-wi= p-us.apache.org/repos/asf?=3Bp==3Bhelix.git;=3Ba==3Btree;=3B= h==3Brefs/heads/helix-provisioning;=3Bhb==3Bhelix-provisioning
= > >=3B
> >=3B Most of the code is here=2C
> >=3B
> >= =3B
> >=3B https:=3B//git-wip-us.apache.org/repos/asf?=3Bp== =3Bhelix.git;=3Ba==3Btree;=3Bf==3Bhelix-core/src/main/java/org/= apache/helix/controller/provisioner;=3Bh==3B2a2650b053206ac4c521bf718= 99dc286cd0ef961;=3Bhb==3Bhelix-provisioning
> >=3B
> >=3B= Here is a YARN specific implementation
> >=3B
> >=3B
> >= =3B https:=3B//git-wip-us.apache.org/repos/asf?=3Bp==3Bhelix.git&#= 59=3Ba==3Btree;=3Bf==3Bhelix-provisioning/src/main/java/org/apache= /helix/provisioning/yarn;=3Bh==3B93d08917feb88e06140b506eff8f498fcb58= 8fca;=3Bhb==3Bhelix-provisioning
> >=3B
> >=3B Its still = a WIP and main goal was to see if we can abstract Resource
> >=3B Man= agers like YARN=2C Mesos=2C EC2 from the application.
> >=3B
> = 2=3B I am working on providing the steps to try out launching a Helix clust= er
> >=3B on YARN.
> >=3B
> >=3B Feedback/Thoughts?=3B= Any interest in contributing to make this production
> >=3B ready &#= 63=3B
> >=3B
> >=3B thanks=2C
> >=3B Kishore G
> >= =3B
> >=3B
> >=3B
> >=3B
> >=3B
> >=3B
> &= #62=3B
> >=3B
= --_05231be6-b63a-45ee-9648-3c8ae4ab7ffa_--