Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 695E11115F for ; Sun, 13 Apr 2014 04:37:32 +0000 (UTC) Received: (qmail 54512 invoked by uid 500); 13 Apr 2014 04:37:31 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 54299 invoked by uid 500); 13 Apr 2014 04:37:29 -0000 Mailing-List: contact dev-help@stratos.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.incubator.apache.org Delivered-To: mailing list dev@stratos.incubator.apache.org Received: (qmail 54290 invoked by uid 99); 13 Apr 2014 04:37:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Apr 2014 04:37:28 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of thilinapiy@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Apr 2014 04:37:23 +0000 Received: by mail-wi0-f182.google.com with SMTP id d1so2676702wiv.9 for ; Sat, 12 Apr 2014 21:37:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=rKJi/oC9fRnQ3pTtAVcvsL9Brm2WUp5k1d1jZo2460c=; b=nYJ6T2p1zBadiB2VYdj5lemK25z9SJs0bePi+SqGH6/ySOZOTNbXSFhjX4J+Gbo98B KYoeV6hZtrVX18pqcD2/dwvsxhXYCXmgupmWb3RFB32p6XXyuQ0XTa/PaQE6hsp7r4fT MkuwmTveiFAjf6ZgyLhhmy+4GtKsMqKnELPwvtihPvQcIJGp1o+Frm+LUqm7RCpAefZh M6sxvHgWQ3noia3RTvGKppQCcIqUr1pQUBQwbFRHSf0oxlwF2uOZmK1abPTPkDe2HfQr Pf5YqAuWb2WVZ0VGE/raudnfsLk0yL10vHBzFPvTqrReGLSQVgRh60fWeBPeRnhmHaNo 75Og== MIME-Version: 1.0 X-Received: by 10.194.186.140 with SMTP id fk12mr97359wjc.47.1397363822421; Sat, 12 Apr 2014 21:37:02 -0700 (PDT) Received: by 10.216.61.199 with HTTP; Sat, 12 Apr 2014 21:37:02 -0700 (PDT) In-Reply-To: References: Date: Sun, 13 Apr 2014 10:07:02 +0530 Message-ID: Subject: Re: Create a base Puppet module From: Thilina Piyasundara To: dev@stratos.incubator.apache.org Content-Type: multipart/alternative; boundary=047d7bea451841e9ae04f6e51eb3 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bea451841e9ae04f6e51eb3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable IMO it is better to use a base class which contains all the basic things like wget, unzip, etc. Create a switch in the init.pp file then you can select what to install and what not to. Eg :- stratos-base =E2=94=9C=E2=94=80=E2=94=80 files =E2=94=9C=E2=94=80=E2=94=80 LICENSE =E2=94=9C=E2=94=80=E2=94=80 manifests =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 environment.pp =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 hosts.pp =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 init.pp =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 java.pp =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 maven.pp =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 packages.pp =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 params.pp =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 users.pp =E2=94=9C=E2=94=80=E2=94=80 metadata.json =E2=94=9C=E2=94=80=E2=94=80 Modulefile =E2=94=9C=E2=94=80=E2=94=80 README =E2=94=9C=E2=94=80=E2=94=80 spec =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 spec_helper.rb |=E2=94=80=E2=94=80 templates =E2=94=9C=E2=94=80=E2=94=80 environment.erb =E2=94=94=E2=94=80=E2=94=80 hosts.erb in nodes.pp node base { class { 'stratos-base': java =3D> true, php =3D> false, packages =3D> [ "wget", "htop", "tree" ], } } node sc.node inherits base { } But it's always better if you can use pre-setuped AMIs or images to reduce puppeting time in the bootup process. On Sun, Apr 13, 2014 at 9:50 AM, Akila Ravihansa Perera wrote: > Hi Nirmal, > > I think I may have not clearly explained the reasons behind the suggested > structure. So I'll list down some of the problems we have in the current > cartridge creation process. These are not major issues but would be nice = to > make it cleaner before the release :-) > > As for the installation of Puppet master/agent, we don't need zip, unzip > or wget (or anything). The puppetinstall script just do a simple "apt-get > install puppet/puppetmaster". We need wget just to download this > puppetinstall and config shell scripts from Stratos Git repo. We can > transfer those scripts to the instance via ssh from another machine. But > still, Puppet does *not* install wget anyway, it just *assumes* wget is > already installed (we only need that to download installation scripts fro= m > Stratos Git repo). Usually wget is already present in almost every > distribution. > > 1. zip/unzip has to be installed *manually* in the current workflow [1] > But this can be done via Puppet. It makes no sense to let users install > zip/unzip manually when we're using Puppet. > > 2. While I was trying to include zip/unzip package to a Puppet module, I > couldn't find a base *class* to add this. I know we have a base *node* bu= t > the official Puppet documentation says we should not include package > declarations in node blocks (nodes.pp). > > 3. Therefore we need a base class to include all those common tools. We > can consider Stratos agent, zip, unzip, tar and Java as common tools. I > believe every cartridge instance must have those? > > 4. Currently we don't have a base *class* to include those packages. I > could include that to Java class since it is being required by all other > *nodes* but it will be a very ugly dependency. > > 5. Actually I tried doing that (and it works too), but when I try to > create a php cartridge it throws this "package already defined error". > Further investigation revealed that php class has included "zip/unzip" > package, hence it doesn't allow any other class to include that > package....that doesn't make any sense to include zip/unzip only in php > class. > > 5. In each *node* declaration, we have this "require java" and "class > {'agent'}" statements. If we are using inheritance, we don't need this in > every node, IMO. We could move that to base node block. > > If you're convinced this needs to be re-factored, I could work on a patch= . > Otherwise, apologies for the noise. > > [1] - > https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Creating+a+Cart= ridge+Image+on+EC2+or+OpenStack > > > Thanks. > > > On Sat, Apr 12, 2014 at 6:47 PM, Nirmal Fernando = wrote: > >> We already have a 'base' node AFAIS and all the other nodes inherited >> from that. >> >> >> On Sat, Apr 12, 2014 at 1:52 PM, Akila Ravihansa Perera < >> ravihansa@wso2.com> wrote: >> >>> >>> On Sat, Apr 12, 2014 at 1:02 PM, Nirmal Fernando >> > wrote: >>> >>>> >>>> >>>> >>>> On Sat, Apr 12, 2014 at 12:49 PM, Akila Ravihansa Perera < >>>> ravihansa@wso2.com> wrote: >>>> >>>>> Hi Nirmal, >>>>> >>>>> We don't need zip/unzip to install Puppet. Actually I managed to >>>>> install a default node without zip/unzip by adding package { 'unzip': >>>>> ensure =3D> installed } to Puppet Java module. >>>>> >>>> >>>> Well, we still need wget? :-) >>>> >>> >>> Not really...we need wget just to download installation scripts. This >>> can be done via ssh :-) >>> >> >> SSH from where? >> >>> >>>>> What I'm trying to address is the redundant code in current Puppet >>>>> master module structure. Currently we have "require java", class >>>>> {'agent':} on each module, which can be avoided by creating a base mo= dule >>>>> and other modules can inherit stuff from this. >>>>> >>>>> For eg. php module has package { 'zip': ensure =3D> installed } >>>>> dependency, and also Java module need this package as well. Instead o= f >>>>> putting these common tools in individual classes we can create a base= class >>>>> and others will inherit stuff from this. >>>>> >>>> >>>> :-) if you check the nodes.pp file, we already leverage puppet >>>> inheritance. >>>> >>> >>> Yes, but Puppet classes do not :-) >>> >>>> >>>>> If you're interested, I can create a patch for this to be reviewed. >>>>> >>>> >>>> It's better if you test it fully for all the existing scenarios >>>> (judgemental) and I think these improvements can go for 4.1 but not fo= r >>>> 4.0, since we're wrapping up 4.0 and this is not a blocker. >>>> >>> >>> Absolutely :-) >>> >>> How about we add zip/unzip packages + Stratos agent class declaration >>> (common tools) to a base *class* and have other *classes* inherit from = that? >>> >>>> >>>>> >>>>> On Sat, Apr 12, 2014 at 12:17 PM, Nirmal Fernando < >>>>> nirmal070125@gmail.com> wrote: >>>>> >>>>>> Hi Akila, >>>>>> >>>>>> If you have a look at the current nodes.pp file, Dinesh have already >>>>>> added a default section, which will be used to create a base image w= ith >>>>>> minimal software (without wget, zip). >>>>>> >>>>>> What problems you are trying to address here? Is it to avoid the >>>>>> redundant code in nodes.pp file? Cause to install puppet you need th= ese >>>>>> wget etc. packages, hence no point of adding it in the default node. >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Apr 12, 2014 at 11:15 AM, Akila Ravihansa Perera < >>>>>> ravihansa@wso2.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I was thinking that we could simplify cartridge creation process by >>>>>>> creating a base Puppet module which would install common tools like= zip, >>>>>>> unzip, tar, wget and also include Java module in this base module. >>>>>>> >>>>>>> Basically, the module structure will be like this (not tested); >>>>>>> >>>>>>> node /base/ { >>>>>>> >>>>>>> >>>>>>> >>>>>>> } >>>>>>> >>>>>>> >>>>>>> class base { >>>>>>> >>>>>>> package { 'unzip': ensure =3D> installed } >>>>>>> package { 'tar': ensure =3D> installed } >>>>>>> package { 'wget': ensure =3D> installed } >>>>>>> >>>>>>> class {'java'} >>>>>>> class {'agent':} >>>>>>> >>>>>>> Class['java'] ~> Class['agent'] >>>>>>> >>>>>>> } >>>>>>> >>>>>>> >>>>>>> Then other Stratos module can include this >>>>>>> >>>>>>> node /lb/ inherits base { >>>>>>> class {'lb': maintenance_mode =3D> 'norestart',} >>>>>>> } >>>>>>> >>>>>>> This will remove lot of redundancy and ugly dependencies we have >>>>>>> now....WDYT? >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Akila Ravihansa Perera >>>>>>> Software Engineer >>>>>>> WSO2 Inc. >>>>>>> http://wso2.com >>>>>>> >>>>>>> Phone: +94 77 64 154 38 >>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Nirmal >>>>>> >>>>>> Nirmal Fernando. >>>>>> PPMC Member & Committer of Apache Stratos, >>>>>> Senior Software Engineer, WSO2 Inc. >>>>>> >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Akila Ravihansa Perera >>>>> Software Engineer >>>>> WSO2 Inc. >>>>> http://wso2.com >>>>> >>>>> Phone: +94 77 64 154 38 >>>>> Blog: http://ravihansa3000.blogspot.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Nirmal >>>> >>>> Nirmal Fernando. >>>> PPMC Member & Committer of Apache Stratos, >>>> Senior Software Engineer, WSO2 Inc. >>>> >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> Akila Ravihansa Perera >>> Software Engineer >>> WSO2 Inc. >>> http://wso2.com >>> >>> Phone: +94 77 64 154 38 >>> Blog: http://ravihansa3000.blogspot.com >>> >> >> >> >> -- >> Best Regards, >> Nirmal >> >> Nirmal Fernando. >> PPMC Member & Committer of Apache Stratos, >> Senior Software Engineer, WSO2 Inc. >> >> Blog: http://nirmalfdo.blogspot.com/ >> > > > > -- > Akila Ravihansa Perera > Software Engineer > WSO2 Inc. > http://wso2.com > > Phone: +94 77 64 154 38 > Blog: http://ravihansa3000.blogspot.com > --=20 Thanks and regards, Thilina Piyasundara. --047d7bea451841e9ae04f6e51eb3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
IMO it is better to use a base class which contains all th= e basic things like wget, unzip, etc. Create a switch in the init.pp file t= hen you can select what to install and what not to.

Eg :-

stratos-base
=E2=94=9C=E2=94=80=E2=94=80 files
=E2=94=9C=E2=94=80=E2=94=80 LICENSE
=E2=94=9C=E2=94=80=E2= =94=80 manifests
=E2=94=82 =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 env= ironment.pp
=E2=94=82 =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 hosts.pp=
=E2=94=82 =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 init.pp
= =E2=94=82 =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 java.pp
=E2=94=82 = =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 maven.pp
=E2=94=82 =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 packages.pp
=E2= =94=82 =C2=A0 =E2=94=9C=E2=94=80=E2=94=80 params.pp
=E2=94=82 =C2= =A0 =E2=94=94=E2=94=80=E2=94=80 users.pp
=E2=94=9C=E2=94=80=E2=94= =80 metadata.json
=E2=94=9C=E2=94=80=E2=94=80 Modulefile
=E2=94=9C=E2=94=80=E2=94=80 README
=E2=94=9C=E2=94=80=E2=94=80 = spec
=E2=94=82 =C2=A0 =E2=94=94=E2=94=80=E2=94=80 spec_helper.rb<= /div>
=C2=A0|=E2=94=80=E2=94=80 templates
=C2=A0 =C2=A0=E2=94=9C=E2=94=80=E2=94=80 environment.erb
=C2= =A0 =C2=A0=E2=94=94=E2=94=80=E2=94=80 hosts.erb

<= /div>

in nodes.pp=C2=A0

node base {
<your configs>

=C2=A0 class { &#= 39;stratos-base':
=C2=A0 =C2=A0 java =3D> true,
=C2=A0 =C2=A0 php =3D> false,=C2=A0 =C2=A0 packages =3D> [ "wget", "htop", &quo= t;tree" ],

=C2=A0 =C2=A0}=C2=A0
}=C2=A0


node sc.node inherits base {
=C2=A0
<your node = configs>

}


But it&= #39;s always better if you can use pre-setuped AMIs or images to reduce pup= peting time in the bootup process.



On Sun, Apr 13, 2014 at 9:50 AM, Akila Ravihansa Perera <ravihansa@wso= 2.com> wrote:
Hi Nirmal,

I think I may have not clearly explained the reasons behind the suggested s= tructure. So I'll list down some of the problems we have in the current= cartridge creation process. These are not major issues but would be nice t= o make it cleaner before the release :-)

As for the installatio= n of Puppet master/agent, we don't need zip, unzip or wget (or anything= ). The puppetinstall script just do a simple "apt-get install puppet/p= uppetmaster". We need wget just to download this puppetinstall and con= fig shell scripts from Stratos Git repo. We can transfer those scripts to t= he instance via ssh from another machine. But still, Puppet does *not* inst= all wget anyway, it just *assumes* wget is already installed (we only need = that to download installation scripts from Stratos Git repo). Usually wget = is already present in almost every distribution.

1. zip/unzip has to be= installed *manually* in the current workflow [1]
But this can be = done via Puppet. It makes no sense to let users install zip/unzip manually = when we're using Puppet.

2. Whi= le I was trying to include zip/unzip package to a Puppet module, I couldn&#= 39;t find a base *class* to add this. I know we have a base *node* but the = official Puppet documentation says we should not include package declaratio= ns in node blocks (nodes.pp).=C2=A0

3. Therefore we need a= base class to include all those common tools. We can consider Stratos agen= t, zip, unzip, tar and Java as common tools. I believe every cartridge inst= ance must have those?

4. Currently we don= 9;t have a base *class* to include those packages. I could include that to = Java class since it is being required by all other *nodes* but it will be a= very ugly dependency.

5. Actually I tried do= ing that (and it works too), but when I try to create a php cartridge it th= rows this "package already defined error". Further investigation = revealed that php class has included "zip/unzip" package, hence i= t doesn't allow any other class to include that package....that doesn&#= 39;t make any sense to include zip/unzip only in php class.

5. In each *node* decl= aration, we have this "require java" and "class {'agent&= #39;}" statements. If we are using inheritance, we don't need this= in every node, IMO. We could move that to base node block.

If you're convince= d this needs to be re-factored, I could work on a patch. Otherwise, apologi= es for the noise.


=

Thanks.


On Sat, Apr 12, 2014 at 6:47 = PM, Nirmal Fernando <nirmal070125@gmail.com> wrote:
We already have a 'base= ' node AFAIS and all the other nodes inherited from that.


On Sat, Apr 12, 2014 at 1:52 PM, Ak= ila Ravihansa Perera <ravihansa@wso2.com> wrote:
=
On Sat, Apr 12, 2014 at 1:02 PM, Nirmal= Fernando <nirmal070125@gmail.com> wrote:



On Sat, Apr 12, 2014 at 12:49 P= M, Akila Ravihansa Perera <ravihansa@wso2.com> wrote:
Hi Nirma= l,

We don't need zip/unzip to install Puppet. Actual= ly I managed to install a default node without zip/unzip by adding=C2=A0package { 'unz= ip': ensure =3D> installed } to Puppet Java module.

Well, we still need wget? :-) =

Not real= ly...we need wget just to download installation scripts. This can be done v= ia ssh :-)=C2=A0

SSH from where?
=

What I'm= trying to address is the redundant code in current Puppet master module st= ructure. Currently we have "require java",=C2=A0class {'agent':} = on each module, which can be avoided by creating a base module and other mo= dules can inherit stuff from this.

For= eg. php module has=C2=A0=C2=A0package { 'zip': ensure =3D> installed } = dependency, and also Java module need this package as well. Instead of putt= ing these common tools in individual classes we can create a base class and= others will inherit stuff from this.

:-) if you check the nodes.pp = file, we already leverage puppet inheritance.
<= /blockquote>

Yes, but Puppet classes do not :-) = =C2=A0
=

If = you're interested, I can create a patch for this to be reviewed.=

It's better if you test it= fully for all the existing scenarios (judgemental) and I think these impro= vements can go for 4.1 but not for 4.0, since we're wrapping up 4.0 and= this is not a blocker.

Absolutely := -) =C2=A0

How about we add zip/unzip packages + St= ratos agent class declaration (common tools) to a base *class* and have oth= er *classes* inherit from that?
=


On Sat, Apr 12, 2014 at 12:17 PM, Nirmal Fernando <= nirmal070125@gm= ail.com> wrote:
Hi Akila,

If you have a look at the current nodes.pp file, D= inesh have already added a default section, which will be used to create a = base image with minimal software (without wget, zip).

What problems you are trying to address here? Is it to avoid the = redundant code in nodes.pp file? Cause to install puppet you need these wge= t etc. packages, hence no point of adding it in the default node.



On Sat, Apr 12, 2014 at 11:15 AM, Akila Ravihansa Perera <ra= vihansa@wso2.com> wrote:
Hi,
=
I was thinking that we could simplify cartridge creation pro= cess by creating a base Puppet module which would install common tools like= zip, unzip, tar, wget and also include Java module in this base module.

Basically, the module structure will be like this (not = tested);

node /base/ {

= =C2=A0 =C2=A0 <declarations here>
=C2=A0=C2=A0
}<= /div>


class base {

=C2=A0 =C2=A0pack= age { 'unzip': ensure =3D> installed }
=C2=A0 =C2= =A0package { 'tar': ensure =3D> installed }
=C2=A0= =C2=A0package { 'wget': ensure =3D> installed }

=C2=A0 =C2=A0class {'java'}
=C2= =A0 =C2=A0class {'agent':}
=C2=A0 =C2=A0
=C2=A0= =C2=A0Class['java'] ~> Class['agent'] =C2=A0
=
}


Then other Stratos module can include this

node /lb/ inherits base { =C2=A0=C2=A0
=C2=A0 class = {'lb': maintenance_mode =C2=A0 =3D> 'norestart',}
<= div>}

This will remove lot of redundancy and ugly dependencies we have= now....WDYT?


--
Akila Ravihansa P= erera
Software Engineer
WSO2 Inc.
http://wso2.com

Phone:=C2=A0+94 77 64 154 38=
Blog:=C2=A0http://ravihansa3000.blogspot.com



--
Best Regards,
Nirmal

Nirmal = Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.




--
Akila Ravihansa Perera
Software Engineer
WSO2 = Inc.
http= ://wso2.com

Phone:=C2=A0+94 77 64 154 3= 8
Blog:=C2=A0http://ravihansa3000.blogspot.com


=
--
Best Regards,
Nirmal

Nirmal Fernando.=
PPMC Member & Committer of Apache Stratos,
Senior Software Engin= eer, WSO2 Inc.




--
Akila Ravihansa Pe= rera
Software Engineer
WSO2 Inc.
http= ://wso2.com

Phone:=C2=A0+94 77 64 154 3= 8
Blog:=C2=A0http://ravihansa3000.blogspot.com



--
<= div dir=3D"ltr">Best Regards,
Nirmal

Nirmal Fernando.
PPMC Mem= ber & Committer of Apache Stratos,
Senior Software Engineer, WSO2 In= c.




--
Akila Ravihansa Perera
Software Engineer
WSO2 = Inc.
http= ://wso2.com

Phone:=C2=A0+94 77 64 154 3= 8
Blog:=C2=A0http://ravihansa3000.blogspot.com



--
= Thanks and regards,
Thilina Piyasund= ara.
 --047d7bea451841e9ae04f6e51eb3--