apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Apache Apex as component in Apache bigtop
Date Tue, 16 Feb 2016 16:22:43 GMT
Please follow the recommendation and continue discussion on the Bigtop list.

--
sent from mobile
On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <chinmay@apache.org> wrote:

> I got a recommendation from bigtop community to drop "-incubating" from
> version.
>
> Here is the mail thread:
>
> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E
>
> Please let me know if removing "-incubating" is fine.
>
> Thanks,
> Chinmay.
>
>
> On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
>
> > I've already started communication there.
> >
> > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <thomas@datatorrent.com>
> > wrote:
> >
> >> This is a question you should ask on the Bigtop mailing list.
> >>
> >> --
> >> sent from mobile
> >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <prad@apache.org> wrote:
> >>
> >> > Yes. In case of RPM, the only restriction on version is that it can
> not
> >> > contain '-'.
> >> > However, as a workaround, one can use 'incubating' as a part of
> >> %{release}.
> >> >
> >> > Version: 3.3.0
> >> > Release: incubating
> >> >
> >> > apex-3.3.0-incubating.noarch.rpm
> >> >
> >> > It is the same practice followed for pre-release packages e.g. with
> rcX,
> >> > alphaX, betaX being used in %{release}.
> >> >
> >> > Regards,
> >> > Pradeep A. Dalvi
> >> >
> >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> >> > wrote:
> >> >
> >> > > Hi Everyone,
> >> > >
> >> > > In one of the previous mails in this thread, we decided that version
> >> of
> >> > the
> >> > > package will be "3.3.0-incubating".
> >> > > The problem that I'm facing is in RPM base version "-" is not
> allowed.
> >> > >
> >> > > Can we have package version to "3.3.0"?
> >> > > i.e. the packages created will be as follows:
> >> > > For RPM: apex-3.3.0-1.noarch.rpm
> >> > > For DEB: apex_3.3.0-1_all.deb
> >> > >
> >> > > There is another integration for Zepplin in Bigtop which is
> incubating
> >> > and
> >> > > is having base version as "0.5.6".
> >> > >
> >> > > Please let me know your thoughts.
> >> > >
> >> > > Thanks,
> >> > > Chinmay.
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar <
> >> chinmay@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Thanks everyone for the feedback.
> >> > > >
> >> > > > Also as Ram and David mentioned, I'll start creating the man page
> as
> >> > > well.
> >> > > >
> >> > > > Thanks,
> >> > > > Chinmay.
> >> > > >
> >> > > >
> >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise <
> >> thomas@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > >> I think we already agreed that we will retain the current script
> >> name
> >> > in
> >> > > >> this bigtop iteration. There is a ticket for renaming, add you
> >> > > suggestions
> >> > > >> there.
> >> > > >>
> >> > > >> --
> >> > > >> sent from mobile
> >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" <
> sandesh@datatorrent.com>
> >> > > wrote:
> >> > > >>
> >> > > >> > +1 for Symlink from apex to dtcli or start the transition
> >> process by
> >> > > >> making
> >> > > >> > dtcli as the symlink to apex cli.
> >> > > >> >
> >> > > >> > Renaming is a breaking change should be done during major
> version
> >> > > number
> >> > > >> > change.
> >> > > >> >
> >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan <
> david@datatorrent.com
> >> >
> >> > > >> wrote:
> >> > > >> >
> >> > > >> > > Apologies for hitting the "Send" button too early.
> >> > > >> > >
> >> > > >> > > What is the scope of the name change?  If we simply have a
> >> symlink
> >> > > >> from
> >> > > >> > > apex to dtcli, then it should have no backward compatibility
> >> > problem
> >> > > >> at
> >> > > >> > > all.
> >> > > >> > >
> >> > > >> > > But the proper name change would involve:
> >> > > >> > > - Change the name of DTCli class to ApexCli
> >> > > >> > > - Change the prompt from dt> to apex>
> >> > > >> > > - Change the initial greeting when running the CLI from
> >> something
> >> > > like
> >> > > >> > "DT
> >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >> > > >> > >
> >> > > >> > > The above changes may break existing scripts that rely on the
> >> the
> >> > > >> string
> >> > > >> > > "DT".
> >> > > >> > > But if folks are okay with this potential problem, I am okay
> >> with
> >> > > the
> >> > > >> > name
> >> > > >> > > change.
> >> > > >> > >
> >> > > >> > > For the man page, we can start with the content of this page:
> >> > > >> > >
> >> > > >> > > http://docs.datatorrent.com/dtcli/
> >> > > >> > >
> >> > > >> > > Plus the command line options when doing dtcli -h:
> >> > > >> > >
> >> > > >> > > usage: DTCli
> >> > > >> > >  -e <arg>    Commands are read from the argument
> >> > > >> > >  -f <arg>    Use the specified prompt at all time
> >> > > >> > >  -h          Print this help
> >> > > >> > >  -kp <arg>   Use the specified kerberos principal
> >> > > >> > >  -kt <arg>   Use the specified kerberos keytab
> >> > > >> > >  -p <arg>    JSONP padding function
> >> > > >> > >  -r          JSON Raw mode
> >> > > >> > >  -v          Verbose mode level 1
> >> > > >> > >  -vv         Verbose mode level 2
> >> > > >> > >  -vvv        Verbose mode level 3
> >> > > >> > >  -vvvv       Verbose mode level 4
> >> > > >> > >
> >> > > >> > > David
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan <
> >> david@datatorrent.com
> >> > >
> >> > > >> > wrote:
> >> > > >> > >
> >> > > >> > > > What is the scope of the name change?  If we simply have a
> >> > symlink
> >> > > >> from
> >> > > >> > > > apex to dtcli, then it should have no backward
> compatibility
> >> > > >> problem at
> >> > > >> > > all.
> >> > > >> > > >
> >> > > >> > > > But the proper name change would involve:
> >> > > >> > > > - Change the name of DTCli class to ApexCli
> >> > > >> > > > - Change the prompt from dt> to apex>
> >> > > >> > > > - Change the initial greeting when running the CLI from
> >> > something
> >> > > >> like
> >> > > >> > > "DT
> >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...".
> >> > > >> > > >
> >> > > >> > > > The above changes may break existing scripts that rely on
> the
> >> > the
> >> > > >> > string
> >> > > >> > > > "DT".
> >> > > >> > > > But if folks are okay with this potential problem, I am
> okay
> >> > with
> >> > > >> the
> >> > > >> > > name
> >> > > >> > > > change.
> >> > > >> > > >
> >> > > >> > > > For the man page, we can start with the content of this
> page:
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <
> >> > > >> chinmay@apache.org
> >> > > >> > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > >> Here is a suggestion I have related to name change:
> >> > > >> > > >>
> >> > > >> > > >> 1. While packaging we change the name of "dtcli" script to
> >> > > "apex".
> >> > > >> > This
> >> > > >> > > is
> >> > > >> > > >> done so that in future when we change the name to "apex",
> >> users
> >> > > of
> >> > > >> > > bigtop
> >> > > >> > > >> apex don't have to transition much.
> >> > > >> > > >> 2. We also keep a symlink named dtcli which points to apex
> >> > > script.
> >> > > >> > This
> >> > > >> > > is
> >> > > >> > > >> for backward compatibility.
> >> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration in
> >> > bigtop.
> >> > > >> This
> >> > > >> > is
> >> > > >> > > >> in
> >> > > >> > > >> the interest of not changing the content of dtcli or apex
> >> > script.
> >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming related
> >> > > changes
> >> > > >> and
> >> > > >> > > >> update bigtop repository later to remove references to
> dtcli
> >> > all
> >> > > >> > > together.
> >> > > >> > > >>
> >> > > >> > > >> Please share your thoughts on above approach.
> >> > > >> > > >>
> >> > > >> > > >> Also, please share what could be the man page content for
> >> > "apex".
> >> > > >> > > >>
> >> > > >> > > >> Thanks,
> >> > > >> > > >> Chinmay.
> >> > > >> > > >>
> >> > > >> > > >>
> >> > > >> > > >>
> >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <
> >> > > >> > chinmay@apache.org>
> >> > > >> > > >> wrote:
> >> > > >> > > >>
> >> > > >> > > >> > Yes.. It'll be build from source tar downloaded from one
> >> of
> >> > the
> >> > > >> > apache
> >> > > >> > > >> > mirror: apache.osuosl.org
> >> > > >> > > >> >
> >> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli.
> This
> >> > > leaves
> >> > > >> us
> >> > > >> > 3
> >> > > >> > > >> > options:
> >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would need a
> >> > change
> >> > > >> while
> >> > > >> > > >> > packaging.
> >> > > >> > > >> >
> >> > > >> > > >> > But if we want to use dtcli as it is, then we would need
> >> > > >> dt-env.sh
> >> > > >> > > >> >
> >> > > >> > > >> >
> >> > > >> > > >> >
> >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <
> >> > > >> > thomas@datatorrent.com
> >> > > >> > > >
> >> > > >> > > >> > wrote:
> >> > > >> > > >> >
> >> > > >> > > >> >> Looks good overall, though there shouldn't be any
> >> dt-env.sh
> >> > > >> > > >> >>
> >> > > >> > > >> >> What will the bigtop package be built from, the source
> >> tar?
> >> > > >> > > >> >>
> >> > > >> > > >> >>
> >> > > >> > > >> >>
> >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> >> > > >> > > chinmay@apache.org
> >> > > >> > > >> >
> >> > > >> > > >> >> wrote:
> >> > > >> > > >> >>
> >> > > >> > > >> >> > Hi Thomas,
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Thanks for the feedback.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > First, we're not changing the name anywhere. We'll
> >> follow
> >> > > what
> >> > > >> > > >> >> currently is
> >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Secondly, I've mentioned a directory structure below
> >> which
> >> > > is
> >> > > >> > > inline
> >> > > >> > > >> >> with
> >> > > >> > > >> >> > other existing integrations with bigtop.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > The need for each file is as follows:
> >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from
> >> > > >> engine/src/main/scripts/
> >> > > >> > > of
> >> > > >> > > >> >> source
> >> > > >> > > >> >> > code apex 3.3.0-incubating.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This
> >> > > searches
> >> > > >> for
> >> > > >> > > >> hadoop
> >> > > >> > > >> >> > binary path and exports an env variable for dtcli to
> >> use
> >> > it.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted
> >> in a
> >> > > >> similar
> >> > > >> > > >> way to
> >> > > >> > > >> >> > how DT Community Edition finds dependency jars.
> >> > > >> > > >> >> > Please note that this is the first iteration list of
> >> jars.
> >> > > I'm
> >> > > >> > > >> trying to
> >> > > >> > > >> >> > narrow this down to only those which are really
> >> required.
> >> > > >> > > >> >> > The test that I'm running to check if dtcli runs fine
> >> with
> >> > > >> given
> >> > > >> > > >> >> dependency
> >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Above is the directory structure which is required
> for
> >> CLI
> >> > > to
> >> > > >> > work.
> >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some
> >> > additional
> >> > > >> > files
> >> > > >> > > >> which
> >> > > >> > > >> >> > makes the like of administrator easier. For eg.
> >> /etc/skel
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Please let me know if this is inline with what you're
> >> > > >> thinking.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > Thanks,
> >> > > >> > > >> >> > Chinmay.
> >> > > >> > > >> >> >
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > .
> >> > > >> > > >> >> > |-- bin
> >> > > >> > > >> >> > |   `-- dtcli
> >> > > >> > > >> >> > |-- conf
> >> > > >> > > >> >> > |   `-- dt-env.sh
> >> > > >> > > >> >> > `-- lib
> >> > > >> > > >> >> >     |-- ant-1.9.2.jar
> >> > > >> > > >> >> >     |-- ant-launcher-1.9.2.jar
> >> > > >> > > >> >> >     |-- apex-api-3.3.0-incubating.jar
> >> > > >> > > >> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> >> > > >> > > >> >> >     |-- apex-common-3.3.0-incubating.jar
> >> > > >> > > >> >> >     |-- apex-engine.jar
> >> > > >> > > >> >> >     |-- apex-engine-tests.jar
> >> > > >> > > >> >> >     |-- async-http-client-1.7.20.jar
> >> > > >> > > >> >> >     |-- bval-core-0.5.jar
> >> > > >> > > >> >> >     |-- bval-jsr303-0.5.jar
> >> > > >> > > >> >> >     |-- commons-beanutils-1.8.3.jar
> >> > > >> > > >> >> >     |-- commons-codec-1.10.jar
> >> > > >> > > >> >> >     |-- commons-lang3-3.1.jar
> >> > > >> > > >> >> >     |-- commons-logging-1.1.3.jar
> >> > > >> > > >> >> >     |-- grizzly-http-servlet-2.1.2.jar
> >> > > >> > > >> >> >     |-- hadoop-common-2.2.0-tests.jar
> >> > > >> > > >> >> >     |-- httpclient-4.3.5.jar
> >> > > >> > > >> >> >     |-- httpcore-4.3.2.jar
> >> > > >> > > >> >> >     |-- jackson-core-asl-1.9.2.jar
> >> > > >> > > >> >> >     |-- jackson-mapper-asl-1.9.2.jar
> >> > > >> > > >> >> >     |-- javax.servlet-3.1.jar
> >> > > >> > > >> >> >     |-- javax.servlet-api-3.0.1.jar
> >> > > >> > > >> >> >     |-- jersey-apache-client4-1.9.jar
> >> > > >> > > >> >> >     |-- jersey-client-1.9.jar
> >> > > >> > > >> >> >     |-- jetty-http-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jetty-io-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jetty-util-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> >> > > >> > > >> >> >     |-- jline-2.11.jar
> >> > > >> > > >> >> >     |-- kryo-2.24.0.jar
> >> > > >> > > >> >> >     |-- mbassador-1.1.9.jar
> >> > > >> > > >> >> >     |-- minlog-1.2.jar
> >> > > >> > > >> >> >     |-- netlet-1.2.0.jar
> >> > > >> > > >> >> >     |-- netty-3.6.6.Final.jar
> >> > > >> > > >> >> >     |-- objenesis-2.1.jar
> >> > > >> > > >> >> >     |-- validation-api-1.1.0.Final.jar
> >> > > >> > > >> >> >     |-- xbean-asm5-shaded-4.3.jar
> >> > > >> > > >> >> >     `-- zip4j-1.3.2.jar
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > 3 directories, 40 files
> >> > > >> > > >> >> >
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> >> > > >> > > >> thomas@datatorrent.com>
> >> > > >> > > >> >> > wrote:
> >> > > >> > > >> >> >
> >> > > >> > > >> >> > > Chinmay,
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > Before discussing where to put the files, let's
> make
> >> > sure
> >> > > >> they
> >> > > >> > > are
> >> > > >> > > >> >> really
> >> > > >> > > >> >> > > needed for the operation of the CLI. As for names,
> >> > > anything
> >> > > >> new
> >> > > >> > > >> needs
> >> > > >> > > >> >> to
> >> > > >> > > >> >> > > reflect Apex in the name and should follow common
> >> > > >> conventions,
> >> > > >> > > >> >> especially
> >> > > >> > > >> >> > > in Bigtop where there are many existing
> integrations
> >> to
> >> > > look
> >> > > >> > at.
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > Thanks,
> >> > > >> > > >> >> > > Thomas
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar
> <
> >> > > >> > > >> >> > > chinmay@datatorrent.com>
> >> > > >> > > >> >> > > wrote:
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli
> >> > > >> considering it
> >> > > >> > > >> works
> >> > > >> > > >> >> > with
> >> > > >> > > >> >> > > DT
> >> > > >> > > >> >> > > > Community edition. We can keep the directory
> >> structure
> >> > > >> > similar
> >> > > >> > > to
> >> > > >> > > >> >> that.
> >> > > >> > > >> >> > > > dt-env.sh has variables which contains
> information
> >> > > >> required
> >> > > >> > for
> >> > > >> > > >> >> dtcli
> >> > > >> > > >> >> > to
> >> > > >> > > >> >> > > > launch.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > 2. Let me check with bigtop community that
> whether
> >> > they
> >> > > >> > > >> facilitate
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > > > installation of rpms/debs before user is created.
> >> In
> >> > > >> either
> >> > > >> > > case,
> >> > > >> > > >> >> > current
> >> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory.
> Also
> >> > > before
> >> > > >> > > putting
> >> > > >> > > >> >> > > anything
> >> > > >> > > >> >> > > > in /etc/skel we need to define what are the
> default
> >> > > >> contents
> >> > > >> > > that
> >> > > >> > > >> >> > should
> >> > > >> > > >> >> > > go
> >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults,
> >> probably we
> >> > > >> should
> >> > > >> > > not
> >> > > >> > > >> >> > > > explicitly add it in /etc/skel.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > 3.  /etc/profile.d approach looks nice. This way
> >> > > contents
> >> > > >> of
> >> > > >> > > >> >> dt-env.sh
> >> > > >> > > >> >> > > are
> >> > > >> > > >> >> > > > present as env variables. But I see a catch
> there.
> >> > > Adding
> >> > > >> > > >> dt-env.sh
> >> > > >> > > >> >> to
> >> > > >> > > >> >> > > > /etc/profile.d would make all the variables
> >> available
> >> > at
> >> > > >> > > runtime
> >> > > >> > > >> all
> >> > > >> > > >> >> > the
> >> > > >> > > >> >> > > > time. I feel a little skeptical about that.
> Maybe a
> >> > > >> possible
> >> > > >> > > >> >> collision
> >> > > >> > > >> >> > > can
> >> > > >> > > >> >> > > > occur with other application vars.
> >> > > >> > > >> >> > > > Moreover, current dtcli does source
> >> > "../conf/dt-env.sh"
> >> > > >> and
> >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway
> >> > > >> available.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening
> at
> >> > > Bigtop
> >> > > >> > > >> >> > > Jira/repository.
> >> > > >> > > >> >> > > > Packaging related code usually goes there.
> (That's
> >> > what
> >> > > >> all
> >> > > >> > the
> >> > > >> > > >> >> > > components
> >> > > >> > > >> >> > > > in bigtop does).
> >> > > >> > > >> >> > > > Having said that, once a PR is created, I'll be
> >> > sharing
> >> > > >> the
> >> > > >> > > link
> >> > > >> > > >> of
> >> > > >> > > >> >> PR
> >> > > >> > > >> >> > > > here, so that apex community as well can review
> it.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > Thanks,
> >> > > >> > > >> >> > > > Chinmay.
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha
> >> Thombare <
> >> > > >> > > >> >> > > > aniruddha@datatorrent.com> wrote:
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > > > > +1 on suggestions and approach.
> >> > > >> > > >> >> > > > > We may need to iron out details about exact
> paths
> >> > etc.
> >> > > >> > > >> >> > > > > Which can be done on jira / PR comments.
> >> > > >> > > >> >> > > > > Is that right @dev?
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> >> > > >> > > >> >> > apache@pradeepdalvi.com>
> >> > > >> > > >> >> > > > > wrote:
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > > > > Inline comments...
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay
> >> > Kolhatkar <
> >> > > >> > > >> >> > > > > > chinmay@datatorrent.com>
> >> > > >> > > >> >> > > > > > wrote:
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT
> >> community
> >> > > >> > edition
> >> > > >> > > >> will
> >> > > >> > > >> >> be
> >> > > >> > > >> >> > > > > included
> >> > > >> > > >> >> > > > > > > there.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > Here is the list of jars I found from
> common
> >> > > >> > dependencies
> >> > > >> > > >> >> between
> >> > > >> > > >> >> > > of
> >> > > >> > > >> >> > > > > apex
> >> > > >> > > >> >> > > > > > > and DT Community edition:
> >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar
> >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar
> >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar
> >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar
> >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar
> >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar
> >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar
> >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar
> >> > > >> > > >> >> > > > > > > bval-core-0.5.jar
> >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar
> >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar
> >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar
> >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar
> >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar
> >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar
> >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar
> >> > > >> > > >> >> > > > > > > jline-2.11.jar
> >> > > >> > > >> >> > > > > > > ant-1.9.2.jar
> >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar
> >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar
> >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar
> >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar
> >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do
> check
> >> > that
> >> > > >> > dtcli
> >> > > >> > > >> works
> >> > > >> > > >> >> > > > properly
> >> > > >> > > >> >> > > > > > for
> >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only
> these
> >> > > >> libraries
> >> > > >> > > >> >> present in
> >> > > >> > > >> >> > > > > > isolation
> >> > > >> > > >> >> > > > > > > without dependency on local m2.
> >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars
> >> which
> >> > are
> >> > > >> used
> >> > > >> > > for
> >> > > >> > > >> >> > compile
> >> > > >> > > >> >> > > > > time
> >> > > >> > > >> >> > > > > > > dependency and not runtime in above list
> >> which I
> >> > > can
> >> > > >> > drop
> >> > > >> > > >> to
> >> > > >> > > >> >> keep
> >> > > >> > > >> >> > > > > package
> >> > > >> > > >> >> > > > > > > size minimal.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > @Anniruddha,
> >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the
> >> > > >> dt-env.sh is
> >> > > >> > > >> >> required.
> >> > > >> > > >> >> > > > Only
> >> > > >> > > >> >> > > > > > > required values I see in it are classpath.
> >> and
> >> > > >> dtcli is
> >> > > >> > > >> >> capable
> >> > > >> > > >> >> > of
> >> > > >> > > >> >> > > > > > building
> >> > > >> > > >> >> > > > > > > it on the fly.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see
> how
> >> > dtcli
> >> > > >> can
> >> > > >> > > find
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > jar
> >> > > >> > > >> >> > > > > path,
> >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we
> >> need a
> >> > > >> change
> >> > > >> > in
> >> > > >> > > >> >> dtcli,
> >> > > >> > > >> >> > > then
> >> > > >> > > >> >> > > > > > > community's opinion is required for whether
> >> > dtcli
> >> > > >> > should
> >> > > >> > > >> >> change
> >> > > >> > > >> >> > in
> >> > > >> > > >> >> > > > our
> >> > > >> > > >> >> > > > > > repo
> >> > > >> > > >> >> > > > > > > or a copy of that with required changes
> >> exist in
> >> > > >> bigtop
> >> > > >> > > >> repo
> >> > > >> > > >> >> > until
> >> > > >> > > >> >> > > we
> >> > > >> > > >> >> > > > > > make
> >> > > >> > > >> >> > > > > > > the dtcli generic enough.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > Can we have these file paths set using
> >> environment
> >> > > >> > > variables
> >> > > >> > > >> >> with
> >> > > >> > > >> >> > > some
> >> > > >> > > >> >> > > > > > default values in dtcli?
> >> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file
> different
> >> > from
> >> > > >> > > >> >> dt-site.xml?
> >> > > >> > > >> >> > Not
> >> > > >> > > >> >> > > > > sure
> >> > > >> > > >> >> > > > > > if
> >> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files &
> Dirs
> >> > from
> >> > > >> > > /etc/skel
> >> > > >> > > >> >> are
> >> > > >> > > >> >> > > > copied
> >> > > >> > > >> >> > > > > to
> >> > > >> > > >> >> > > > > > > home of new user when useradd program is
> >> called.
> >> > > But
> >> > > >> > for
> >> > > >> > > >> >> existing
> >> > > >> > > >> >> > > > users
> >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli
> >> > creates
> >> > > >> > ~/.dt/
> >> > > >> > > on
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > fly
> >> > > >> > > >> >> > > > if
> >> > > >> > > >> >> > > > > > not
> >> > > >> > > >> >> > > > > > > encountered for the first time. Again
> >> correct me
> >> > > if
> >> > > >> I'm
> >> > > >> > > >> wrong.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > Yes, you are right. However this would
> >> probably be
> >> > > >> > > necessary
> >> > > >> > > >> >> step
> >> > > >> > > >> >> > for
> >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed
> during
> >> OS
> >> > > >> install
> >> > > >> > > and
> >> > > >> > > >> >> > before
> >> > > >> > > >> >> > > > user
> >> > > >> > > >> >> > > > > > accounts were created.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm
> not
> >> > sure
> >> > > >> its
> >> > > >> > the
> >> > > >> > > >> best
> >> > > >> > > >> >> > > idea.
> >> > > >> > > >> >> > > > > > We'll
> >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by
> >> default
> >> > > >> > present
> >> > > >> > > in
> >> > > >> > > >> >> path
> >> > > >> > > >> >> > > i.e.
> >> > > >> > > >> >> > > > > > > /usr/bin.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > For other env variables specific to apex,
> I'll
> >> > > prefer
> >> > > >> to
> >> > > >> > > use
> >> > > >> > > >> >> > > dt-env.sh
> >> > > >> > > >> >> > > > > and
> >> > > >> > > >> >> > > > > > > source it in dtcli rather than changing
> >> bashrc
> >> > > >> etc...
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/
> >> shall
> >> > > be
> >> > > >> the
> >> > > >> > > >> >> cleaner
> >> > > >> > > >> >> > way
> >> > > >> > > >> >> > > > to
> >> > > >> > > >> >> > > > > > achieve the same.
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > Thanks,
> >> > > >> > > >> >> > > > > > > Chinmay.
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha
> >> > > Thombare
> >> > > >> <
> >> > > >> > > >> >> > > > > > > aniruddha@datatorrent.com> wrote:
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > > > > Hi,
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > @Chinmay,
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > We need to consider following:
> >> > > >> > > >> >> > > > > > > > System wide default config files can be
> >> > located
> >> > > at
> >> > > >> > > >> following
> >> > > >> > > >> >> > > > > locations:
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to
> change
> >> > > dtcli
> >> > > >> > > >> behaviour
> >> > > >> > > >> >> on
> >> > > >> > > >> >> > > how
> >> > > >> > > >> >> > > > it
> >> > > >> > > >> >> > > > > > > finds
> >> > > >> > > >> >> > > > > > > > those locations)
> >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to
> >> change
> >> > > dtcli
> >> > > >> > > >> >> behaviour on
> >> > > >> > > >> >> > > how
> >> > > >> > > >> >> > > > > it
> >> > > >> > > >> >> > > > > > > > finds those locations)
> >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other
> files
> >> > (for
> >> > > >> new
> >> > > >> > > users
> >> > > >> > > >> >> that
> >> > > >> > > >> >> > > will
> >> > > >> > > >> >> > > > > be
> >> > > >> > > >> >> > > > > > > > created in system in future.)
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc /
> >> > bashprofile
> >> > > >> for
> >> > > >> > > >> >> population
> >> > > >> > > >> >> > > any
> >> > > >> > > >> >> > > > > > > > variables that are required.
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > @dev, please put in your comments /
> >> > suggestion.
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > Thanks,
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > Aniruddha
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas
> >> Weise
> >> > <
> >> > > >> > > >> >> > > > > thomas@datatorrent.com>
> >> > > >> > > >> >> > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > > > > -->
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay
> >> > > >> Kolhatkar <
> >> > > >> > > >> >> > > > > > > > chinmay@datatorrent.com
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Hello Everyone!!
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort
> >> (rpm+deb)
> >> > > of
> >> > > >> > apex,
> >> > > >> > > >> here
> >> > > >> > > >> >> > are
> >> > > >> > > >> >> > > > some
> >> > > >> > > >> >> > > > > > > > > proposals
> >> > > >> > > >> >> > > > > > > > > > about package structure etc..
> >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing
> >> > list, I
> >> > > >> have
> >> > > >> > > >> some
> >> > > >> > > >> >> > > question
> >> > > >> > > >> >> > > > > for
> >> > > >> > > >> >> > > > > > > > apex
> >> > > >> > > >> >> > > > > > > > > > community.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of apex
> >> > package
> >> > > >> for
> >> > > >> > > both
> >> > > >> > > >> >> deb &
> >> > > >> > > >> >> > > > rpm:
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> >> > > >> > > >> >> > > > > > > > > >
> >> /usr/lib/apex/lib/apex-api-<version>.jar
> >> > > >> > > >> >> > > > > > > > > >
> >> > /usr/lib/apex/lib/apex-engine-<version>.jar
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> >> > > >> > > >> >> > > > > > > > > >
> >> > /usr/lib/apex/lib/apex-common-<version>.jar
> >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent
> >> jars>
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli ->
> /usr/lib/apex/bin/dtcli
> >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> >> > > >> > > >> >> > > > > > > > > >
> /usr/share/doc/apex/license/LICENSE.txt
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt
> >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating.
> >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd
> >> party
> >> > > >> jars
> >> > > >> > > which
> >> > > >> > > >> >> are
> >> > > >> > > >> >> > > > > required
> >> > > >> > > >> >> > > > > > > for
> >> > > >> > > >> >> > > > > > > > > apex
> >> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies are
> >> > > packaged
> >> > > >> as
> >> > > >> > > >> part of
> >> > > >> > > >> >> > > > rpm/deb
> >> > > >> > > >> >> > > > > by
> >> > > >> > > >> >> > > > > > > any
> >> > > >> > > >> >> > > > > > > > > > software in bigtop.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > Can you specify what those jars are in
> >> > Bigtop
> >> > > >> > > context?
> >> > > >> > > >> >> Same
> >> > > >> > > >> >> > as
> >> > > >> > > >> >> > > > > > shipped
> >> > > >> > > >> >> > > > > > > > with
> >> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ?
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of
> >> > > corresponding
> >> > > >> of
> >> > > >> > > 3rd
> >> > > >> > > >> >> party
> >> > > >> > > >> >> > > jars
> >> > > >> > > >> >> > > > > > which
> >> > > >> > > >> >> > > > > > > > > needs
> >> > > >> > > >> >> > > > > > > > > > to included while packaging.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Questions related to this:
> >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as
> >> > "apex"
> >> > > >> > instead
> >> > > >> > > >> of
> >> > > >> > > >> >> > > "dtcli"
> >> > > >> > > >> >> > > > in
> >> > > >> > > >> >> > > > > > > > bigtop
> >> > > >> > > >> >> > > > > > > > > > package?
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > I think we should keep the name until
> we
> >> are
> >> > > >> able
> >> > > >> > to
> >> > > >> > > >> >> change
> >> > > >> > > >> >> > it
> >> > > >> > > >> >> > > in
> >> > > >> > > >> >> > > > > > Apex.
> >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing.
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop
> >> have
> >> > > man
> >> > > >> > page
> >> > > >> > > >> for
> >> > > >> > > >> >> > their
> >> > > >> > > >> >> > > > > > > > > executables.
> >> > > >> > > >> >> > > > > > > > > > I think we should have it too for
> >> dtcli.
> >> > Is
> >> > > >> there
> >> > > >> > > any
> >> > > >> > > >> >> > > > > documentation
> >> > > >> > > >> >> > > > > > > > > which I
> >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I use
> >> > output
> >> > > >> of
> >> > > >> > > >> "dtcli
> >> > > >> > > >> >> > > --help"?
> >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex
> >> in
> >> > > >> Bigtop
> >> > > >> > as
> >> > > >> > > >> >> 3.3.0 OR
> >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating?
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > Has to be -incubating
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of
> bigtop
> >> to
> >> > > >> depend
> >> > > >> > on
> >> > > >> > > >> >> 2.7.1
> >> > > >> > > >> >> > > > > version
> >> > > >> > > >> >> > > > > > of
> >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we
> see
> >> > with
> >> > > >> this
> >> > > >> > > >> >> > dependency?
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > Should work without changed. I thought
> we
> >> > > >> certified
> >> > > >> > > >> >> against
> >> > > >> > > >> >> > > > 2.7.0?
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror
> from
> >> > which
> >> > > >> > bigtop
> >> > > >> > > >> will
> >> > > >> > > >> >> > pick
> >> > > >> > > >> >> > > > the
> >> > > >> > > >> >> > > > > > > apex
> >> > > >> > > >> >> > > > > > > > > > source for compilation and packaging.
> >> > Please
> >> > > >> > > correct
> >> > > >> > > >> if
> >> > > >> > > >> >> > > > > incorrect:
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > >
> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >> > > >> > > >> >> > > > > > > > > >     NOTE: apache.osuosl.org is the
> >> mirror
> >> > > >> used
> >> > > >> > by
> >> > > >> > > >> all
> >> > > >> > > >> >> the
> >> > > >> > > >> >> > > > > > softwares
> >> > > >> > > >> >> > > > > > > in
> >> > > >> > > >> >> > > > > > > > > > bigtop.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Please share your opinion.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > Thanks,
> >> > > >> > > >> >> > > > > > > > > > Chinmay.
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM,
> >> Chinmay
> >> > > >> > Kolhatkar
> >> > > >> > > <
> >> > > >> > > >> >> > > > > > > > > > chinmay@datatorrent.com>
> >> > > >> > > >> >> > > > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the
> >> packaging.
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > I've already started discussion on
> >> > bigtop
> >> > > >> dev
> >> > > >> > > >> mailing
> >> > > >> > > >> >> > list
> >> > > >> > > >> >> > > > for
> >> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira
> for
> >> the
> >> > > >> same.
> >> > > >> > > For
> >> > > >> > > >> >> this
> >> > > >> > > >> >> > > > > > communities
> >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira:
> >> > > >> > BIGTOP-2313.
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM,
> >> Thomas
> >> > > >> Weise <
> >> > > >> > > >> >> > > > > > > thomas@datatorrent.com
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > > wrote:
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >> Chinmay,
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under
> >> > prerequisites
> >> > > >> that
> >> > > >> > > >> would
> >> > > >> > > >> >> be a
> >> > > >> > > >> >> > > > > > problem.
> >> > > >> > > >> >> > > > > > > We
> >> > > >> > > >> >> > > > > > > > > > >> looked
> >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility
> >> as
> >> > > part
> >> > > >> of
> >> > > >> > > >> >> becoming
> >> > > >> > > >> >> > an
> >> > > >> > > >> >> > > > > > > incubator
> >> > > >> > > >> >> > > > > > > > > > >> project.
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging
> during
> >> > the
> >> > > >> next
> >> > > >> > > >> weeks.
> >> > > >> > > >> >> > Since
> >> > > >> > > >> >> > > > the
> >> > > >> > > >> >> > > > > > > work
> >> > > >> > > >> >> > > > > > > > > will
> >> > > >> > > >> >> > > > > > > > > > >> be
> >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related
> discussions
> >> and
> >> > > >> JIRAs
> >> > > >> > > >> should
> >> > > >> > > >> >> > also
> >> > > >> > > >> >> > > be
> >> > > >> > > >> >> > > > > > > there.
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the
> packaging
> >> in
> >> > > >> place
> >> > > >> > by
> >> > > >> > > >> end
> >> > > >> > > >> >> Feb.
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> Thanks,
> >> > > >> > > >> >> > > > > > > > > > >> Thomas
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM,
> >> > Chinmay
> >> > > >> > > >> Kolhatkar <
> >> > > >> > > >> >> > > > > > > > > > >> chinmay@datatorrent.com>
> >> > > >> > > >> >> > > > > > > > > > >> wrote:
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >> > Hi All,
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on adding
> >> > Apache
> >> > > >> Apex
> >> > > >> > > as a
> >> > > >> > > >> >> > > component
> >> > > >> > > >> >> > > > > to
> >> > > >> > > >> >> > > > > > > > Apache
> >> > > >> > > >> >> > > > > > > > > > >> > Bigtop.
> >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system
> for
> >> > the
> >> > > >> > Apache
> >> > > >> > > >> big
> >> > > >> > > >> >> data
> >> > > >> > > >> >> > > > > > > ecosystem.
> >> > > >> > > >> >> > > > > > > > > > >> Several
> >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most
> >> recently
> >> > > EMR.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task
> in
> >> > > >> APEXCORE
> >> > > >> > for
> >> > > >> > > >> the
> >> > > >> > > >> >> > same:
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as
> >> > > follows:
> >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and
> >> soft
> >> > > >> > > expectation
> >> > > >> > > >> >> for
> >> > > >> > > >> >> > new
> >> > > >> > > >> >> > > > > > > > components
> >> > > >> > > >> >> > > > > > > > > to
> >> > > >> > > >> >> > > > > > > > > > >> get
> >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop.
> >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it:
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> >
> >> > > >> > > >> >>
> >> > > >> > > >>
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be
> standard
> >> ASF
> >> > > >> based
> >> > > >> > > >> >> > > requirements,
> >> > > >> > > >> >> > > > > but
> >> > > >> > > >> >> > > > > > > few
> >> > > >> > > >> >> > > > > > > > > need
> >> > > >> > > >> >> > > > > > > > > > >> to
> >> > > >> > > >> >> > > > > > > > > > >> > be checked for:
> >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are
> expected
> >> to
> >> > be
> >> > > >> > > Licensed
> >> > > >> > > >> >> under
> >> > > >> > > >> >> > > > > Apache
> >> > > >> > > >> >> > > > > > > > > License,
> >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their
> >> dependencies
> >> > are
> >> > > >> > > expected
> >> > > >> > > >> >> to be
> >> > > >> > > >> >> > > > > > > compatible
> >> > > >> > > >> >> > > > > > > > > with
> >> > > >> > > >> >> > > > > > > > > > >> this
> >> > > >> > > >> >> > > > > > > > > > >> > license)
> >> > > >> > > >> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but
> >> need
> >> > to
> >> > > >> > check
> >> > > >> > > if
> >> > > >> > > >> >> > > > > dependencies
> >> > > >> > > >> >> > > > > > of
> >> > > >> > > >> >> > > > > > > > > Apex
> >> > > >> > > >> >> > > > > > > > > > >> are
> >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I
> >> guess
> >> > > >> would
> >> > > >> > > be a
> >> > > >> > > >> >> > > > > verification
> >> > > >> > > >> >> > > > > > > > check.
> >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are
> expected
> >> to
> >> > be
> >> > > >> > > >> compatible
> >> > > >> > > >> >> > with
> >> > > >> > > >> >> > > > all
> >> > > >> > > >> >> > > > > of
> >> > > >> > > >> >> > > > > > > the
> >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop
> >> > > >> distribution
> >> > > >> > > is
> >> > > >> > > >> >> > > targeting
> >> > > >> > > >> >> > > > > > > > > > >> >     - This needs verifying
> whether
> >> > our
> >> > > >> > > software
> >> > > >> > > >> >> runs
> >> > > >> > > >> >> > > fine
> >> > > >> > > >> >> > > > in
> >> > > >> > > >> >> > > > > > > > > centos-6
> >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04
> >> > > debian-8
> >> > > >> > > >> >> > opensuse-13.2.
> >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should
> be
> >> > > added
> >> > > >> for
> >> > > >> > > >> >> > deployment.
> >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test
> artifacts
> >> > which
> >> > > >> goes
> >> > > >> > > >> beyond
> >> > > >> > > >> >> > > smoke
> >> > > >> > > >> >> > > > > test
> >> > > >> > > >> >> > > > > > > > > > >> >     - These are basically the
> >> > > integration
> >> > > >> > > tests
> >> > > >> > > >> for
> >> > > >> > > >> >> > > > > > verification
> >> > > >> > > >> >> > > > > > > > > after
> >> > > >> > > >> >> > > > > > > > > > >> the
> >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft
> >> > requirement,
> >> > > >> but
> >> > > >> > > aim
> >> > > >> > > >> is
> >> > > >> > > >> >> to
> >> > > >> > > >> >> > > > > achieve
> >> > > >> > > >> >> > > > > > > this
> >> > > >> > > >> >> > > > > > > > > as
> >> > > >> > > >> >> > > > > > > > > > >> well
> >> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation why
> >> not
> >> > to
> >> > > >> > > include.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the link
> >> which
> >> > > >> > > explicitly
> >> > > >> > > >> >> needs
> >> > > >> > > >> >> > to
> >> > > >> > > >> >> > > > be
> >> > > >> > > >> >> > > > > > > > checked
> >> > > >> > > >> >> > > > > > > > > > >> other
> >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us
> know.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as
> component
> >> to
> >> > > >> Bigtop*
> >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive of
> >> > Bigtop,
> >> > > >> it
> >> > > >> > was
> >> > > >> > > >> >> learnt
> >> > > >> > > >> >> > > that
> >> > > >> > > >> >> > > > > the
> >> > > >> > > >> >> > > > > > > > > bigtop
> >> > > >> > > >> >> > > > > > > > > > >> > community want to see the
> >> addition of
> >> > > new
> >> > > >> > > >> >> components
> >> > > >> > > >> >> > in
> >> > > >> > > >> >> > > > > > phases.
> >> > > >> > > >> >> > > > > > > > Here
> >> > > >> > > >> >> > > > > > > > > > are
> >> > > >> > > >> >> > > > > > > > > > >> > the phases:
> >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging
> >> > > >> > > >> >> > > > > > > > > > >> >     - This needs creating of
> >> package
> >> > > i.e.
> >> > > >> > rpm
> >> > > >> > > &
> >> > > >> > > >> deb
> >> > > >> > > >> >> > > files.
> >> > > >> > > >> >> > > > > > > > > > >> >     - documentations/READMEs,
> >> > LICENSE,
> >> > > >> > > >> DISCLAMER,
> >> > > >> > > >> >> > NOTES
> >> > > >> > > >> >> > > > etc
> >> > > >> > > >> >> > > > > if
> >> > > >> > > >> >> > > > > > > any
> >> > > >> > > >> >> > > > > > > > > > >> needed.
> >> > > >> > > >> >> > > > > > > > > > >> >     - Any documentation that
> need
> >> to
> >> > be
> >> > > >> > added
> >> > > >> > > to
> >> > > >> > > >> >> > > > > distribution
> >> > > >> > > >> >> > > > > > of
> >> > > >> > > >> >> > > > > > > > our
> >> > > >> > > >> >> > > > > > > > > > >> > software.
> >> > > >> > > >> >> > > > > > > > > > >> >     - Any license information of
> >> > > >> > dependencies
> >> > > >> > > >> >> required
> >> > > >> > > >> >> > > to
> >> > > >> > > >> >> > > > be
> >> > > >> > > >> >> > > > > > > added
> >> > > >> > > >> >> > > > > > > > > to
> >> > > >> > > >> >> > > > > > > > > > >> > package
> >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> >> > > >> > > >> >> > > > > > > > > > >> >     - Adding smoke test for
> >> > packaging.
> >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic
> >> > > >> deployment
> >> > > >> > and
> >> > > >> > > >> >> > > > configuration
> >> > > >> > > >> >> > > > > > > > > > >> >     - Add puppet recipes for
> >> > automatic
> >> > > >> > > >> deployment
> >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests
> >> > > >> > > >> >> > > > > > > > > > >> >     - For verification of
> >> > deployments.
> >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance:
> >> > > >> > > >> >> > > > > > > > > > >> >     Run 'gradle rat' to make
> sure
> >> all
> >> > > new
> >> > > >> > > stuff
> >> > > >> > > >> is
> >> > > >> > > >> >> > > > compliant
> >> > > >> > > >> >> > > > > > > with
> >> > > >> > > >> >> > > > > > > > > ASF
> >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you add
> >> code
> >> > > >> > licenses
> >> > > >> > > >> >> under
> >> > > >> > > >> >> > > > > different
> >> > > >> > > >> >> > > > > > > > > > licenses,
> >> > > >> > > >> >> > > > > > > > > > >> > those would need to be listed in
> >> the
> >> > > >> NOTICE.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on
> the
> >> > > >> approach.
> >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding
> >> > communication
> >> > > >> on
> >> > > >> > > >> bigtop
> >> > > >> > > >> >> > > mailing
> >> > > >> > > >> >> > > > > list
> >> > > >> > > >> >> > > > > > > as
> >> > > >> > > >> >> > > > > > > > > > well.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > We have some specific
> >> > > >> questions/suggestions
> >> > > >> > > >> >> related to
> >> > > >> > > >> >> > > > what
> >> > > >> > > >> >> > > > > > > should
> >> > > >> > > >> >> > > > > > > > > be
> >> > > >> > > >> >> > > > > > > > > > >> the
> >> > > >> > > >> >> > > > > > > > > > >> > content of the package and what
> >> > should
> >> > > be
> >> > > >> > the
> >> > > >> > > >> smoke
> >> > > >> > > >> >> > > tests,
> >> > > >> > > >> >> > > > > but
> >> > > >> > > >> >> > > > > > > in
> >> > > >> > > >> >> > > > > > > > > the
> >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too much
> >> > content
> >> > > >> > here,
> >> > > >> > > >> we'll
> >> > > >> > > >> >> > put
> >> > > >> > > >> >> > > > the
> >> > > >> > > >> >> > > > > > > > > questions
> >> > > >> > > >> >> > > > > > > > > > >> as a
> >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this
> mailthread.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >> > Thanks,
> >> > > >> > > >> >> > > > > > > > > > >> > Chinmay.
> >> > > >> > > >> >> > > > > > > > > > >> >
> >> > > >> > > >> >> > > > > > > > > > >>
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > > >
> >> > > >> > > >> >> > > > > > > > >
> >> > > >> > > >> >> > > > > > > >
> >> > > >> > > >> >> > > > > > >
> >> > > >> > > >> >> > > > > >
> >> > > >> > > >> >> > > > >
> >> > > >> > > >> >> > > >
> >> > > >> > > >> >> > >
> >> > > >> > > >> >> >
> >> > > >> > > >> >>
> >> > > >> > > >> >
> >> > > >> > > >> >
> >> > > >> > > >>
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message