apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@apache.org>
Subject Re: Apache Apex as component in Apache bigtop
Date Mon, 15 Feb 2016 04:16:51 GMT
Hi Thomas,

There are only 2 things that are required from dt-env.sh:
1. DT_HADOOP variable which is exported by dt-env.sh This variable has
absolute path for hadoop binary
2. A find_hadoop() method which find the path of hadoop binary.

JAVA_HOME is not required for this.

The idea is not to change the dtcli. Current dtcli (in apex
3.3.0-incubating src code), sources dt-env.sh for above 2 information.
Hence included dt-env.sh

If the community agrees, we can make dtcli more generic so that in future
packing of bigtop, dt-env.sh can be completely excluded OR at least change
the name to apex-env.sh.

Thanks,
Chinmay.



On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise <thomas@datatorrent.com>
wrote:

> Chinmay,
>
> Can you please specify what settings in dt-env.sh are required in a bigtop
> environment?
>
> Location of hadoop and JAVA_HOME should be provided by the stack already?
>
> Thanks,
> Thomas
>
>
> On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
>
> > Here is a suggestion I have related to name change:
> >
> > 1. While packaging we change the name of "dtcli" script to "apex". This
> is
> > done so that in future when we change the name to "apex", users of bigtop
> > apex don't have to transition much.
> > 2. We also keep a symlink named dtcli which points to apex script. This
> is
> > for backward compatibility.
> > 3. We still keep conf/dt-env.sh for first integration in bigtop. This is
> in
> > the interest of not changing the content of dtcli or apex script.
> > 4. In apex 3.4.0, we take care of all these naming related changes and
> > update bigtop repository later to remove references to dtcli all
> together.
> >
> > Please share your thoughts on above approach.
> >
> > Also, please share what could be the man page content for "apex".
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <chinmay@apache.org>
> > wrote:
> >
> > > Yes.. It'll be build from source tar downloaded from one of the apache
> > > mirror: apache.osuosl.org
> > >
> > > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
> > > options:
> > > If we should not have any dt-env.sh, dtcli would need a change while
> > > packaging.
> > >
> > > But if we want to use dtcli as it is, then we would need dt-env.sh
> > >
> > >
> > >
> > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <thomas@datatorrent.com>
> > > wrote:
> > >
> > >> Looks good overall, though there shouldn't be any dt-env.sh
> > >>
> > >> What will the bigtop package be built from, the source tar?
> > >>
> > >>
> > >>
> > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > >> wrote:
> > >>
> > >> > Hi Thomas,
> > >> >
> > >> > Thanks for the feedback.
> > >> >
> > >> > First, we're not changing the name anywhere. We'll follow what
> > >> currently is
> > >> > in source tarball for 3.3.0-incubating version.
> > >> >
> > >> > Secondly, I've mentioned a directory structure below which is inline
> > >> with
> > >> > other existing integrations with bigtop.
> > >> >
> > >> > The need for each file is as follows:
> > >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
> > >> source
> > >> > code apex 3.3.0-incubating.
> > >> >
> > >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
> > hadoop
> > >> > binary path and exports an env variable for dtcli to use it.
> > >> >
> > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
> way
> > to
> > >> > how DT Community Edition finds dependency jars.
> > >> > Please note that this is the first iteration list of jars. I'm
> trying
> > to
> > >> > narrow this down to only those which are really required.
> > >> > The test that I'm running to check if dtcli runs fine with given
> > >> dependency
> > >> > is to launch+shutdown+kill+status for pi demo.
> > >> >
> > >> > Above is the directory structure which is required for CLI to work.
> > >> > What aniruddha and pradeep has mentioned are some additional files
> > which
> > >> > makes the like of administrator easier. For eg. /etc/skel
> > >> >
> > >> > Please let me know if this is inline with what you're thinking.
> > >> >
> > >> > Thanks,
> > >> > Chinmay.
> > >> >
> > >> >
> > >> > .
> > >> > |-- bin
> > >> > |   `-- dtcli
> > >> > |-- conf
> > >> > |   `-- dt-env.sh
> > >> > `-- lib
> > >> >     |-- ant-1.9.2.jar
> > >> >     |-- ant-launcher-1.9.2.jar
> > >> >     |-- apex-api-3.3.0-incubating.jar
> > >> >     |-- apex-bufferserver-3.3.0-incubating.jar
> > >> >     |-- apex-common-3.3.0-incubating.jar
> > >> >     |-- apex-engine.jar
> > >> >     |-- apex-engine-tests.jar
> > >> >     |-- async-http-client-1.7.20.jar
> > >> >     |-- bval-core-0.5.jar
> > >> >     |-- bval-jsr303-0.5.jar
> > >> >     |-- commons-beanutils-1.8.3.jar
> > >> >     |-- commons-codec-1.10.jar
> > >> >     |-- commons-lang3-3.1.jar
> > >> >     |-- commons-logging-1.1.3.jar
> > >> >     |-- grizzly-http-servlet-2.1.2.jar
> > >> >     |-- hadoop-common-2.2.0-tests.jar
> > >> >     |-- httpclient-4.3.5.jar
> > >> >     |-- httpcore-4.3.2.jar
> > >> >     |-- jackson-core-asl-1.9.2.jar
> > >> >     |-- jackson-mapper-asl-1.9.2.jar
> > >> >     |-- javax.servlet-3.1.jar
> > >> >     |-- javax.servlet-api-3.0.1.jar
> > >> >     |-- jersey-apache-client4-1.9.jar
> > >> >     |-- jersey-client-1.9.jar
> > >> >     |-- jetty-http-8.1.10.v20130312.jar
> > >> >     |-- jetty-io-8.1.10.v20130312.jar
> > >> >     |-- jetty-util-8.1.10.v20130312.jar
> > >> >     |-- jetty-websocket-8.1.10.v20130312.jar
> > >> >     |-- jline-2.11.jar
> > >> >     |-- kryo-2.24.0.jar
> > >> >     |-- mbassador-1.1.9.jar
> > >> >     |-- minlog-1.2.jar
> > >> >     |-- netlet-1.2.0.jar
> > >> >     |-- netty-3.6.6.Final.jar
> > >> >     |-- objenesis-2.1.jar
> > >> >     |-- validation-api-1.1.0.Final.jar
> > >> >     |-- xbean-asm5-shaded-4.3.jar
> > >> >     `-- zip4j-1.3.2.jar
> > >> >
> > >> > 3 directories, 40 files
> > >> >
> > >> >
> > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > >> > wrote:
> > >> >
> > >> > > Chinmay,
> > >> > >
> > >> > > Before discussing where to put the files, let's make sure they are
> > >> really
> > >> > > needed for the operation of the CLI. As for names, anything new
> > needs
> > >> to
> > >> > > reflect Apex in the name and should follow common conventions,
> > >> especially
> > >> > > in Bigtop where there are many existing integrations to look at.
> > >> > >
> > >> > > Thanks,
> > >> > > Thomas
> > >> > >
> > >> > >
> > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> > >> > > chinmay@datatorrent.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Really good points Pradeep and Aniruddha.
> > >> > > >
> > >> > > > 1. I believe we won't need to change the dtcli considering it
> > works
> > >> > with
> > >> > > DT
> > >> > > > Community edition. We can keep the directory structure similar
> to
> > >> that.
> > >> > > > dt-env.sh has variables which contains information required for
> > >> dtcli
> > >> > to
> > >> > > > launch.
> > >> > > >
> > >> > > > 2. Let me check with bigtop community that whether they
> facilitate
> > >> the
> > >> > > > installation of rpms/debs before user is created. In either
> case,
> > >> > current
> > >> > > > dtcli creates a .dt folder in home directory. Also before
> putting
> > >> > > anything
> > >> > > > in /etc/skel we need to define what are the default contents
> that
> > >> > should
> > >> > > go
> > >> > > > to ~/.dt/ folder. If there is no defaults, probably we should
> not
> > >> > > > explicitly add it in /etc/skel.
> > >> > > >
> > >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
> > >> dt-env.sh
> > >> > > are
> > >> > > > present as env variables. But I see a catch there. Adding
> > dt-env.sh
> > >> to
> > >> > > > /etc/profile.d would make all the variables available at runtime
> > all
> > >> > the
> > >> > > > time. I feel a little skeptical about that. Maybe a possible
> > >> collision
> > >> > > can
> > >> > > > occur with other application vars.
> > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
> > >> > > >
> > >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
> > >> > > Jira/repository.
> > >> > > > Packaging related code usually goes there. (That's what all the
> > >> > > components
> > >> > > > in bigtop does).
> > >> > > > Having said that, once a PR is created, I'll be sharing the link
> > of
> > >> PR
> > >> > > > here, so that apex community as well can review it.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Chinmay.
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > >> > > > aniruddha@datatorrent.com> wrote:
> > >> > > >
> > >> > > > > +1 on suggestions and approach.
> > >> > > > > We may need to iron out details about exact paths etc.
> > >> > > > > Which can be done on jira / PR comments.
> > >> > > > > Is that right @dev?
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
> > >> > apache@pradeepdalvi.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Inline comments...
> > >> > > > > >
> > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > >> > > > > > chinmay@datatorrent.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > @Thomas, Not all the jar present in DT community edition
> > will
> > >> be
> > >> > > > > included
> > >> > > > > > > there.
> > >> > > > > > >
> > >> > > > > > > Here is the list of jars I found from common dependencies
> > >> between
> > >> > > of
> > >> > > > > apex
> > >> > > > > > > and DT Community edition:
> > >> > > > > > > netlet-1.2.0.jar
> > >> > > > > > > kryo-2.24.0.jar
> > >> > > > > > > jackson-core-asl-1.9.2.jar
> > >> > > > > > > jackson-mapper-asl-1.9.2.jar
> > >> > > > > > > async-http-client-1.7.20.jar
> > >> > > > > > > netty-3.6.6.Final.jar
> > >> > > > > > > validation-api-1.1.0.Final.jar
> > >> > > > > > > bval-jsr303-0.5.jar
> > >> > > > > > > bval-core-0.5.jar
> > >> > > > > > > commons-lang3-3.1.jar
> > >> > > > > > > commons-beanutils-1.8.3.jar
> > >> > > > > > > httpclient-4.3.5.jar
> > >> > > > > > > commons-codec-1.10.jar
> > >> > > > > > > zip4j-1.3.2.jar
> > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
> > >> > > > > > > xbean-asm5-shaded-4.3.jar
> > >> > > > > > > jersey-apache-client4-1.9.jar
> > >> > > > > > > jline-2.11.jar
> > >> > > > > > > ant-1.9.2.jar
> > >> > > > > > > ant-launcher-1.9.2.jar
> > >> > > > > > > mbassador-1.1.9.jar
> > >> > > > > > > jackson-jaxrs-1.9.2.jar
> > >> > > > > > > jackson-xc-1.9.2.jar
> > >> > > > > > > hadoop-common-2.2.0-tests.jar
> > >> > > > > > >
> > >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
> > works
> > >> > > > properly
> > >> > > > > > for
> > >> > > > > > > launch+shutdown+kill of apps with only these libraries
> > >> present in
> > >> > > > > > isolation
> > >> > > > > > > without dependency on local m2.
> > >> > > > > > > I'm believe that there are unwanted jars which are used
> for
> > >> > compile
> > >> > > > > time
> > >> > > > > > > dependency and not runtime in above list which I can drop
> to
> > >> keep
> > >> > > > > package
> > >> > > > > > > size minimal.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > @Anniruddha,
> > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
> > >> required.
> > >> > > > Only
> > >> > > > > > > required values I see in it are classpath. and dtcli is
> > >> capable
> > >> > of
> > >> > > > > > building
> > >> > > > > > > it on the fly.
> > >> > > > > > >
> > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can
> find
> > >> the
> > >> > jar
> > >> > > > > path,
> > >> > > > > > > dt-env.sh and all such conf files. If we need a change in
> > >> dtcli,
> > >> > > then
> > >> > > > > > > community's opinion is required for whether dtcli should
> > >> change
> > >> > in
> > >> > > > our
> > >> > > > > > repo
> > >> > > > > > > or a copy of that with required changes exist in bigtop
> repo
> > >> > until
> > >> > > we
> > >> > > > > > make
> > >> > > > > > > the dtcli generic enough.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > Can we have these file paths set using environment variables
> > >> with
> > >> > > some
> > >> > > > > > default values in dtcli?
> > >> > > > > > Then we can set such params in dt-env.sh.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > 3. dt-sited.xml -> How is this file different from
> > >> dt-site.xml?
> > >> > Not
> > >> > > > > sure
> > >> > > > > > if
> > >> > > > > > > adding in /etc/skel would help. Files & Dirs from
> /etc/skel
> > >> are
> > >> > > > copied
> > >> > > > > to
> > >> > > > > > > home of new user when useradd program is called. But for
> > >> existing
> > >> > > > users
> > >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/
> on
> > >> the
> > >> > fly
> > >> > > > if
> > >> > > > > > not
> > >> > > > > > > encountered for the first time. Again correct me if I'm
> > wrong.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > Yes, you are right. However this would probably be necessary
> > >> step
> > >> > for
> > >> > > > > > rpm/deb, as they may also get installed during OS install
> and
> > >> > before
> > >> > > > user
> > >> > > > > > accounts were created.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
> > best
> > >> > > idea.
> > >> > > > > > We'll
> > >> > > > > > > anyway put dtcli in location which is by default present
> in
> > >> path
> > >> > > i.e.
> > >> > > > > > > /usr/bin.
> > >> > > > > >
> > >> > > > > > For other env variables specific to apex, I'll prefer to use
> > >> > > dt-env.sh
> > >> > > > > and
> > >> > > > > > > source it in dtcli rather than changing bashrc etc...
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
> > >> cleaner
> > >> > way
> > >> > > > to
> > >> > > > > > achieve the same.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > > Chinmay.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > >> > > > > > > aniruddha@datatorrent.com> wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi,
> > >> > > > > > > >
> > >> > > > > > > > @Chinmay,
> > >> > > > > > > >
> > >> > > > > > > > We need to consider following:
> > >> > > > > > > > System wide default config files can be located at
> > following
> > >> > > > > locations:
> > >> > > > > > > >
> > >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
> > behaviour
> > >> on
> > >> > > how
> > >> > > > it
> > >> > > > > > > finds
> > >> > > > > > > > those locations)
> > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
> > >> behaviour on
> > >> > > how
> > >> > > > > it
> > >> > > > > > > > finds those locations)
> > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new
> users
> > >> that
> > >> > > will
> > >> > > > > be
> > >> > > > > > > > created in system in future.)
> > >> > > > > > > >
> > >> > > > > > > > We may also have to modify bashrc / bashprofile for
> > >> population
> > >> > > any
> > >> > > > > > > > variables that are required.
> > >> > > > > > > >
> > >> > > > > > > > @dev, please put in your comments / suggestion.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Aniruddha
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > >> > > > > thomas@datatorrent.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > -->
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > >> > > > > > > > chinmay@datatorrent.com
> > >> > > > > > > > > >
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hello Everyone!!
> > >> > > > > > > > > >
> > >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
> > here
> > >> > are
> > >> > > > some
> > >> > > > > > > > > proposals
> > >> > > > > > > > > > about package structure etc..
> > >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
> some
> > >> > > question
> > >> > > > > for
> > >> > > > > > > > apex
> > >> > > > > > > > > > community.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Proposed Directory structure of apex package for
> both
> > >> deb &
> > >> > > > rpm:
> > >> > > > > > > > > >
> > >> > > > > > > > > > /usr/lib/apex/bin/dtcli
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > >> > > > > > > > >
> > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
> > >> > > > > > > > > > /usr/share/doc/apex/NOTICE
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > <version> = 3.3.0-incubating.
> > >> > > > > > > > > > <other dependent jars> = All the 3rd party jars
> which
> > >> are
> > >> > > > > required
> > >> > > > > > > for
> > >> > > > > > > > > apex
> > >> > > > > > > > > > to run. Usually the dependencies are packaged as
> part
> > of
> > >> > > > rpm/deb
> > >> > > > > by
> > >> > > > > > > any
> > >> > > > > > > > > > software in bigtop.
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Can you specify what those jars are in Bigtop context?
> > >> Same
> > >> > as
> > >> > > > > > shipped
> > >> > > > > > > > with
> > >> > > > > > > > > DT community addition under lib/ ?
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
> > >> party
> > >> > > jars
> > >> > > > > > which
> > >> > > > > > > > > needs
> > >> > > > > > > > > > to included while packaging.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Questions related to this:
> > >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
> of
> > >> > > "dtcli"
> > >> > > > in
> > >> > > > > > > > bigtop
> > >> > > > > > > > > > package?
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > I think we should keep the name until we are able to
> > >> change
> > >> > it
> > >> > > in
> > >> > > > > > Apex.
> > >> > > > > > > > > Otherwise this may get confusing.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
> > for
> > >> > their
> > >> > > > > > > > > executables.
> > >> > > > > > > > > > I think we should have it too for dtcli. Is there
> any
> > >> > > > > documentation
> > >> > > > > > > > > which I
> > >> > > > > > > > > > can convert to man page? or can I use output of
> "dtcli
> > >> > > --help"?
> > >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
> > >> 3.3.0 OR
> > >> > > > > > > > > > 3.3.0-incubating?
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Has to be -incubating
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
> > >> 2.7.1
> > >> > > > > version
> > >> > > > > > of
> > >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
> > >> > dependency?
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Should work without changed. I thought we certified
> > >> against
> > >> > > > 2.7.0?
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
> > will
> > >> > pick
> > >> > > > the
> > >> > > > > > > apex
> > >> > > > > > > > > > source for compilation and packaging. Please correct
> > if
> > >> > > > > incorrect:
> > >> > > > > > > > > >
> > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
> all
> > >> the
> > >> > > > > > softwares
> > >> > > > > > > in
> > >> > > > > > > > > > bigtop.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Please share your opinion.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Chinmay.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > >> > > > > > > > > > chinmay@datatorrent.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Yes.. Starting to work on the packaging.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I've already started discussion on bigtop dev
> > mailing
> > >> > list
> > >> > > > for
> > >> > > > > > > > > > > integration. Also created a Jira for the same. For
> > >> this
> > >> > > > > > communities
> > >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > >> > > > > > > thomas@datatorrent.com
> > >> > > > > > > > >
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >> Chinmay,
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> I don't see anything under prerequisites that
> would
> > >> be a
> > >> > > > > > problem.
> > >> > > > > > > We
> > >> > > > > > > > > > >> looked
> > >> > > > > > > > > > >> at the ASF licencing compatibility as part of
> > >> becoming
> > >> > an
> > >> > > > > > > incubator
> > >> > > > > > > > > > >> project.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Please focus on the packaging during the next
> > weeks.
> > >> > Since
> > >> > > > the
> > >> > > > > > > work
> > >> > > > > > > > > will
> > >> > > > > > > > > > >> be
> > >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
> > should
> > >> > also
> > >> > > be
> > >> > > > > > > there.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Would be good to have the packaging in place by
> end
> > >> Feb.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Thanks,
> > >> > > > > > > > > > >> Thomas
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
> Kolhatkar
> > <
> > >> > > > > > > > > > >> chinmay@datatorrent.com>
> > >> > > > > > > > > > >> wrote:
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> > Hi All,
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > We're planning a work on adding Apache Apex as
> a
> > >> > > component
> > >> > > > > to
> > >> > > > > > > > Apache
> > >> > > > > > > > > > >> > Bigtop.
> > >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache
> big
> > >> data
> > >> > > > > > > ecosystem.
> > >> > > > > > > > > > >> Several
> > >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
> > the
> > >> > same:
> > >> > > > > > > > > > >> >
> > https://issues.apache.org/jira/browse/APEXCORE-331
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Proposed plan of execution is as follows:
> > >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
> > >> > > > > > > > > > >> > Apache bigtop has some hard and soft
> expectation
> > >> for
> > >> > new
> > >> > > > > > > > components
> > >> > > > > > > > > to
> > >> > > > > > > > > > >> get
> > >> > > > > > > > > > >> > integrated into Bigtop.
> > >> > > > > > > > > > >> > Here is the list of it:
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Most of them seems to be standard ASF based
> > >> > > requirements,
> > >> > > > > but
> > >> > > > > > > few
> > >> > > > > > > > > need
> > >> > > > > > > > > > >> to
> > >> > > > > > > > > > >> > be checked for:
> > >> > > > > > > > > > >> > 1. Software projects are expected to be
> Licensed
> > >> under
> > >> > > > > Apache
> > >> > > > > > > > > License,
> > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are
> expected
> > >> to be
> > >> > > > > > > compatible
> > >> > > > > > > > > with
> > >> > > > > > > > > > >> this
> > >> > > > > > > > > > >> > license)
> > >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check
> if
> > >> > > > > dependencies
> > >> > > > > > of
> > >> > > > > > > > > Apex
> > >> > > > > > > > > > >> are
> > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be
> a
> > >> > > > > verification
> > >> > > > > > > > check.
> > >> > > > > > > > > > >> > 2. Software projects are expected to be
> > compatible
> > >> > with
> > >> > > > all
> > >> > > > > of
> > >> > > > > > > the
> > >> > > > > > > > > > >> > supported platforms that Bigtop distribution is
> > >> > > targeting
> > >> > > > > > > > > > >> >     - This needs verifying whether our software
> > >> runs
> > >> > > fine
> > >> > > > in
> > >> > > > > > > > > centos-6
> > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
> > >> > opensuse-13.2.
> > >> > > > > > > > > > >> > 3. What smoke tests that should be added for
> > >> > deployment.
> > >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
> > beyond
> > >> > > smoke
> > >> > > > > test
> > >> > > > > > > > > > >> >     - These are basically the integration tests
> > for
> > >> > > > > > verification
> > >> > > > > > > > > after
> > >> > > > > > > > > > >> the
> > >> > > > > > > > > > >> > deployment. This is a soft requirement, but aim
> > is
> > >> to
> > >> > > > > achieve
> > >> > > > > > > this
> > >> > > > > > > > > as
> > >> > > > > > > > > > >> well
> > >> > > > > > > > > > >> > or at least have explanation why not to
> include.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > If there are any from the link which explicitly
> > >> needs
> > >> > to
> > >> > > > be
> > >> > > > > > > > checked
> > >> > > > > > > > > > >> other
> > >> > > > > > > > > > >> > than above 4, please let us know.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
> > >> learnt
> > >> > > that
> > >> > > > > the
> > >> > > > > > > > > bigtop
> > >> > > > > > > > > > >> > community want to see the addition of new
> > >> components
> > >> > in
> > >> > > > > > phases.
> > >> > > > > > > > Here
> > >> > > > > > > > > > are
> > >> > > > > > > > > > >> > the phases:
> > >> > > > > > > > > > >> > 1. Packaging
> > >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
> > deb
> > >> > > files.
> > >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
> DISCLAMER,
> > >> > NOTES
> > >> > > > etc
> > >> > > > > if
> > >> > > > > > > any
> > >> > > > > > > > > > >> needed.
> > >> > > > > > > > > > >> >     - Any documentation that need to be added
> to
> > >> > > > > distribution
> > >> > > > > > of
> > >> > > > > > > > our
> > >> > > > > > > > > > >> > software.
> > >> > > > > > > > > > >> >     - Any license information of dependencies
> > >> required
> > >> > > to
> > >> > > > be
> > >> > > > > > > added
> > >> > > > > > > > > to
> > >> > > > > > > > > > >> > package
> > >> > > > > > > > > > >> > 2. Smoke tests (at very least)
> > >> > > > > > > > > > >> >     - Adding smoke test for packaging.
> > >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > >> > > > configuration
> > >> > > > > > > > > > >> >     - Add puppet recipes for automatic
> deployment
> > >> > > > > > > > > > >> > 4. Integration tests
> > >> > > > > > > > > > >> >     - For verification of deployments.
> > >> > > > > > > > > > >> > 5. license clearance:
> > >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
> > is
> > >> > > > compliant
> > >> > > > > > > with
> > >> > > > > > > > > ASF
> > >> > > > > > > > > > >> > license requirements. If you add code licenses
> > >> under
> > >> > > > > different
> > >> > > > > > > > > > licenses,
> > >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Please share your thoughts on the approach.
> > >> > > > > > > > > > >> > We'll start corresponding communication on
> bigtop
> > >> > > mailing
> > >> > > > > list
> > >> > > > > > > as
> > >> > > > > > > > > > well.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > We have some specific questions/suggestions
> > >> related to
> > >> > > > what
> > >> > > > > > > should
> > >> > > > > > > > > be
> > >> > > > > > > > > > >> the
> > >> > > > > > > > > > >> > content of the package and what should be the
> > smoke
> > >> > > tests,
> > >> > > > > but
> > >> > > > > > > in
> > >> > > > > > > > > the
> > >> > > > > > > > > > >> > interest of not having too much content here,
> > we'll
> > >> > put
> > >> > > > the
> > >> > > > > > > > > questions
> > >> > > > > > > > > > >> as a
> > >> > > > > > > > > > >> > separate mail in this mailthread.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Thanks,
> > >> > > > > > > > > > >> > Chinmay.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message