apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yan <da...@datatorrent.com>
Subject Re: Apache Apex as component in Apache bigtop
Date Mon, 15 Feb 2016 17:57:41 GMT
Apologies for hitting the "Send" button too early.

What is the scope of the name change?  If we simply have a symlink from
apex to dtcli, then it should have no backward compatibility problem at all.

But the proper name change would involve:
- Change the name of DTCli class to ApexCli
- Change the prompt from dt> to apex>
- Change the initial greeting when running the CLI from something like "DT
CLI 3.3.0-incubating ..." to "Apex CLI ...".

The above changes may break existing scripts that rely on the the string
"DT".
But if folks are okay with this potential problem, I am okay with the name
change.

For the man page, we can start with the content of this page:

http://docs.datatorrent.com/dtcli/

Plus the command line options when doing dtcli -h:

usage: DTCli
 -e <arg>    Commands are read from the argument
 -f <arg>    Use the specified prompt at all time
 -h          Print this help
 -kp <arg>   Use the specified kerberos principal
 -kt <arg>   Use the specified kerberos keytab
 -p <arg>    JSONP padding function
 -r          JSON Raw mode
 -v          Verbose mode level 1
 -vv         Verbose mode level 2
 -vvv        Verbose mode level 3
 -vvvv       Verbose mode level 4

David


On Mon, Feb 15, 2016 at 9:56 AM, David Yan <david@datatorrent.com> wrote:

> What is the scope of the name change?  If we simply have a symlink from
> apex to dtcli, then it should have no backward compatibility problem at all.
>
> But the proper name change would involve:
> - Change the name of DTCli class to ApexCli
> - Change the prompt from dt> to apex>
> - Change the initial greeting when running the CLI from something like "DT
> CLI 3.3.0-incubating ..." to "Apex CLI ...".
>
> The above changes may break existing scripts that rely on the the string
> "DT".
> But if folks are okay with this potential problem, I am okay with the name
> change.
>
> For the man page, we can start with the content of this page:
>
>
>
>
> On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
>
>> Here is a suggestion I have related to name change:
>>
>> 1. While packaging we change the name of "dtcli" script to "apex". This is
>> done so that in future when we change the name to "apex", users of bigtop
>> apex don't have to transition much.
>> 2. We also keep a symlink named dtcli which points to apex script. This is
>> for backward compatibility.
>> 3. We still keep conf/dt-env.sh for first integration in bigtop. This is
>> in
>> the interest of not changing the content of dtcli or apex script.
>> 4. In apex 3.4.0, we take care of all these naming related changes and
>> update bigtop repository later to remove references to dtcli all together.
>>
>> Please share your thoughts on above approach.
>>
>> Also, please share what could be the man page content for "apex".
>>
>> Thanks,
>> Chinmay.
>>
>>
>>
>> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar <chinmay@apache.org>
>> wrote:
>>
>> > Yes.. It'll be build from source tar downloaded from one of the apache
>> > mirror: apache.osuosl.org
>> >
>> > The reason for dt-env.sh is it's sourced from dtcli. This leaves us 3
>> > options:
>> > If we should not have any dt-env.sh, dtcli would need a change while
>> > packaging.
>> >
>> > But if we want to use dtcli as it is, then we would need dt-env.sh
>> >
>> >
>> >
>> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise <thomas@datatorrent.com>
>> > wrote:
>> >
>> >> Looks good overall, though there shouldn't be any dt-env.sh
>> >>
>> >> What will the bigtop package be built from, the source tar?
>> >>
>> >>
>> >>
>> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar <chinmay@apache.org
>> >
>> >> wrote:
>> >>
>> >> > Hi Thomas,
>> >> >
>> >> > Thanks for the feedback.
>> >> >
>> >> > First, we're not changing the name anywhere. We'll follow what
>> >> currently is
>> >> > in source tarball for 3.3.0-incubating version.
>> >> >
>> >> > Secondly, I've mentioned a directory structure below which is inline
>> >> with
>> >> > other existing integrations with bigtop.
>> >> >
>> >> > The need for each file is as follows:
>> >> > 1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of
>> >> source
>> >> > code apex 3.3.0-incubating.
>> >> >
>> >> > 2. conf/dt-env.sh -> This is source from dtcli. This searches for
>> hadoop
>> >> > binary path and exports an env variable for dtcli to use it.
>> >> >
>> >> > 3. lib/ -> Libraries for CLI. This list is extracted in a similar
>> way to
>> >> > how DT Community Edition finds dependency jars.
>> >> > Please note that this is the first iteration list of jars. I'm
>> trying to
>> >> > narrow this down to only those which are really required.
>> >> > The test that I'm running to check if dtcli runs fine with given
>> >> dependency
>> >> > is to launch+shutdown+kill+status for pi demo.
>> >> >
>> >> > Above is the directory structure which is required for CLI to work.
>> >> > What aniruddha and pradeep has mentioned are some additional files
>> which
>> >> > makes the like of administrator easier. For eg. /etc/skel
>> >> >
>> >> > Please let me know if this is inline with what you're thinking.
>> >> >
>> >> > Thanks,
>> >> > Chinmay.
>> >> >
>> >> >
>> >> > .
>> >> > |-- bin
>> >> > |   `-- dtcli
>> >> > |-- conf
>> >> > |   `-- dt-env.sh
>> >> > `-- lib
>> >> >     |-- ant-1.9.2.jar
>> >> >     |-- ant-launcher-1.9.2.jar
>> >> >     |-- apex-api-3.3.0-incubating.jar
>> >> >     |-- apex-bufferserver-3.3.0-incubating.jar
>> >> >     |-- apex-common-3.3.0-incubating.jar
>> >> >     |-- apex-engine.jar
>> >> >     |-- apex-engine-tests.jar
>> >> >     |-- async-http-client-1.7.20.jar
>> >> >     |-- bval-core-0.5.jar
>> >> >     |-- bval-jsr303-0.5.jar
>> >> >     |-- commons-beanutils-1.8.3.jar
>> >> >     |-- commons-codec-1.10.jar
>> >> >     |-- commons-lang3-3.1.jar
>> >> >     |-- commons-logging-1.1.3.jar
>> >> >     |-- grizzly-http-servlet-2.1.2.jar
>> >> >     |-- hadoop-common-2.2.0-tests.jar
>> >> >     |-- httpclient-4.3.5.jar
>> >> >     |-- httpcore-4.3.2.jar
>> >> >     |-- jackson-core-asl-1.9.2.jar
>> >> >     |-- jackson-mapper-asl-1.9.2.jar
>> >> >     |-- javax.servlet-3.1.jar
>> >> >     |-- javax.servlet-api-3.0.1.jar
>> >> >     |-- jersey-apache-client4-1.9.jar
>> >> >     |-- jersey-client-1.9.jar
>> >> >     |-- jetty-http-8.1.10.v20130312.jar
>> >> >     |-- jetty-io-8.1.10.v20130312.jar
>> >> >     |-- jetty-util-8.1.10.v20130312.jar
>> >> >     |-- jetty-websocket-8.1.10.v20130312.jar
>> >> >     |-- jline-2.11.jar
>> >> >     |-- kryo-2.24.0.jar
>> >> >     |-- mbassador-1.1.9.jar
>> >> >     |-- minlog-1.2.jar
>> >> >     |-- netlet-1.2.0.jar
>> >> >     |-- netty-3.6.6.Final.jar
>> >> >     |-- objenesis-2.1.jar
>> >> >     |-- validation-api-1.1.0.Final.jar
>> >> >     |-- xbean-asm5-shaded-4.3.jar
>> >> >     `-- zip4j-1.3.2.jar
>> >> >
>> >> > 3 directories, 40 files
>> >> >
>> >> >
>> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <
>> thomas@datatorrent.com>
>> >> > wrote:
>> >> >
>> >> > > Chinmay,
>> >> > >
>> >> > > Before discussing where to put the files, let's make sure they are
>> >> really
>> >> > > needed for the operation of the CLI. As for names, anything new
>> needs
>> >> to
>> >> > > reflect Apex in the name and should follow common conventions,
>> >> especially
>> >> > > in Bigtop where there are many existing integrations to look at.
>> >> > >
>> >> > > Thanks,
>> >> > > Thomas
>> >> > >
>> >> > >
>> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
>> >> > > chinmay@datatorrent.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Really good points Pradeep and Aniruddha.
>> >> > > >
>> >> > > > 1. I believe we won't need to change the dtcli considering it
>> works
>> >> > with
>> >> > > DT
>> >> > > > Community edition. We can keep the directory structure similar to
>> >> that.
>> >> > > > dt-env.sh has variables which contains information required for
>> >> dtcli
>> >> > to
>> >> > > > launch.
>> >> > > >
>> >> > > > 2. Let me check with bigtop community that whether they
>> facilitate
>> >> the
>> >> > > > installation of rpms/debs before user is created. In either case,
>> >> > current
>> >> > > > dtcli creates a .dt folder in home directory. Also before putting
>> >> > > anything
>> >> > > > in /etc/skel we need to define what are the default contents that
>> >> > should
>> >> > > go
>> >> > > > to ~/.dt/ folder. If there is no defaults, probably we should not
>> >> > > > explicitly add it in /etc/skel.
>> >> > > >
>> >> > > > 3.  /etc/profile.d approach looks nice. This way contents of
>> >> dt-env.sh
>> >> > > are
>> >> > > > present as env variables. But I see a catch there. Adding
>> dt-env.sh
>> >> to
>> >> > > > /etc/profile.d would make all the variables available at runtime
>> all
>> >> > the
>> >> > > > time. I feel a little skeptical about that. Maybe a possible
>> >> collision
>> >> > > can
>> >> > > > occur with other application vars.
>> >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" and
>> >> > > > "~/.dt/dt-env.sh". So those variables are anyway available.
>> >> > > >
>> >> > > > @Aniruddha, the Jira and PR effort is happening at Bigtop
>> >> > > Jira/repository.
>> >> > > > Packaging related code usually goes there. (That's what all the
>> >> > > components
>> >> > > > in bigtop does).
>> >> > > > Having said that, once a PR is created, I'll be sharing the link
>> of
>> >> PR
>> >> > > > here, so that apex community as well can review it.
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Chinmay.
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
>> >> > > > aniruddha@datatorrent.com> wrote:
>> >> > > >
>> >> > > > > +1 on suggestions and approach.
>> >> > > > > We may need to iron out details about exact paths etc.
>> >> > > > > Which can be done on jira / PR comments.
>> >> > > > > Is that right @dev?
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <
>> >> > apache@pradeepdalvi.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Inline comments...
>> >> > > > > >
>> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
>> >> > > > > > chinmay@datatorrent.com>
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > > > @Thomas, Not all the jar present in DT community edition
>> will
>> >> be
>> >> > > > > included
>> >> > > > > > > there.
>> >> > > > > > >
>> >> > > > > > > Here is the list of jars I found from common dependencies
>> >> between
>> >> > > of
>> >> > > > > apex
>> >> > > > > > > and DT Community edition:
>> >> > > > > > > netlet-1.2.0.jar
>> >> > > > > > > kryo-2.24.0.jar
>> >> > > > > > > jackson-core-asl-1.9.2.jar
>> >> > > > > > > jackson-mapper-asl-1.9.2.jar
>> >> > > > > > > async-http-client-1.7.20.jar
>> >> > > > > > > netty-3.6.6.Final.jar
>> >> > > > > > > validation-api-1.1.0.Final.jar
>> >> > > > > > > bval-jsr303-0.5.jar
>> >> > > > > > > bval-core-0.5.jar
>> >> > > > > > > commons-lang3-3.1.jar
>> >> > > > > > > commons-beanutils-1.8.3.jar
>> >> > > > > > > httpclient-4.3.5.jar
>> >> > > > > > > commons-codec-1.10.jar
>> >> > > > > > > zip4j-1.3.2.jar
>> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar
>> >> > > > > > > xbean-asm5-shaded-4.3.jar
>> >> > > > > > > jersey-apache-client4-1.9.jar
>> >> > > > > > > jline-2.11.jar
>> >> > > > > > > ant-1.9.2.jar
>> >> > > > > > > ant-launcher-1.9.2.jar
>> >> > > > > > > mbassador-1.1.9.jar
>> >> > > > > > > jackson-jaxrs-1.9.2.jar
>> >> > > > > > > jackson-xc-1.9.2.jar
>> >> > > > > > > hadoop-common-2.2.0-tests.jar
>> >> > > > > > >
>> >> > > > > > > Ofcourse, I'll be running some tests do check that dtcli
>> works
>> >> > > > properly
>> >> > > > > > for
>> >> > > > > > > launch+shutdown+kill of apps with only these libraries
>> >> present in
>> >> > > > > > isolation
>> >> > > > > > > without dependency on local m2.
>> >> > > > > > > I'm believe that there are unwanted jars which are used for
>> >> > compile
>> >> > > > > time
>> >> > > > > > > dependency and not runtime in above list which I can drop
>> to
>> >> keep
>> >> > > > > package
>> >> > > > > > > size minimal.
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > @Anniruddha,
>> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is
>> >> required.
>> >> > > > Only
>> >> > > > > > > required values I see in it are classpath. and dtcli is
>> >> capable
>> >> > of
>> >> > > > > > building
>> >> > > > > > > it on the fly.
>> >> > > > > > >
>> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find
>> >> the
>> >> > jar
>> >> > > > > path,
>> >> > > > > > > dt-env.sh and all such conf files. If we need a change in
>> >> dtcli,
>> >> > > then
>> >> > > > > > > community's opinion is required for whether dtcli should
>> >> change
>> >> > in
>> >> > > > our
>> >> > > > > > repo
>> >> > > > > > > or a copy of that with required changes exist in bigtop
>> repo
>> >> > until
>> >> > > we
>> >> > > > > > make
>> >> > > > > > > the dtcli generic enough.
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > > Can we have these file paths set using environment variables
>> >> with
>> >> > > some
>> >> > > > > > default values in dtcli?
>> >> > > > > > Then we can set such params in dt-env.sh.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > > 3. dt-sited.xml -> How is this file different from
>> >> dt-site.xml?
>> >> > Not
>> >> > > > > sure
>> >> > > > > > if
>> >> > > > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel
>> >> are
>> >> > > > copied
>> >> > > > > to
>> >> > > > > > > home of new user when useradd program is called. But for
>> >> existing
>> >> > > > users
>> >> > > > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on
>> >> the
>> >> > fly
>> >> > > > if
>> >> > > > > > not
>> >> > > > > > > encountered for the first time. Again correct me if I'm
>> wrong.
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > > Yes, you are right. However this would probably be necessary
>> >> step
>> >> > for
>> >> > > > > > rpm/deb, as they may also get installed during OS install and
>> >> > before
>> >> > > > user
>> >> > > > > > accounts were created.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
>> best
>> >> > > idea.
>> >> > > > > > We'll
>> >> > > > > > > anyway put dtcli in location which is by default present in
>> >> path
>> >> > > i.e.
>> >> > > > > > > /usr/bin.
>> >> > > > > >
>> >> > > > > > For other env variables specific to apex, I'll prefer to use
>> >> > > dt-env.sh
>> >> > > > > and
>> >> > > > > > > source it in dtcli rather than changing bashrc etc...
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the
>> >> cleaner
>> >> > way
>> >> > > > to
>> >> > > > > > achieve the same.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > >
>> >> > > > > > Thanks,
>> >> > > > > > > Chinmay.
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
>> >> > > > > > > aniruddha@datatorrent.com> wrote:
>> >> > > > > > >
>> >> > > > > > > > Hi,
>> >> > > > > > > >
>> >> > > > > > > > @Chinmay,
>> >> > > > > > > >
>> >> > > > > > > > We need to consider following:
>> >> > > > > > > > System wide default config files can be located at
>> following
>> >> > > > > locations:
>> >> > > > > > > >
>> >> > > > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli
>> behaviour
>> >> on
>> >> > > how
>> >> > > > it
>> >> > > > > > > finds
>> >> > > > > > > > those locations)
>> >> > > > > > > > /etc/apex/dt-site.xml (we may have to change dtcli
>> >> behaviour on
>> >> > > how
>> >> > > > > it
>> >> > > > > > > > finds those locations)
>> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users
>> >> that
>> >> > > will
>> >> > > > > be
>> >> > > > > > > > created in system in future.)
>> >> > > > > > > >
>> >> > > > > > > > We may also have to modify bashrc / bashprofile for
>> >> population
>> >> > > any
>> >> > > > > > > > variables that are required.
>> >> > > > > > > >
>> >> > > > > > > > @dev, please put in your comments / suggestion.
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Thanks,
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Aniruddha
>> >> > > > > > > >
>> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
>> >> > > > > thomas@datatorrent.com>
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > -->
>> >> > > > > > > > >
>> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
>> >> > > > > > > > chinmay@datatorrent.com
>> >> > > > > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > > > Hello Everyone!!
>> >> > > > > > > > > >
>> >> > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
>> here
>> >> > are
>> >> > > > some
>> >> > > > > > > > > proposals
>> >> > > > > > > > > > about package structure etc..
>> >> > > > > > > > > > Before posting it on bbigtop mailing list, I have
>> some
>> >> > > question
>> >> > > > > for
>> >> > > > > > > > apex
>> >> > > > > > > > > > community.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Proposed Directory structure of apex package for both
>> >> deb &
>> >> > > > rpm:
>> >> > > > > > > > > >
>> >> > > > > > > > > > /usr/lib/apex/bin/dtcli
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
>> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars>
>> >> > > > > > > > >
>> >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
>> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
>> >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
>> >> > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
>> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG
>> >> > > > > > > > > > /usr/share/doc/apex/NOTICE
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > <version> = 3.3.0-incubating.
>> >> > > > > > > > > > <other dependent jars> = All the 3rd party jars which
>> >> are
>> >> > > > > required
>> >> > > > > > > for
>> >> > > > > > > > > apex
>> >> > > > > > > > > > to run. Usually the dependencies are packaged as
>> part of
>> >> > > > rpm/deb
>> >> > > > > by
>> >> > > > > > > any
>> >> > > > > > > > > > software in bigtop.
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Can you specify what those jars are in Bigtop context?
>> >> Same
>> >> > as
>> >> > > > > > shipped
>> >> > > > > > > > with
>> >> > > > > > > > > DT community addition under lib/ ?
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd
>> >> party
>> >> > > jars
>> >> > > > > > which
>> >> > > > > > > > > needs
>> >> > > > > > > > > > to included while packaging.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Questions related to this:
>> >> > > > > > > > > > 1. Should we call the cli of apex as "apex" instead
>> of
>> >> > > "dtcli"
>> >> > > > in
>> >> > > > > > > > bigtop
>> >> > > > > > > > > > package?
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > I think we should keep the name until we are able to
>> >> change
>> >> > it
>> >> > > in
>> >> > > > > > Apex.
>> >> > > > > > > > > Otherwise this may get confusing.
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > 2. I see that all softwares in bigtop have man page
>> for
>> >> > their
>> >> > > > > > > > > executables.
>> >> > > > > > > > > > I think we should have it too for dtcli. Is there any
>> >> > > > > documentation
>> >> > > > > > > > > which I
>> >> > > > > > > > > > can convert to man page? or can I use output of
>> "dtcli
>> >> > > --help"?
>> >> > > > > > > > > > 3. Do we want to call version of apex in Bigtop as
>> >> 3.3.0 OR
>> >> > > > > > > > > > 3.3.0-incubating?
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Has to be -incubating
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on
>> >> 2.7.1
>> >> > > > > version
>> >> > > > > > of
>> >> > > > > > > > > > bigtop hadoop? Any problems that we see with this
>> >> > dependency?
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Should work without changed. I thought we certified
>> >> against
>> >> > > > 2.7.0?
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > > 5. Following is the apache mirror from which bigtop
>> will
>> >> > pick
>> >> > > > the
>> >> > > > > > > apex
>> >> > > > > > > > > > source for compilation and packaging. Please correct
>> if
>> >> > > > > incorrect:
>> >> > > > > > > > > >
>> >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
>> >> > > > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
>> all
>> >> the
>> >> > > > > > softwares
>> >> > > > > > > in
>> >> > > > > > > > > > bigtop.
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > Please share your opinion.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Thanks,
>> >> > > > > > > > > > Chinmay.
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
>> >> > > > > > > > > > chinmay@datatorrent.com>
>> >> > > > > > > > > > wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > > > Yes.. Starting to work on the packaging.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > I've already started discussion on bigtop dev
>> mailing
>> >> > list
>> >> > > > for
>> >> > > > > > > > > > > integration. Also created a Jira for the same. For
>> >> this
>> >> > > > > > communities
>> >> > > > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
>> >> > > > > > > thomas@datatorrent.com
>> >> > > > > > > > >
>> >> > > > > > > > > > > wrote:
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >> Chinmay,
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> I don't see anything under prerequisites that
>> would
>> >> be a
>> >> > > > > > problem.
>> >> > > > > > > We
>> >> > > > > > > > > > >> looked
>> >> > > > > > > > > > >> at the ASF licencing compatibility as part of
>> >> becoming
>> >> > an
>> >> > > > > > > incubator
>> >> > > > > > > > > > >> project.
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> Please focus on the packaging during the next
>> weeks.
>> >> > Since
>> >> > > > the
>> >> > > > > > > work
>> >> > > > > > > > > will
>> >> > > > > > > > > > >> be
>> >> > > > > > > > > > >> part of Bigtop, related discussions and JIRAs
>> should
>> >> > also
>> >> > > be
>> >> > > > > > > there.
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> Would be good to have the packaging in place by
>> end
>> >> Feb.
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> Thanks,
>> >> > > > > > > > > > >> Thomas
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
>> Kolhatkar <
>> >> > > > > > > > > > >> chinmay@datatorrent.com>
>> >> > > > > > > > > > >> wrote:
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >> > Hi All,
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > We're planning a work on adding Apache Apex as a
>> >> > > component
>> >> > > > > to
>> >> > > > > > > > Apache
>> >> > > > > > > > > > >> > Bigtop.
>> >> > > > > > > > > > >> > Bigtop is the packaging system for the Apache
>> big
>> >> data
>> >> > > > > > > ecosystem.
>> >> > > > > > > > > > >> Several
>> >> > > > > > > > > > >> > Hadoop distros use it, most recently EMR.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Here is the tracking Jira task in APEXCORE for
>> the
>> >> > same:
>> >> > > > > > > > > > >> >
>> https://issues.apache.org/jira/browse/APEXCORE-331
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Proposed plan of execution is as follows:
>> >> > > > > > > > > > >> > *Step 1) Handle prerequisites*
>> >> > > > > > > > > > >> > Apache bigtop has some hard and soft expectation
>> >> for
>> >> > new
>> >> > > > > > > > components
>> >> > > > > > > > > to
>> >> > > > > > > > > > >> get
>> >> > > > > > > > > > >> > integrated into Bigtop.
>> >> > > > > > > > > > >> > Here is the list of it:
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >>
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Most of them seems to be standard ASF based
>> >> > > requirements,
>> >> > > > > but
>> >> > > > > > > few
>> >> > > > > > > > > need
>> >> > > > > > > > > > >> to
>> >> > > > > > > > > > >> > be checked for:
>> >> > > > > > > > > > >> > 1. Software projects are expected to be Licensed
>> >> under
>> >> > > > > Apache
>> >> > > > > > > > > License,
>> >> > > > > > > > > > >> > Version 2.0 (and their dependencies are expected
>> >> to be
>> >> > > > > > > compatible
>> >> > > > > > > > > with
>> >> > > > > > > > > > >> this
>> >> > > > > > > > > > >> > license)
>> >> > > > > > > > > > >> >     - Apex is under ASL 2.0 but need to check if
>> >> > > > > dependencies
>> >> > > > > > of
>> >> > > > > > > > > Apex
>> >> > > > > > > > > > >> are
>> >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
>> >> > > > > verification
>> >> > > > > > > > check.
>> >> > > > > > > > > > >> > 2. Software projects are expected to be
>> compatible
>> >> > with
>> >> > > > all
>> >> > > > > of
>> >> > > > > > > the
>> >> > > > > > > > > > >> > supported platforms that Bigtop distribution is
>> >> > > targeting
>> >> > > > > > > > > > >> >     - This needs verifying whether our software
>> >> runs
>> >> > > fine
>> >> > > > in
>> >> > > > > > > > > centos-6
>> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
>> >> > opensuse-13.2.
>> >> > > > > > > > > > >> > 3. What smoke tests that should be added for
>> >> > deployment.
>> >> > > > > > > > > > >> > 4. Identifying the test artifacts which goes
>> beyond
>> >> > > smoke
>> >> > > > > test
>> >> > > > > > > > > > >> >     - These are basically the integration tests
>> for
>> >> > > > > > verification
>> >> > > > > > > > > after
>> >> > > > > > > > > > >> the
>> >> > > > > > > > > > >> > deployment. This is a soft requirement, but aim
>> is
>> >> to
>> >> > > > > achieve
>> >> > > > > > > this
>> >> > > > > > > > > as
>> >> > > > > > > > > > >> well
>> >> > > > > > > > > > >> > or at least have explanation why not to include.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > If there are any from the link which explicitly
>> >> needs
>> >> > to
>> >> > > > be
>> >> > > > > > > > checked
>> >> > > > > > > > > > >> other
>> >> > > > > > > > > > >> > than above 4, please let us know.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
>> >> > > > > > > > > > >> > From one of the mail archive of Bigtop, it was
>> >> learnt
>> >> > > that
>> >> > > > > the
>> >> > > > > > > > > bigtop
>> >> > > > > > > > > > >> > community want to see the addition of new
>> >> components
>> >> > in
>> >> > > > > > phases.
>> >> > > > > > > > Here
>> >> > > > > > > > > > are
>> >> > > > > > > > > > >> > the phases:
>> >> > > > > > > > > > >> > 1. Packaging
>> >> > > > > > > > > > >> >     - This needs creating of package i.e. rpm &
>> deb
>> >> > > files.
>> >> > > > > > > > > > >> >     - documentations/READMEs, LICENSE,
>> DISCLAMER,
>> >> > NOTES
>> >> > > > etc
>> >> > > > > if
>> >> > > > > > > any
>> >> > > > > > > > > > >> needed.
>> >> > > > > > > > > > >> >     - Any documentation that need to be added to
>> >> > > > > distribution
>> >> > > > > > of
>> >> > > > > > > > our
>> >> > > > > > > > > > >> > software.
>> >> > > > > > > > > > >> >     - Any license information of dependencies
>> >> required
>> >> > > to
>> >> > > > be
>> >> > > > > > > added
>> >> > > > > > > > > to
>> >> > > > > > > > > > >> > package
>> >> > > > > > > > > > >> > 2. Smoke tests (at very least)
>> >> > > > > > > > > > >> >     - Adding smoke test for packaging.
>> >> > > > > > > > > > >> > 3. Puppet recipes for automatic deployment and
>> >> > > > configuration
>> >> > > > > > > > > > >> >     - Add puppet recipes for automatic
>> deployment
>> >> > > > > > > > > > >> > 4. Integration tests
>> >> > > > > > > > > > >> >     - For verification of deployments.
>> >> > > > > > > > > > >> > 5. license clearance:
>> >> > > > > > > > > > >> >     Run 'gradle rat' to make sure all new stuff
>> is
>> >> > > > compliant
>> >> > > > > > > with
>> >> > > > > > > > > ASF
>> >> > > > > > > > > > >> > license requirements. If you add code licenses
>> >> under
>> >> > > > > different
>> >> > > > > > > > > > licenses,
>> >> > > > > > > > > > >> > those would need to be listed in the NOTICE.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Please share your thoughts on the approach.
>> >> > > > > > > > > > >> > We'll start corresponding communication on
>> bigtop
>> >> > > mailing
>> >> > > > > list
>> >> > > > > > > as
>> >> > > > > > > > > > well.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > We have some specific questions/suggestions
>> >> related to
>> >> > > > what
>> >> > > > > > > should
>> >> > > > > > > > > be
>> >> > > > > > > > > > >> the
>> >> > > > > > > > > > >> > content of the package and what should be the
>> smoke
>> >> > > tests,
>> >> > > > > but
>> >> > > > > > > in
>> >> > > > > > > > > the
>> >> > > > > > > > > > >> > interest of not having too much content here,
>> we'll
>> >> > put
>> >> > > > the
>> >> > > > > > > > > questions
>> >> > > > > > > > > > >> as a
>> >> > > > > > > > > > >> > separate mail in this mailthread.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >> > Thanks,
>> >> > > > > > > > > > >> > Chinmay.
>> >> > > > > > > > > > >> >
>> >> > > > > > > > > > >>
>> >> > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message