apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep A. Dalvi" <apa...@pradeepdalvi.com>
Subject Re: Apache Apex as component in Apache bigtop
Date Wed, 10 Feb 2016 14:23:12 GMT
Inline comments...

On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <chinmay@datatorrent.com>
wrote:

> @Thomas, Not all the jar present in DT community edition will be included
> there.
>
> Here is the list of jars I found from common dependencies between of apex
> and DT Community edition:
> netlet-1.2.0.jar
> kryo-2.24.0.jar
> jackson-core-asl-1.9.2.jar
> jackson-mapper-asl-1.9.2.jar
> async-http-client-1.7.20.jar
> netty-3.6.6.Final.jar
> validation-api-1.1.0.Final.jar
> bval-jsr303-0.5.jar
> bval-core-0.5.jar
> commons-lang3-3.1.jar
> commons-beanutils-1.8.3.jar
> httpclient-4.3.5.jar
> commons-codec-1.10.jar
> zip4j-1.3.2.jar
> jetty-websocket-8.1.10.v20130312.jar
> xbean-asm5-shaded-4.3.jar
> jersey-apache-client4-1.9.jar
> jline-2.11.jar
> ant-1.9.2.jar
> ant-launcher-1.9.2.jar
> mbassador-1.1.9.jar
> jackson-jaxrs-1.9.2.jar
> jackson-xc-1.9.2.jar
> hadoop-common-2.2.0-tests.jar
>
> Ofcourse, I'll be running some tests do check that dtcli works properly for
> launch+shutdown+kill of apps with only these libraries present in isolation
> without dependency on local m2.
> I'm believe that there are unwanted jars which are used for compile time
> dependency and not runtime in above list which I can drop to keep package
> size minimal.
>
>
> @Anniruddha,
> 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only
> required values I see in it are classpath. and dtcli is capable of building
> it on the fly.
>
> 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar path,
> dt-env.sh and all such conf files. If we need a change in dtcli, then
> community's opinion is required for whether dtcli should change in our repo
> or a copy of that with required changes exist in bigtop repo until we make
> the dtcli generic enough.
>

Can we have these file paths set using environment variables with some
default values in dtcli?
Then we can set such params in dt-env.sh.


> 3. dt-sited.xml -> How is this file different from dt-site.xml? Not sure if
> adding in /etc/skel would help. Files & Dirs from /etc/skel are copied to
> home of new user when useradd program is called. But for existing users
> this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if not
> encountered for the first time. Again correct me if I'm wrong.
>

Yes, you are right. However this would probably be necessary step for
rpm/deb, as they may also get installed during OS install and before user
accounts were created.


> 4. Changing bashrc & bash_profile -> I'm not sure its the best idea. We'll
> anyway put dtcli in location which is by default present in path i.e.
> /usr/bin.

For other env variables specific to apex, I'll prefer to use dt-env.sh and
> source it in dtcli rather than changing bashrc etc...


Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way to
achieve the same.


>
Thanks,
> Chinmay.
>
>
>
> On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > Hi,
> >
> > @Chinmay,
> >
> > We need to consider following:
> > System wide default config files can be located at following locations:
> >
> > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it
> finds
> > those locations)
> > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it
> > finds those locations)
> > /etc/skel/.dt/dt-sited.xml and other files (for new users that will be
> > created in system in future.)
> >
> > We may also have to modify bashrc / bashprofile for population any
> > variables that are required.
> >
> > @dev, please put in your comments / suggestion.
> >
> >
> >
> > Thanks,
> >
> >
> > Aniruddha
> >
> > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <thomas@datatorrent.com>
> > wrote:
> >
> > > -->
> > >
> > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > Hello Everyone!!
> > > >
> > > > Continuing with packaging effort (rpm+deb) of apex, here are some
> > > proposals
> > > > about package structure etc..
> > > > Before posting it on bbigtop mailing list, I have some question for
> > apex
> > > > community.
> > > >
> > > > Proposed Directory structure of apex package for both deb & rpm:
> > > >
> > > > /usr/lib/apex/bin/dtcli
> > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > /usr/lib/apex/lib/<other dependent jars>
> > >
> > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > /usr/share/doc/apex/CHANGELOG
> > > > /usr/share/doc/apex/NOTICE
> > > >
> > > >
> > > > <version> = 3.3.0-incubating.
> > > > <other dependent jars> = All the 3rd party jars which are required
> for
> > > apex
> > > > to run. Usually the dependencies are packaged as part of rpm/deb by
> any
> > > > software in bigtop.
> > > >
> > >
> > > Can you specify what those jars are in Bigtop context? Same as shipped
> > with
> > > DT community addition under lib/ ?
> > >
> > >
> > > > <package LICENSE> = Licenses of corresponding of 3rd party jars
which
> > > needs
> > > > to included while packaging.
> > > >
> > > > Questions related to this:
> > > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in
> > bigtop
> > > > package?
> > > >
> > >
> > > I think we should keep the name until we are able to change it in Apex.
> > > Otherwise this may get confusing.
> > >
> > >
> > > > 2. I see that all softwares in bigtop have man page for their
> > > executables.
> > > > I think we should have it too for dtcli. Is there any documentation
> > > which I
> > > > can convert to man page? or can I use output of "dtcli --help"?
> > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > 3.3.0-incubating?
> > > >
> > >
> > > Has to be -incubating
> > >
> > >
> > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
> > > > bigtop hadoop? Any problems that we see with this dependency?
> > > >
> > >
> > > Should work without changed. I thought we certified against 2.7.0?
> > >
> > >
> > > > 5. Following is the apache mirror from which bigtop will pick the
> apex
> > > > source for compilation and packaging. Please correct if incorrect:
> > > >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > >     NOTE: apache.osuosl.org is the mirror used by all the softwares
> in
> > > > bigtop.
> > > >
> > > >
> > > > Please share your opinion.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Yes.. Starting to work on the packaging.
> > > > >
> > > > > I've already started discussion on bigtop dev mailing list for
> > > > > integration. Also created a Jira for the same. For this communities
> > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> thomas@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Chinmay,
> > > > >>
> > > > >> I don't see anything under prerequisites that would be a problem.
> We
> > > > >> looked
> > > > >> at the ASF licencing compatibility as part of becoming an
> incubator
> > > > >> project.
> > > > >>
> > > > >> Please focus on the packaging during the next weeks. Since the
> work
> > > will
> > > > >> be
> > > > >> part of Bigtop, related discussions and JIRAs should also be
> there.
> > > > >>
> > > > >> Would be good to have the packaging in place by end Feb.
> > > > >>
> > > > >> Thanks,
> > > > >> Thomas
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > >> chinmay@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi All,
> > > > >> >
> > > > >> > We're planning a work on adding Apache Apex as a component
to
> > Apache
> > > > >> > Bigtop.
> > > > >> > Bigtop is the packaging system for the Apache big data
> ecosystem.
> > > > >> Several
> > > > >> > Hadoop distros use it, most recently EMR.
> > > > >> >
> > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > >> >
> > > > >> > Proposed plan of execution is as follows:
> > > > >> > *Step 1) Handle prerequisites*
> > > > >> > Apache bigtop has some hard and soft expectation for new
> > components
> > > to
> > > > >> get
> > > > >> > integrated into Bigtop.
> > > > >> > Here is the list of it:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > >> >
> > > > >> > Most of them seems to be standard ASF based requirements,
but
> few
> > > need
> > > > >> to
> > > > >> > be checked for:
> > > > >> > 1. Software projects are expected to be Licensed under Apache
> > > License,
> > > > >> > Version 2.0 (and their dependencies are expected to be
> compatible
> > > with
> > > > >> this
> > > > >> > license)
> > > > >> >     - Apex is under ASL 2.0 but need to check if dependencies
of
> > > Apex
> > > > >> are
> > > > >> > compatible with ASL 2.0. This I guess would be a verification
> > check.
> > > > >> > 2. Software projects are expected to be compatible with
all of
> the
> > > > >> > supported platforms that Bigtop distribution is targeting
> > > > >> >     - This needs verifying whether our software runs fine
in
> > > centos-6
> > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > >> > 3. What smoke tests that should be added for deployment.
> > > > >> > 4. Identifying the test artifacts which goes beyond smoke
test
> > > > >> >     - These are basically the integration tests for verification
> > > after
> > > > >> the
> > > > >> > deployment. This is a soft requirement, but aim is to achieve
> this
> > > as
> > > > >> well
> > > > >> > or at least have explanation why not to include.
> > > > >> >
> > > > >> > If there are any from the link which explicitly needs to
be
> > checked
> > > > >> other
> > > > >> > than above 4, please let us know.
> > > > >> >
> > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > >> > From one of the mail archive of Bigtop, it was learnt that
the
> > > bigtop
> > > > >> > community want to see the addition of new components in
phases.
> > Here
> > > > are
> > > > >> > the phases:
> > > > >> > 1. Packaging
> > > > >> >     - This needs creating of package i.e. rpm & deb
files.
> > > > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES
etc if
> any
> > > > >> needed.
> > > > >> >     - Any documentation that need to be added to distribution
of
> > our
> > > > >> > software.
> > > > >> >     - Any license information of dependencies required to
be
> added
> > > to
> > > > >> > package
> > > > >> > 2. Smoke tests (at very least)
> > > > >> >     - Adding smoke test for packaging.
> > > > >> > 3. Puppet recipes for automatic deployment and configuration
> > > > >> >     - Add puppet recipes for automatic deployment
> > > > >> > 4. Integration tests
> > > > >> >     - For verification of deployments.
> > > > >> > 5. license clearance:
> > > > >> >     Run 'gradle rat' to make sure all new stuff is compliant
> with
> > > ASF
> > > > >> > license requirements. If you add code licenses under different
> > > > licenses,
> > > > >> > those would need to be listed in the NOTICE.
> > > > >> >
> > > > >> > Please share your thoughts on the approach.
> > > > >> > We'll start corresponding communication on bigtop mailing
list
> as
> > > > well.
> > > > >> >
> > > > >> > We have some specific questions/suggestions related to what
> should
> > > be
> > > > >> the
> > > > >> > content of the package and what should be the smoke tests,
but
> in
> > > the
> > > > >> > interest of not having too much content here, we'll put
the
> > > questions
> > > > >> as a
> > > > >> > separate mail in this mailthread.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Chinmay.
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message