apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniruddha Thombare <anirud...@datatorrent.com>
Subject Re: Apache Apex as component in Apache bigtop
Date Wed, 10 Feb 2016 07:37:52 GMT
Hi,

@Chinmay,

We need to consider following:
System wide default config files can be located at following locations:

/etc/apex/dt-env.sh  (we may have to change dtcli behaviour on how it finds
those locations)
/etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it
finds those locations)
/etc/skel/.dt/dt-sited.xml and other files (for new users that will be
created in system in future.)

We may also have to modify bashrc / bashprofile for population any
variables that are required.

@dev, please put in your comments / suggestion.



Thanks,


Aniruddha

On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <thomas@datatorrent.com>
wrote:

> -->
>
> On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <chinmay@datatorrent.com
> >
> wrote:
>
> > Hello Everyone!!
> >
> > Continuing with packaging effort (rpm+deb) of apex, here are some
> proposals
> > about package structure etc..
> > Before posting it on bbigtop mailing list, I have some question for apex
> > community.
> >
> > Proposed Directory structure of apex package for both deb & rpm:
> >
> > /usr/lib/apex/bin/dtcli
> > /usr/lib/apex/lib/apex-api-<version>.jar
> > /usr/lib/apex/lib/apex-engine-<version>.jar
> > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > /usr/lib/apex/lib/apex-common-<version>.jar
> > /usr/lib/apex/lib/<other dependent jars>
>
> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > /usr/share/doc/man/man1/dtcli.1.gz
> > /usr/share/doc/apex/license/LICENSE.txt
> > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > /usr/share/doc/apex/CHANGELOG
> > /usr/share/doc/apex/NOTICE
> >
> >
> > <version> = 3.3.0-incubating.
> > <other dependent jars> = All the 3rd party jars which are required for
> apex
> > to run. Usually the dependencies are packaged as part of rpm/deb by any
> > software in bigtop.
> >
>
> Can you specify what those jars are in Bigtop context? Same as shipped with
> DT community addition under lib/ ?
>
>
> > <package LICENSE> = Licenses of corresponding of 3rd party jars which
> needs
> > to included while packaging.
> >
> > Questions related to this:
> > 1. Should we call the cli of apex as "apex" instead of "dtcli" in bigtop
> > package?
> >
>
> I think we should keep the name until we are able to change it in Apex.
> Otherwise this may get confusing.
>
>
> > 2. I see that all softwares in bigtop have man page for their
> executables.
> > I think we should have it too for dtcli. Is there any documentation
> which I
> > can convert to man page? or can I use output of "dtcli --help"?
> > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > 3.3.0-incubating?
> >
>
> Has to be -incubating
>
>
> > 4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
> > bigtop hadoop? Any problems that we see with this dependency?
> >
>
> Should work without changed. I thought we certified against 2.7.0?
>
>
> > 5. Following is the apache mirror from which bigtop will pick the apex
> > source for compilation and packaging. Please correct if incorrect:
> >     http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> >     NOTE: apache.osuosl.org is the mirror used by all the softwares in
> > bigtop.
> >
> >
> > Please share your opinion.
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> >
> >
> > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Yes.. Starting to work on the packaging.
> > >
> > > I've already started discussion on bigtop dev mailing list for
> > > integration. Also created a Jira for the same. For this communities
> > > reference, here is the bigtop Jira: BIGTOP-2313.
> > >
> > >
> > >
> > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <thomas@datatorrent.com>
> > > wrote:
> > >
> > >> Chinmay,
> > >>
> > >> I don't see anything under prerequisites that would be a problem. We
> > >> looked
> > >> at the ASF licencing compatibility as part of becoming an incubator
> > >> project.
> > >>
> > >> Please focus on the packaging during the next weeks. Since the work
> will
> > >> be
> > >> part of Bigtop, related discussions and JIRAs should also be there.
> > >>
> > >> Would be good to have the packaging in place by end Feb.
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >>
> > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > >> chinmay@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > We're planning a work on adding Apache Apex as a component to Apache
> > >> > Bigtop.
> > >> > Bigtop is the packaging system for the Apache big data ecosystem.
> > >> Several
> > >> > Hadoop distros use it, most recently EMR.
> > >> >
> > >> > Here is the tracking Jira task in APEXCORE for the same:
> > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > >> >
> > >> > Proposed plan of execution is as follows:
> > >> > *Step 1) Handle prerequisites*
> > >> > Apache bigtop has some hard and soft expectation for new components
> to
> > >> get
> > >> > integrated into Bigtop.
> > >> > Here is the list of it:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > >> >
> > >> > Most of them seems to be standard ASF based requirements, but few
> need
> > >> to
> > >> > be checked for:
> > >> > 1. Software projects are expected to be Licensed under Apache
> License,
> > >> > Version 2.0 (and their dependencies are expected to be compatible
> with
> > >> this
> > >> > license)
> > >> >     - Apex is under ASL 2.0 but need to check if dependencies of
> Apex
> > >> are
> > >> > compatible with ASL 2.0. This I guess would be a verification check.
> > >> > 2. Software projects are expected to be compatible with all of the
> > >> > supported platforms that Bigtop distribution is targeting
> > >> >     - This needs verifying whether our software runs fine in
> centos-6
> > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > >> > 3. What smoke tests that should be added for deployment.
> > >> > 4. Identifying the test artifacts which goes beyond smoke test
> > >> >     - These are basically the integration tests for verification
> after
> > >> the
> > >> > deployment. This is a soft requirement, but aim is to achieve this
> as
> > >> well
> > >> > or at least have explanation why not to include.
> > >> >
> > >> > If there are any from the link which explicitly needs to be checked
> > >> other
> > >> > than above 4, please let us know.
> > >> >
> > >> > *Step 2) Adding Apex as component to Bigtop*
> > >> > From one of the mail archive of Bigtop, it was learnt that the
> bigtop
> > >> > community want to see the addition of new components in phases. Here
> > are
> > >> > the phases:
> > >> > 1. Packaging
> > >> >     - This needs creating of package i.e. rpm & deb files.
> > >> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
> > >> needed.
> > >> >     - Any documentation that need to be added to distribution of our
> > >> > software.
> > >> >     - Any license information of dependencies required to be added
> to
> > >> > package
> > >> > 2. Smoke tests (at very least)
> > >> >     - Adding smoke test for packaging.
> > >> > 3. Puppet recipes for automatic deployment and configuration
> > >> >     - Add puppet recipes for automatic deployment
> > >> > 4. Integration tests
> > >> >     - For verification of deployments.
> > >> > 5. license clearance:
> > >> >     Run 'gradle rat' to make sure all new stuff is compliant with
> ASF
> > >> > license requirements. If you add code licenses under different
> > licenses,
> > >> > those would need to be listed in the NOTICE.
> > >> >
> > >> > Please share your thoughts on the approach.
> > >> > We'll start corresponding communication on bigtop mailing list as
> > well.
> > >> >
> > >> > We have some specific questions/suggestions related to what should
> be
> > >> the
> > >> > content of the package and what should be the smoke tests, but in
> the
> > >> > interest of not having too much content here, we'll put the
> questions
> > >> as a
> > >> > separate mail in this mailthread.
> > >> >
> > >> > Thanks,
> > >> > Chinmay.
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message