Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CABAipVbukmrJbv+7HYUEsQM2r773uox3=6SJ+2cvLzMh9BTptQ@mail.gmail.com>
References: 
 <CABAipVbE5v-6iOwR=Porg1w51hwQ-nXwPZkAzqx3dfKfry1ueQ@mail.gmail.com>
	<CAKJfLDMYNwKB7SesG88V0pH9-8sx+hS6znBMJU2UDAZKjDNYYw@mail.gmail.com>
	<CABAipVbukmrJbv+7HYUEsQM2r773uox3=6SJ+2cvLzMh9BTptQ@mail.gmail.com>
Date: Tue, 9 Feb 2016 22:35:30 +0530
Message-ID: 
 <CABAipVZA=YoREzuoUXnUuorNesG2uwBoWn09wuA=fBV3wp6NAg@mail.gmail.com>
Subject: Re: Apache Apex as component in Apache bigtop
From: Chinmay Kolhatkar <chinmay@datatorrent.com>
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c1274c22c9ac052b5953a0

--001a11c1274c22c9ac052b5953a0
Content-Type: text/plain; charset=UTF-8

Hello Everyone!!

Continuing with packaging effort (rpm+deb) of apex, here are some proposals
about package structure etc..
Before posting it on bbigtop mailing list, I have some question for apex
community.

Proposed Directory structure of apex package for both deb & rpm:

/usr/lib/apex/bin/dtcli
/usr/lib/apex/lib/apex-api-<version>.jar
/usr/lib/apex/lib/apex-engine-<version>.jar
/usr/lib/apex/lib/apex-bufferserver-<version>.jar
/usr/lib/apex/lib/apex-common-<version>.jar
/usr/lib/apex/lib/<other dependent jars>
/usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
/usr/share/doc/man/man1/dtcli.1.gz
/usr/share/doc/apex/license/LICENSE.txt
/usr/share/doc/apex/license/<package>-LICENSE.txt
/usr/share/doc/apex/CHANGELOG
/usr/share/doc/apex/NOTICE


<version> = 3.3.0-incubating.
<other dependent jars> = All the 3rd party jars which are required for apex
to run. Usually the dependencies are packaged as part of rpm/deb by any
software in bigtop.
<package LICENSE> = Licenses of corresponding of 3rd party jars which needs
to included while packaging.

Questions related to this:
1. Should we call the cli of apex as "apex" instead of "dtcli" in bigtop
package?
2. I see that all softwares in bigtop have man page for their executables.
I think we should have it too for dtcli. Is there any documentation which I
can convert to man page? or can I use output of "dtcli --help"?
3. Do we want to call version of apex in Bigtop as 3.3.0 OR
3.3.0-incubating?
4. Is it ok for apex package of bigtop to depend on 2.7.1  version of
bigtop hadoop? Any problems that we see with this dependency?
5. Following is the apache mirror from which bigtop will pick the apex
source for compilation and packaging. Please correct if incorrect:
    http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
    NOTE: apache.osuosl.org is the mirror used by all the softwares in
bigtop.


Please share your opinion.

Thanks,
Chinmay.


On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <chinmay@datatorrent.com>
wrote:

> Yes.. Starting to work on the packaging.
>
> I've already started discussion on bigtop dev mailing list for
> integration. Also created a Jira for the same. For this communities
> reference, here is the bigtop Jira: BIGTOP-2313.
>
>
>
> On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <thomas@datatorrent.com>
> wrote:
>
>> Chinmay,
>>
>> I don't see anything under prerequisites that would be a problem. We
>> looked
>> at the ASF licencing compatibility as part of becoming an incubator
>> project.
>>
>> Please focus on the packaging during the next weeks. Since the work will
>> be
>> part of Bigtop, related discussions and JIRAs should also be there.
>>
>> Would be good to have the packaging in place by end Feb.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
>> chinmay@datatorrent.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > We're planning a work on adding Apache Apex as a component to Apache
>> > Bigtop.
>> > Bigtop is the packaging system for the Apache big data ecosystem.
>> Several
>> > Hadoop distros use it, most recently EMR.
>> >
>> > Here is the tracking Jira task in APEXCORE for the same:
>> > https://issues.apache.org/jira/browse/APEXCORE-331
>> >
>> > Proposed plan of execution is as follows:
>> > *Step 1) Handle prerequisites*
>> > Apache bigtop has some hard and soft expectation for new components to
>> get
>> > integrated into Bigtop.
>> > Here is the list of it:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
>> >
>> > Most of them seems to be standard ASF based requirements, but few need
>> to
>> > be checked for:
>> > 1. Software projects are expected to be Licensed under Apache License,
>> > Version 2.0 (and their dependencies are expected to be compatible with
>> this
>> > license)
>> >     - Apex is under ASL 2.0 but need to check if dependencies of Apex
>> are
>> > compatible with ASL 2.0. This I guess would be a verification check.
>> > 2. Software projects are expected to be compatible with all of the
>> > supported platforms that Bigtop distribution is targeting
>> >     - This needs verifying whether our software runs fine in centos-6
>> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
>> > 3. What smoke tests that should be added for deployment.
>> > 4. Identifying the test artifacts which goes beyond smoke test
>> >     - These are basically the integration tests for verification after
>> the
>> > deployment. This is a soft requirement, but aim is to achieve this as
>> well
>> > or at least have explanation why not to include.
>> >
>> > If there are any from the link which explicitly needs to be checked
>> other
>> > than above 4, please let us know.
>> >
>> > *Step 2) Adding Apex as component to Bigtop*
>> > From one of the mail archive of Bigtop, it was learnt that the bigtop
>> > community want to see the addition of new components in phases. Here are
>> > the phases:
>> > 1. Packaging
>> >     - This needs creating of package i.e. rpm & deb files.
>> >     - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any
>> needed.
>> >     - Any documentation that need to be added to distribution of our
>> > software.
>> >     - Any license information of dependencies required to be added to
>> > package
>> > 2. Smoke tests (at very least)
>> >     - Adding smoke test for packaging.
>> > 3. Puppet recipes for automatic deployment and configuration
>> >     - Add puppet recipes for automatic deployment
>> > 4. Integration tests
>> >     - For verification of deployments.
>> > 5. license clearance:
>> >     Run 'gradle rat' to make sure all new stuff is compliant with ASF
>> > license requirements. If you add code licenses under different licenses,
>> > those would need to be listed in the NOTICE.
>> >
>> > Please share your thoughts on the approach.
>> > We'll start corresponding communication on bigtop mailing list as well.
>> >
>> > We have some specific questions/suggestions related to what should be
>> the
>> > content of the package and what should be the smoke tests, but in the
>> > interest of not having too much content here, we'll put the questions
>> as a
>> > separate mail in this mailthread.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>>
>
>

--001a11c1274c22c9ac052b5953a0--