apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Apache Apex as component in Apache bigtop
Date Wed, 10 Feb 2016 16:23:31 GMT
Hi Aniruddha,

I like the idea of skeleton directory. But I don't see any use of it,
unless we've anything in default.
If you have something in your mind which should be default, can you please
share it?

Thanks,
Chinmay.


On Wed, Feb 10, 2016 at 9:49 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> On point 2:
> Skeleton directory is for default home dir structure for future users.
> By avoiding that, we can't inconvenience users and admins.
> Defaults in that configuration can be specified or evaluated at install.
>
>
>
> On Wed, 10 Feb 2016 9:37 pm Chinmay Kolhatkar <chinmay@datatorrent.com>
> wrote:
>
> > Really good points Pradeep and Aniruddha.
> >
> > 1. I believe we won't need to change the dtcli considering it works with
> DT
> > Community edition. We can keep the directory structure similar to that.
> > dt-env.sh has variables which contains information required for dtcli to
> > launch.
> >
> > 2. Let me check with bigtop community that whether they facilitate the
> > installation of rpms/debs before user is created. In either case, current
> > dtcli creates a .dt folder in home directory. Also before putting
> anything
> > in /etc/skel we need to define what are the default contents that should
> go
> > to ~/.dt/ folder. If there is no defaults, probably we should not
> > explicitly add it in /etc/skel.
> >
> > 3.  /etc/profile.d approach looks nice. This way contents of dt-env.sh
> are
> > present as env variables. But I see a catch there. Adding dt-env.sh to
> > /etc/profile.d would make all the variables available at runtime all the
> > time. I feel a little skeptical about that. Maybe a possible collision
> can
> > occur with other application vars.
> > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > "~/.dt/dt-env.sh". So those variables are anyway available.
> >
> > @Aniruddha, the Jira and PR effort is happening at Bigtop
> Jira/repository.
> > Packaging related code usually goes there. (That's what all the
> components
> > in bigtop does).
> > Having said that, once a PR is created, I'll be sharing the link of PR
> > here, so that apex community as well can review it.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > aniruddha@datatorrent.com> wrote:
> >
> > > +1 on suggestions and approach.
> > > We may need to iron out details about exact paths etc.
> > > Which can be done on jira / PR comments.
> > > Is that right @dev?
> > >
> > >
> > >
> > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <apache@pradeepdalvi.com>
> > > wrote:
> > >
> > > > Inline comments...
> > > >
> > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > @Thomas, Not all the jar present in DT community edition will be
> > > included
> > > > > there.
> > > > >
> > > > > Here is the list of jars I found from common dependencies between
> of
> > > apex
> > > > > and DT Community edition:
> > > > > netlet-1.2.0.jar
> > > > > kryo-2.24.0.jar
> > > > > jackson-core-asl-1.9.2.jar
> > > > > jackson-mapper-asl-1.9.2.jar
> > > > > async-http-client-1.7.20.jar
> > > > > netty-3.6.6.Final.jar
> > > > > validation-api-1.1.0.Final.jar
> > > > > bval-jsr303-0.5.jar
> > > > > bval-core-0.5.jar
> > > > > commons-lang3-3.1.jar
> > > > > commons-beanutils-1.8.3.jar
> > > > > httpclient-4.3.5.jar
> > > > > commons-codec-1.10.jar
> > > > > zip4j-1.3.2.jar
> > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > xbean-asm5-shaded-4.3.jar
> > > > > jersey-apache-client4-1.9.jar
> > > > > jline-2.11.jar
> > > > > ant-1.9.2.jar
> > > > > ant-launcher-1.9.2.jar
> > > > > mbassador-1.1.9.jar
> > > > > jackson-jaxrs-1.9.2.jar
> > > > > jackson-xc-1.9.2.jar
> > > > > hadoop-common-2.2.0-tests.jar
> > > > >
> > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > properly
> > > > for
> > > > > launch+shutdown+kill of apps with only these libraries present in
> > > > isolation
> > > > > without dependency on local m2.
> > > > > I'm believe that there are unwanted jars which are used for compile
> > > time
> > > > > dependency and not runtime in above list which I can drop to keep
> > > package
> > > > > size minimal.
> > > > >
> > > > >
> > > > > @Anniruddha,
> > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> > Only
> > > > > required values I see in it are classpath. and dtcli is capable of
> > > > building
> > > > > it on the fly.
> > > > >
> > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the
jar
> > > path,
> > > > > dt-env.sh and all such conf files. If we need a change in dtcli,
> then
> > > > > community's opinion is required for whether dtcli should change in
> > our
> > > > repo
> > > > > or a copy of that with required changes exist in bigtop repo until
> we
> > > > make
> > > > > the dtcli generic enough.
> > > > >
> > > >
> > > > Can we have these file paths set using environment variables with
> some
> > > > default values in dtcli?
> > > > Then we can set such params in dt-env.sh.
> > > >
> > > >
> > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml?
Not
> > > sure
> > > > if
> > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > copied
> > > to
> > > > > home of new user when useradd program is called. But for existing
> > users
> > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly
> > if
> > > > not
> > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > >
> > > >
> > > > Yes, you are right. However this would probably be necessary step for
> > > > rpm/deb, as they may also get installed during OS install and before
> > user
> > > > accounts were created.
> > > >
> > > >
> > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the
best
> idea.
> > > > We'll
> > > > > anyway put dtcli in location which is by default present in path
> i.e.
> > > > > /usr/bin.
> > > >
> > > > For other env variables specific to apex, I'll prefer to use
> dt-env.sh
> > > and
> > > > > source it in dtcli rather than changing bashrc etc...
> > > >
> > > >
> > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way
> > to
> > > > achieve the same.
> > > >
> > > >
> > > > >
> > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > aniruddha@datatorrent.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > @Chinmay,
> > > > > >
> > > > > > We need to consider following:
> > > > > > System wide default config files can be located at following
> > > locations:
> > > > > >
> > > > > > /etc/apex/dt-env.sh  (we may have to change dtcli behaviour
on
> how
> > it
> > > > > finds
> > > > > > those locations)
> > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour
on
> how
> > > it
> > > > > > finds those locations)
> > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that
> will
> > > be
> > > > > > created in system in future.)
> > > > > >
> > > > > > We may also have to modify bashrc / bashprofile for population
> any
> > > > > > variables that are required.
> > > > > >
> > > > > > @dev, please put in your comments / suggestion.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >
> > > > > > Aniruddha
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > thomas@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > -->
> > > > > > >
> > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Everyone!!
> > > > > > > >
> > > > > > > > Continuing with packaging effort (rpm+deb) of apex,
here are
> > some
> > > > > > > proposals
> > > > > > > > about package structure etc..
> > > > > > > > Before posting it on bbigtop mailing list, I have
some
> question
> > > for
> > > > > > apex
> > > > > > > > community.
> > > > > > > >
> > > > > > > > Proposed Directory structure of apex package for both
deb &
> > rpm:
> > > > > > > >
> > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > >
> > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > >
> > > > > > > >
> > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > <other dependent jars> = All the 3rd party jars
which are
> > > required
> > > > > for
> > > > > > > apex
> > > > > > > > to run. Usually the dependencies are packaged as part
of
> > rpm/deb
> > > by
> > > > > any
> > > > > > > > software in bigtop.
> > > > > > > >
> > > > > > >
> > > > > > > Can you specify what those jars are in Bigtop context?
Same as
> > > > shipped
> > > > > > with
> > > > > > > DT community addition under lib/ ?
> > > > > > >
> > > > > > >
> > > > > > > > <package LICENSE> = Licenses of corresponding
of 3rd party
> jars
> > > > which
> > > > > > > needs
> > > > > > > > to included while packaging.
> > > > > > > >
> > > > > > > > Questions related to this:
> > > > > > > > 1. Should we call the cli of apex as "apex" instead
of
> "dtcli"
> > in
> > > > > > bigtop
> > > > > > > > package?
> > > > > > > >
> > > > > > >
> > > > > > > I think we should keep the name until we are able to change
it
> in
> > > > Apex.
> > > > > > > Otherwise this may get confusing.
> > > > > > >
> > > > > > >
> > > > > > > > 2. I see that all softwares in bigtop have man page
for their
> > > > > > > executables.
> > > > > > > > I think we should have it too for dtcli. Is there
any
> > > documentation
> > > > > > > which I
> > > > > > > > can convert to man page? or can I use output of "dtcli
> --help"?
> > > > > > > > 3. Do we want to call version of apex in Bigtop as
3.3.0 OR
> > > > > > > > 3.3.0-incubating?
> > > > > > > >
> > > > > > >
> > > > > > > Has to be -incubating
> > > > > > >
> > > > > > >
> > > > > > > > 4. Is it ok for apex package of bigtop to depend on
2.7.1
> > > version
> > > > of
> > > > > > > > bigtop hadoop? Any problems that we see with this
dependency?
> > > > > > > >
> > > > > > >
> > > > > > > Should work without changed. I thought we certified against
> > 2.7.0?
> > > > > > >
> > > > > > >
> > > > > > > > 5. Following is the apache mirror from which bigtop
will pick
> > the
> > > > > apex
> > > > > > > > source for compilation and packaging. Please correct
if
> > > incorrect:
> > > > > > > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > >     NOTE: apache.osuosl.org is the mirror used by
all the
> > > > softwares
> > > > > in
> > > > > > > > bigtop.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please share your opinion.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar
<
> > > > > > > > chinmay@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > >
> > > > > > > > > I've already started discussion on bigtop dev
mailing list
> > for
> > > > > > > > > integration. Also created a Jira for the same.
For this
> > > > communities
> > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise
<
> > > > > thomas@datatorrent.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Chinmay,
> > > > > > > > >>
> > > > > > > > >> I don't see anything under prerequisites
that would be a
> > > > problem.
> > > > > We
> > > > > > > > >> looked
> > > > > > > > >> at the ASF licencing compatibility as part
of becoming an
> > > > > incubator
> > > > > > > > >> project.
> > > > > > > > >>
> > > > > > > > >> Please focus on the packaging during the
next weeks. Since
> > the
> > > > > work
> > > > > > > will
> > > > > > > > >> be
> > > > > > > > >> part of Bigtop, related discussions and JIRAs
should also
> be
> > > > > there.
> > > > > > > > >>
> > > > > > > > >> Would be good to have the packaging in place
by end Feb.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Thomas
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay
Kolhatkar <
> > > > > > > > >> chinmay@datatorrent.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi All,
> > > > > > > > >> >
> > > > > > > > >> > We're planning a work on adding Apache
Apex as a
> component
> > > to
> > > > > > Apache
> > > > > > > > >> > Bigtop.
> > > > > > > > >> > Bigtop is the packaging system for the
Apache big data
> > > > > ecosystem.
> > > > > > > > >> Several
> > > > > > > > >> > Hadoop distros use it, most recently
EMR.
> > > > > > > > >> >
> > > > > > > > >> > Here is the tracking Jira task in APEXCORE
for the same:
> > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > >> >
> > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > >> > Apache bigtop has some hard and soft
expectation for new
> > > > > > components
> > > > > > > to
> > > > > > > > >> get
> > > > > > > > >> > integrated into Bigtop.
> > > > > > > > >> > Here is the list of it:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > >> >
> > > > > > > > >> > Most of them seems to be standard ASF
based
> requirements,
> > > but
> > > > > few
> > > > > > > need
> > > > > > > > >> to
> > > > > > > > >> > be checked for:
> > > > > > > > >> > 1. Software projects are expected to
be Licensed under
> > > Apache
> > > > > > > License,
> > > > > > > > >> > Version 2.0 (and their dependencies
are expected to be
> > > > > compatible
> > > > > > > with
> > > > > > > > >> this
> > > > > > > > >> > license)
> > > > > > > > >> >     - Apex is under ASL 2.0 but need
to check if
> > > dependencies
> > > > of
> > > > > > > Apex
> > > > > > > > >> are
> > > > > > > > >> > compatible with ASL 2.0. This I guess
would be a
> > > verification
> > > > > > check.
> > > > > > > > >> > 2. Software projects are expected to
be compatible with
> > all
> > > of
> > > > > the
> > > > > > > > >> > supported platforms that Bigtop distribution
is
> targeting
> > > > > > > > >> >     - This needs verifying whether our
software runs
> fine
> > in
> > > > > > > centos-6
> > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8
opensuse-13.2.
> > > > > > > > >> > 3. What smoke tests that should be added
for deployment.
> > > > > > > > >> > 4. Identifying the test artifacts which
goes beyond
> smoke
> > > test
> > > > > > > > >> >     - These are basically the integration
tests for
> > > > verification
> > > > > > > after
> > > > > > > > >> the
> > > > > > > > >> > deployment. This is a soft requirement,
but aim is to
> > > achieve
> > > > > this
> > > > > > > as
> > > > > > > > >> well
> > > > > > > > >> > or at least have explanation why not
to include.
> > > > > > > > >> >
> > > > > > > > >> > If there are any from the link which
explicitly needs to
> > be
> > > > > > checked
> > > > > > > > >> other
> > > > > > > > >> > than above 4, please let us know.
> > > > > > > > >> >
> > > > > > > > >> > *Step 2) Adding Apex as component to
Bigtop*
> > > > > > > > >> > From one of the mail archive of Bigtop,
it was learnt
> that
> > > the
> > > > > > > bigtop
> > > > > > > > >> > community want to see the addition of
new components in
> > > > phases.
> > > > > > Here
> > > > > > > > are
> > > > > > > > >> > the phases:
> > > > > > > > >> > 1. Packaging
> > > > > > > > >> >     - This needs creating of package
i.e. rpm & deb
> files.
> > > > > > > > >> >     - documentations/READMEs, LICENSE,
DISCLAMER, NOTES
> > etc
> > > if
> > > > > any
> > > > > > > > >> needed.
> > > > > > > > >> >     - Any documentation that need to
be added to
> > > distribution
> > > > of
> > > > > > our
> > > > > > > > >> > software.
> > > > > > > > >> >     - Any license information of dependencies
required
> to
> > be
> > > > > added
> > > > > > > to
> > > > > > > > >> > package
> > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > >> >     - Adding smoke test for packaging.
> > > > > > > > >> > 3. Puppet recipes for automatic deployment
and
> > configuration
> > > > > > > > >> >     - Add puppet recipes for automatic
deployment
> > > > > > > > >> > 4. Integration tests
> > > > > > > > >> >     - For verification of deployments.
> > > > > > > > >> > 5. license clearance:
> > > > > > > > >> >     Run 'gradle rat' to make sure all
new stuff is
> > compliant
> > > > > with
> > > > > > > ASF
> > > > > > > > >> > license requirements. If you add code
licenses under
> > > different
> > > > > > > > licenses,
> > > > > > > > >> > those would need to be listed in the
NOTICE.
> > > > > > > > >> >
> > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > >> > We'll start corresponding communication
on bigtop
> mailing
> > > list
> > > > > as
> > > > > > > > well.
> > > > > > > > >> >
> > > > > > > > >> > We have some specific questions/suggestions
related to
> > what
> > > > > should
> > > > > > > be
> > > > > > > > >> the
> > > > > > > > >> > content of the package and what should
be the smoke
> tests,
> > > but
> > > > > in
> > > > > > > the
> > > > > > > > >> > interest of not having too much content
here, we'll put
> > the
> > > > > > > questions
> > > > > > > > >> as a
> > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Chinmay.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message