apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lakshmi Velineni <laks...@datatorrent.com>
Subject Re: example applications in malhar
Date Sun, 12 Mar 2017 02:36:36 GMT
Hi,

I have come up with the following list of applications to move from
datatorrent/examples into apex-malhar/examples. There were many examples
that do a similar functionality of moving data around from different
sources to destinations, so I separated them all into a separate folder
called "Ingestion".

csvformatter

dedup

dynamic-partition

enricher

exactly-once

filter

innerjoin

maprapp

parser

partition

recordreader

throttle

topnwords

transform

unifiers

*Ingestion/*

  cassandrainput

  cassandraoutput

  fileio

  fileio-multidir

  fileio-simple

  fileoutput

  filetojdbc

  hdfs2kafka

  hdfs-sync

  jdbcingest

  jdbctijdbc

  jmsActiveMQ

  jmsSqs

  kafka

  s3-to-hdfs-sync

  s3output


Let me know your thoughts.


Thanks

Lakshmi Prasanna







On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre <amol@datatorrent.com> wrote:

> Yes, should merge samples into examples. Ideally the names of the examples
> should be more descriptive in terms of "how to" as opposed to a title. PI
> demo for example has lots of ways to do compute. So it could be named as
> "pi - distributed compute" in examples. Similarly if others can be named to
> bring out features to look at from the examples, it would make examples
> more useful to readers.
>
> Thks
> Amol
>
>
>
> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>
> www.datatorrent.com  |  apex.apache.org
>
> *Join us at Apex Big Data World-San Jose
> <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
> [image: http://www.apexbigdata.com/san-jose-register.html]
> <http://www.apexbigdata.com/san-jose-register.html>
>
> On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare <sanjay@datatorrent.com>
> wrote:
>
> > + for renaming to examples. While we are at it, how about merging
> "samples"
> > also in the new "examples" ?
> >
> > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <ram@datatorrent.com>
> > wrote:
> >
> > > +1 for renaming to "examples"
> > >
> > > Ram
> > >
> > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> > lakshmi@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > I am ready to bring the examples over into the demos folder. I was
> > > > wondering if anybody has any input on Thomas's suggestion to rename
> the
> > > > demos folder to examples. I would rather do that first and then bring
> > the
> > > > examples over instead of doing it the other way around as that would
> > lead
> > > > to refactoring the new examples again.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > > lakshmi@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Since the examples have little history I was planning to have two
> > > > > commits for every example, one for the code as the primary author
> of
> > > > > the example and another containing pom.xml and other changes to
> make
> > > > > it work under malhar.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > > <lakshmi@datatorrent.com> wrote:
> > > > > > Thanks for the suggestions and I am working on the process to
> > migrate
> > > > the
> > > > > > examples with the guidelines you mentioned. I will send out
a
> list
> > of
> > > > > > examples and the destination modules very soon.
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > > thomas.weise@gmail.com>
> > > > > > wrote:
> > > > > >>
> > > > > >> Maybe a good first step would be to identify which examples
to
> > bring
> > > > > over
> > > > > >> and where appropriate how to structure them in Malhar (for
> > example,
> > > I
> > > > > see
> > > > > >> multiple hdfs related apps that could go into the same Maven
> > > module).
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <thw@apache.org>
> > > wrote:
> > > > > >>
> > > > > >> > That would be great. There are a few things to consider
when
> > > working
> > > > > on
> > > > > >> > it:
> > > > > >> >
> > > > > >> > * preserve attribtion
> > > > > >> > * ensure there is a test that runs the application
in the CI
> > > > > >> > * check that dependencies are compatible license
> > > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > > >> >
> > > > > >> > etc.
> > > > > >> >
> > > > > >> > Existing examples are under https://github.com/apache/
> > > > > >> > apex-malhar/tree/master/demos
> > > > > >> >
> > > > > >> > Perhaps we should rename it to "examples"
> > > > > >> >
> > > > > >> > I also propose that each app has a README and we add
those for
> > > > > existing
> > > > > >> > apps as well.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Thomas
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni
<
> > > > > >> > lakshmi@datatorrent.com> wrote:
> > > > > >> >
> > > > > >> >>   Can i work on this?
> > > > > >> >>
> > > > > >> >> Thanks
> > > > > >> >> Lakshmi Prasanna
> > > > > >> >>
> > > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra
Putta <
> > > > > >> >> ashwinchandrap@gmail.com> wrote:
> > > > > >> >>
> > > > > >> >> > Here is the JIRA:
> > > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > > > >> >> >
> > > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol Kekre
<
> > > > amol@datatorrent.com>
> > > > > >> >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Good idea to consolidate them into Malhar.
We should
> bring
> > in
> > > > as
> > > > > >> >> > > many
> > > > > >> >> > from
> > > > > >> >> > > this gitHub as possible.
> > > > > >> >> > >
> > > > > >> >> > > Thks
> > > > > >> >> > > Amol
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM, Thomas
Weise
> > > > > >> >> > > <thomas@datatorrent.com>
> > > > > >> >> > > wrote:
> > > > > >> >> > >
> > > > > >> >> > > > I'm also for consolidating these
different example
> > > locations.
> > > > > We
> > > > > >> >> should
> > > > > >> >> > > > also look if all of it is still
relevant.
> > > > > >> >> > > >
> > > > > >> >> > > > The stuff from the DT repository
needs to be brought
> into
> > > > shape
> > > > > >> >> > > > wrt
> > > > > >> >> > > > licensing, checkstyle, CI support
etc.
> > > > > >> >> > > >
> > > > > >> >> > > >
> > > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34 PM,
Pramod Immaneni <
> > > > > >> >> > pramod@datatorrent.com>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > > > Sounds like a good idea. How
about merging demos with
> > > apps
> > > > as
> > > > > >> >> well?
> > > > > >> >> > > > >
> > > > > >> >> > > > > On Tue, Sep 6, 2016 at 4:30
PM, Ashwin Chandra Putta
> <
> > > > > >> >> > > > > ashwinchandrap@gmail.com>
wrote:
> > > > > >> >> > > > >
> > > > > >> >> > > > > > Hi All,
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > We have a lot of examples
for apex malhar operators
> > in
> > > > the
> > > > > >> >> > following
> > > > > >> >> > > > > > repository which resides
outside of malhar.
> > > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > > > master/tutorials
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Now that it has grown
quite a bit, does it make
> sense
> > > to
> > > > > >> >> > > > > > bring
> > > > > >> >> some
> > > > > >> >> > > of
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > most common examples to
malhar repository? Probably
> > > under
> > > > > >> >> > > > > > apps
> > > > > >> >> > > > directory?
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > That way folks looking
at malhar repository will
> have
> > > > some
> > > > > >> >> samples
> > > > > >> >> > to
> > > > > >> >> > > > > look
> > > > > >> >> > > > > > at without having to search
elsewhere.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > --
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Regards,
> > > > > >> >> > > > > > Ashwin.
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > --
> > > > > >> >> >
> > > > > >> >> > Regards,
> > > > > >> >> > Ashwin.
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >> >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > _______________________________________________________
> > >
> > > Munagala V. Ramanath
> > >
> > > Software Engineer
> > >
> > > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> > >
> > > www.datatorrent.com  |  apex.apache.org
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message