apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: example applications in malhar
Date Sun, 12 Mar 2017 17:13:45 GMT
I think a flat list with good naming would make it easier to find a
matching example.

topnwords ? There are already two related examples in Malhar: wordcount and
twitter

Can some of the examples be consolidated into a single example package,
fileioXXX/fileoutput look like candidates.


On Sat, Mar 11, 2017 at 6:36 PM, Lakshmi Velineni <lakshmi@datatorrent.com>
wrote:

> Hi,
>
> I have come up with the following list of applications to move from
> datatorrent/examples into apex-malhar/examples. There were many examples
> that do a similar functionality of moving data around from different
> sources to destinations, so I separated them all into a separate folder
> called "Ingestion".
>
> csvformatter
>
> dedup
>
> dynamic-partition
>
> enricher
>
> exactly-once
>
> filter
>
> innerjoin
>
> maprapp
>
> parser
>
> partition
>
> recordreader
>
> throttle
>
> topnwords
>
> transform
>
> unifiers
>
> *Ingestion/*
>
>   cassandrainput
>
>   cassandraoutput
>
>   fileio
>
>   fileio-multidir
>
>   fileio-simple
>
>   fileoutput
>
>   filetojdbc
>
>   hdfs2kafka
>
>   hdfs-sync
>
>   jdbcingest
>
>   jdbctijdbc
>
>   jmsActiveMQ
>
>   jmsSqs
>
>   kafka
>
>   s3-to-hdfs-sync
>
>   s3output
>
>
> Let me know your thoughts.
>
>
> Thanks
>
> Lakshmi Prasanna
>
>
>
>
>
>
>
> On Thu, Feb 23, 2017 at 10:34 AM, Amol Kekre <amol@datatorrent.com> wrote:
>
> > Yes, should merge samples into examples. Ideally the names of the
> examples
> > should be more descriptive in terms of "how to" as opposed to a title. PI
> > demo for example has lots of ways to do compute. So it could be named as
> > "pi - distributed compute" in examples. Similarly if others can be named
> to
> > bring out features to look at from the examples, it would make examples
> > more useful to readers.
> >
> > Thks
> > Amol
> >
> >
> >
> > E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> >
> > www.datatorrent.com  |  apex.apache.org
> >
> > *Join us at Apex Big Data World-San Jose
> > <http://www.apexbigdata.com/san-jose.html>, April 4, 2017!*
> > [image: http://www.apexbigdata.com/san-jose-register.html]
> > <http://www.apexbigdata.com/san-jose-register.html>
> >
> > On Thu, Feb 23, 2017 at 10:07 AM, Sanjay Pujare <sanjay@datatorrent.com>
> > wrote:
> >
> > > + for renaming to examples. While we are at it, how about merging
> > "samples"
> > > also in the new "examples" ?
> > >
> > > On Thu, Feb 23, 2017 at 9:47 AM, Munagala Ramanath <
> ram@datatorrent.com>
> > > wrote:
> > >
> > > > +1 for renaming to "examples"
> > > >
> > > > Ram
> > > >
> > > > On Thu, Feb 23, 2017 at 9:12 AM, Lakshmi Velineni <
> > > lakshmi@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > I am ready to bring the examples over into the demos folder. I was
> > > > > wondering if anybody has any input on Thomas's suggestion to rename
> > the
> > > > > demos folder to examples. I would rather do that first and then
> bring
> > > the
> > > > > examples over instead of doing it the other way around as that
> would
> > > lead
> > > > > to refactoring the new examples again.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Jan 25, 2017 at 8:12 AM, Lakshmi Velineni <
> > > > lakshmi@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Since the examples have little history I was planning to have
two
> > > > > > commits for every example, one for the code as the primary author
> > of
> > > > > > the example and another containing pom.xml and other changes
to
> > make
> > > > > > it work under malhar.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Nov 2, 2016 at 9:49 PM, Lakshmi Velineni
> > > > > > <lakshmi@datatorrent.com> wrote:
> > > > > > > Thanks for the suggestions and I am working on the process
to
> > > migrate
> > > > > the
> > > > > > > examples with the guidelines you mentioned. I will send
out a
> > list
> > > of
> > > > > > > examples and the destination modules very soon.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 27, 2016 at 1:43 PM, Thomas Weise <
> > > > thomas.weise@gmail.com>
> > > > > > > wrote:
> > > > > > >>
> > > > > > >> Maybe a good first step would be to identify which
examples to
> > > bring
> > > > > > over
> > > > > > >> and where appropriate how to structure them in Malhar
(for
> > > example,
> > > > I
> > > > > > see
> > > > > > >> multiple hdfs related apps that could go into the same
Maven
> > > > module).
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Oct 25, 2016 at 1:00 PM, Thomas Weise <thw@apache.org
> >
> > > > wrote:
> > > > > > >>
> > > > > > >> > That would be great. There are a few things to
consider when
> > > > working
> > > > > > on
> > > > > > >> > it:
> > > > > > >> >
> > > > > > >> > * preserve attribtion
> > > > > > >> > * ensure there is a test that runs the application
in the CI
> > > > > > >> > * check that dependencies are compatible license
> > > > > > >> > * maybe extract common boilerplate code from pom.xml
> > > > > > >> >
> > > > > > >> > etc.
> > > > > > >> >
> > > > > > >> > Existing examples are under https://github.com/apache/
> > > > > > >> > apex-malhar/tree/master/demos
> > > > > > >> >
> > > > > > >> > Perhaps we should rename it to "examples"
> > > > > > >> >
> > > > > > >> > I also propose that each app has a README and
we add those
> for
> > > > > > existing
> > > > > > >> > apps as well.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Thomas
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Oct 25, 2016 at 12:49 PM, Lakshmi Velineni
<
> > > > > > >> > lakshmi@datatorrent.com> wrote:
> > > > > > >> >
> > > > > > >> >>   Can i work on this?
> > > > > > >> >>
> > > > > > >> >> Thanks
> > > > > > >> >> Lakshmi Prasanna
> > > > > > >> >>
> > > > > > >> >> On Mon, Sep 12, 2016 at 9:41 PM, Ashwin Chandra
Putta <
> > > > > > >> >> ashwinchandrap@gmail.com> wrote:
> > > > > > >> >>
> > > > > > >> >> > Here is the JIRA:
> > > > > > >> >> > https://issues.apache.org/jira/browse/APEXMALHAR-2233
> > > > > > >> >> >
> > > > > > >> >> > On Tue, Sep 6, 2016 at 10:20 PM, Amol
Kekre <
> > > > > amol@datatorrent.com>
> > > > > > >> >> wrote:
> > > > > > >> >> >
> > > > > > >> >> > > Good idea to consolidate them into
Malhar. We should
> > bring
> > > in
> > > > > as
> > > > > > >> >> > > many
> > > > > > >> >> > from
> > > > > > >> >> > > this gitHub as possible.
> > > > > > >> >> > >
> > > > > > >> >> > > Thks
> > > > > > >> >> > > Amol
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > > On Tue, Sep 6, 2016 at 6:02 PM,
Thomas Weise
> > > > > > >> >> > > <thomas@datatorrent.com>
> > > > > > >> >> > > wrote:
> > > > > > >> >> > >
> > > > > > >> >> > > > I'm also for consolidating
these different example
> > > > locations.
> > > > > > We
> > > > > > >> >> should
> > > > > > >> >> > > > also look if all of it is still
relevant.
> > > > > > >> >> > > >
> > > > > > >> >> > > > The stuff from the DT repository
needs to be brought
> > into
> > > > > shape
> > > > > > >> >> > > > wrt
> > > > > > >> >> > > > licensing, checkstyle, CI support
etc.
> > > > > > >> >> > > >
> > > > > > >> >> > > >
> > > > > > >> >> > > > On Tue, Sep 6, 2016 at 4:34
PM, Pramod Immaneni <
> > > > > > >> >> > pramod@datatorrent.com>
> > > > > > >> >> > > > wrote:
> > > > > > >> >> > > >
> > > > > > >> >> > > > > Sounds like a good idea.
How about merging demos
> with
> > > > apps
> > > > > as
> > > > > > >> >> well?
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > On Tue, Sep 6, 2016 at
4:30 PM, Ashwin Chandra
> Putta
> > <
> > > > > > >> >> > > > > ashwinchandrap@gmail.com>
wrote:
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > > Hi All,
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > We have a lot of
examples for apex malhar
> operators
> > > in
> > > > > the
> > > > > > >> >> > following
> > > > > > >> >> > > > > > repository which
resides outside of malhar.
> > > > > > >> >> > > > > > https://github.com/DataTorrent/examples/tree/
> > > > > > master/tutorials
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Now that it has grown
quite a bit, does it make
> > sense
> > > > to
> > > > > > >> >> > > > > > bring
> > > > > > >> >> some
> > > > > > >> >> > > of
> > > > > > >> >> > > > > the
> > > > > > >> >> > > > > > most common examples
to malhar repository?
> Probably
> > > > under
> > > > > > >> >> > > > > > apps
> > > > > > >> >> > > > directory?
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > That way folks looking
at malhar repository will
> > have
> > > > > some
> > > > > > >> >> samples
> > > > > > >> >> > to
> > > > > > >> >> > > > > look
> > > > > > >> >> > > > > > at without having
to search elsewhere.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > --
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Regards,
> > > > > > >> >> > > > > > Ashwin.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > >
> > > > > > >> >> > > >
> > > > > > >> >> > >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > --
> > > > > > >> >> >
> > > > > > >> >> > Regards,
> > > > > > >> >> > Ashwin.
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >
> > > > > > >> >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > _______________________________________________________
> > > >
> > > > Munagala V. Ramanath
> > > >
> > > > Software Engineer
> > > >
> > > > E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> > > >
> > > > www.datatorrent.com  |  apex.apache.org
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message