apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isha Arkatkar <i...@datatorrent.com>
Subject Re: Why are parser and formatter operator hidden in Malhar/contrib/schema
Date Thu, 17 Dec 2015 19:12:14 GMT
Hi Shubham,

      I think it is somewhat subjective what goes in Contrib Vs Library.
Here is what I understood about general guideline:
       An operator would go in  malhar-library if it does not have any
other library dependency than what is already available. If operator needs
to include a dependency, it could be added to Contrib but not library.

     If we add an external library dependency in Malhar-library, the size
of the lib jar keeps on growing. So, if we refer malhar-lib in a project,
the size of the total package would increase, even if we may not directly
use all external libs.

    For this reason, in pull request #137
<https://github.com/apache/incubator-apex-malhar/pull/137> , I move only
Json and XML parsers to malhar-library. Csv one had library reference to
supercsv, so it is still in contrib.

  Please correct if I missed something. :)

Thanks!
Isha

On Wed, Dec 16, 2015 at 11:27 PM, Shubham Pathak <shubham@datatorrent.com>
wrote:

> Hi,
>
> What do we mean by "additional dependency" in case of deciding contrib vs
> lib ?
> As i understand, "additional dependency"  is when an operator is
> interacting with external technologies . For e.g kafka, MQ , HBase etc.
> By this definition even CSV parsers need to be moved to malhar-lib.
>
> Thanks,
> Shubham
>
>
> On Thu, Dec 17, 2015 at 5:39 AM, Isha Arkatkar <isha@datatorrent.com>
> wrote:
>
> > Hi Chandni,
> >
> >    I have moved that as well to lib, as the parsers depended on that.
> >
> > Thanks,
> > Isha
> >
> > On Wed, Dec 16, 2015 at 1:01 PM, Chandni Singh <chandni@datatorrent.com>
> > wrote:
> >
> > > There is a converter package under com.datatorrent.contrib which has a
> > > Converter API. This belongs in library as well.
> > >
> > > Thanks,
> > > Chandni
> > >
> > > On Tue, Dec 15, 2015 at 1:29 PM, Chandni Singh <
> chandni@datatorrent.com>
> > > wrote:
> > >
> > > > Isha,
> > > >
> > > > Thanks for moving this. When you move these files, please place then
> > > under
> > > > a package which reflects its functionality. I don't see the need for
> > > > package called schema.
> > > >
> > > > Thanks,
> > > > Chandni
> > > >
> > > > On Tue, Dec 15, 2015 at 12:31 PM, Isha Arkatkar <
> isha@datatorrent.com>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >>   For csv parser there is an additional dependency. So, I'll move
> only
> > > >> json
> > > >> and xml to new location.
> > > >>
> > > >> Thanks,
> > > >> Isha
> > > >>
> > > >> On Tue, Dec 15, 2015 at 11:42 AM, Thomas Weise <
> > thomas@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > As long as the operators don't introduce additional dependencies
> > they
> > > >> > should be in lib.
> > > >> >
> > > >> >
> > > >> > On Tue, Dec 15, 2015 at 9:34 AM, Shubham Pathak <
> > > >> shubham@datatorrent.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Chandni,
> > > >> > >
> > > >> > > I had written those operators.
> > > >> > > Here is the jira for that
> > > >> https://malhar.atlassian.net/browse/MLHR-1838
> > > >> > > You would find the entire discussion there.
> > > >> > >
> > > >> > > Why are all these operator under Malhar/contrib and not
> Malhar/lib
> > > >> > > When i was writing the code i saw AbstractCsvParser in contriib
> > and
> > > >> hence
> > > >> > > added there.
> > > >> > >
> > > >> > > Recently i got to know which operators must go in contrib
and
> what
> > > >> must
> > > >> > go
> > > >> > > in lib.
> > > >> > > By that definition, these operators must belong to lib.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Shubham
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Dec 15, 2015 at 1:32 PM, Chandni Singh <
> > > >> chandni@datatorrent.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi,
> > > >> > > >
> > > >> > > > I just came across couple of formatter and parser operators
> > which
> > > >> are
> > > >> > > under
> > > >> > > > Malhar/contrib/schema.
> > > >> > > >
> > > >> > > > I have couple of questions:
> > > >> > > > 1. What does schema denote here?
> > > >> > > > 2. Why formatter/parser which are functions are placed
under
> > > schema
> > > >> > > > package?
> > > >> > > > 2. Why are all these operator under Malhar/contrib
and not
> > > >> Malhar/lib
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Chandni
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message