apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandesh Hegde <sand...@datatorrent.com>
Subject Re: Why are parser and formatter operator hidden in Malhar/contrib/schema
Date Thu, 17 Dec 2015 21:06:33 GMT
Even my understanding was along the lines, "if an unit test can't be run on
a developer machine without installing dependencies ( RabbitMq/redis ec),
it should go into contrib".

Documenting the exact process as to what goes where helps.

On Thu, Dec 17, 2015 at 12:16 PM Chinmay Kolhatkar <chinmay@datatorrent.com>
wrote:

> Hi Isha,
>
> I think what Shubham meant to say is any operator which has a dependency on
> external entity (not library) should go in contrib. Others should go in
> library.
>
> For eg. DB related operators needs an instance/server of respect database
> to be running. Without that running server, the operator cannot satisfy the
> functionality expected. Hence it goes into contrib.
>
> I think none of the parsers will have dependency on external entity, hence
> they should go in library and not contrib.
>
> - Chinmay.
> On 18 Dec 2015 00:42, "Isha Arkatkar" <isha@datatorrent.com> wrote:
>
> > Hi Shubham,
> >
> >       I think it is somewhat subjective what goes in Contrib Vs Library.
> > Here is what I understood about general guideline:
> >        An operator would go in  malhar-library if it does not have any
> > other library dependency than what is already available. If operator
> needs
> > to include a dependency, it could be added to Contrib but not library.
> >
> >      If we add an external library dependency in Malhar-library, the size
> > of the lib jar keeps on growing. So, if we refer malhar-lib in a project,
> > the size of the total package would increase, even if we may not directly
> > use all external libs.
> >
> >     For this reason, in pull request #137
> > <https://github.com/apache/incubator-apex-malhar/pull/137> , I move only
> > Json and XML parsers to malhar-library. Csv one had library reference to
> > supercsv, so it is still in contrib.
> >
> >   Please correct if I missed something. :)
> >
> > Thanks!
> > Isha
> >
> > On Wed, Dec 16, 2015 at 11:27 PM, Shubham Pathak <
> shubham@datatorrent.com>
> > wrote:
> >
> > > Hi,
> > >
> > > What do we mean by "additional dependency" in case of deciding contrib
> vs
> > > lib ?
> > > As i understand, "additional dependency"  is when an operator is
> > > interacting with external technologies . For e.g kafka, MQ , HBase etc.
> > > By this definition even CSV parsers need to be moved to malhar-lib.
> > >
> > > Thanks,
> > > Shubham
> > >
> > >
> > > On Thu, Dec 17, 2015 at 5:39 AM, Isha Arkatkar <isha@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Chandni,
> > > >
> > > >    I have moved that as well to lib, as the parsers depended on that.
> > > >
> > > > Thanks,
> > > > Isha
> > > >
> > > > On Wed, Dec 16, 2015 at 1:01 PM, Chandni Singh <
> > chandni@datatorrent.com>
> > > > wrote:
> > > >
> > > > > There is a converter package under com.datatorrent.contrib which
> has
> > a
> > > > > Converter API. This belongs in library as well.
> > > > >
> > > > > Thanks,
> > > > > Chandni
> > > > >
> > > > > On Tue, Dec 15, 2015 at 1:29 PM, Chandni Singh <
> > > chandni@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Isha,
> > > > > >
> > > > > > Thanks for moving this. When you move these files, please place
> > then
> > > > > under
> > > > > > a package which reflects its functionality. I don't see the
need
> > for
> > > > > > package called schema.
> > > > > >
> > > > > > Thanks,
> > > > > > Chandni
> > > > > >
> > > > > > On Tue, Dec 15, 2015 at 12:31 PM, Isha Arkatkar <
> > > isha@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >>   For csv parser there is an additional dependency. So,
I'll
> move
> > > only
> > > > > >> json
> > > > > >> and xml to new location.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Isha
> > > > > >>
> > > > > >> On Tue, Dec 15, 2015 at 11:42 AM, Thomas Weise <
> > > > thomas@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > As long as the operators don't introduce additional
> dependencies
> > > > they
> > > > > >> > should be in lib.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Dec 15, 2015 at 9:34 AM, Shubham Pathak <
> > > > > >> shubham@datatorrent.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Chandni,
> > > > > >> > >
> > > > > >> > > I had written those operators.
> > > > > >> > > Here is the jira for that
> > > > > >> https://malhar.atlassian.net/browse/MLHR-1838
> > > > > >> > > You would find the entire discussion there.
> > > > > >> > >
> > > > > >> > > Why are all these operator under Malhar/contrib
and not
> > > Malhar/lib
> > > > > >> > > When i was writing the code i saw AbstractCsvParser
in
> > contriib
> > > > and
> > > > > >> hence
> > > > > >> > > added there.
> > > > > >> > >
> > > > > >> > > Recently i got to know which operators must go
in contrib
> and
> > > what
> > > > > >> must
> > > > > >> > go
> > > > > >> > > in lib.
> > > > > >> > > By that definition, these operators must belong
to lib.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Shubham
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Tue, Dec 15, 2015 at 1:32 PM, Chandni Singh
<
> > > > > >> chandni@datatorrent.com>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi,
> > > > > >> > > >
> > > > > >> > > > I just came across couple of formatter and
parser
> operators
> > > > which
> > > > > >> are
> > > > > >> > > under
> > > > > >> > > > Malhar/contrib/schema.
> > > > > >> > > >
> > > > > >> > > > I have couple of questions:
> > > > > >> > > > 1. What does schema denote here?
> > > > > >> > > > 2. Why formatter/parser which are functions
are placed
> under
> > > > > schema
> > > > > >> > > > package?
> > > > > >> > > > 2. Why are all these operator under Malhar/contrib
and not
> > > > > >> Malhar/lib
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Chandni
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message