apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Dalvi <pradeep.da...@datatorrent.com>
Subject Re: Projection Operator in Malhar
Date Mon, 14 Mar 2016 06:23:50 GMT
With the current schema support, Output POJO classes will be user specified.

I've created JIRA for Projection Operator:
https://issues.apache.org/jira/browse/APEXMALHAR-2015

On Thu, Mar 10, 2016 at 11:11 AM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> +1 for the Projection operator and for the name.
>
> One question: Will the two output POJO classes be user supplied? Or will
> they be dynamically generated?
> In other words, will the user, in addition to specifying the list of
> selected and dropped fields from the incoming POJO, also define the
> corresponding classes?
>
> -Bhupesh
>
> On Thu, Mar 10, 2016 at 11:07 AM, Yogi Devendra <yogidevendra@apache.org>
> wrote:
>
> > @Priyanka
> >
> > Projection is standard term from relational algebra
> > https://en.wikipedia.org/wiki/Projection_(relational_algebra)
> >
> > +1 for the name projection.
> >
> > ~ Yogi
> >
> > On 10 March 2016 at 11:01, Priyanka Gugale <priyanka@datatorrent.com>
> > wrote:
> >
> > > +1.
> > >
> > > This name is not self explanatory, I can't think of anything better as
> of
> > > now. If someone have suggestions please add it here. I will also try to
> > > find some options.
> > >
> > > -Priyanka
> > >
> > > On Thu, Mar 10, 2016 at 10:17 AM, Sandeep Deshmukh <
> > > sandeep@datatorrent.com>
> > > wrote:
> > >
> > > > + 1 on projection w/o adding or updating fields. IMO, the purpose is
> to
> > > > split the record into something that is useful and something that is
> > not.
> > > >
> > > > Regards,
> > > > Sandeep
> > > >
> > > > On Wed, Mar 9, 2016 at 9:08 PM, Chinmay Kolhatkar <
> chinmay@apache.org>
> > > > wrote:
> > > >
> > > > > +1 for projection operator.
> > > > >
> > > > > @Sandesh, Adding of new fields and updating will be done as a part
> of
> > > > > Enricher and Transform operator respectively.
> > > > > I've sent another mail about Transform operator before.
> > > > >
> > > > >
> > > > > On Wed, Mar 9, 2016 at 8:54 PM, Sandesh Hegde <
> > sandesh@datatorrent.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Adding of new fields and updating of fields can be part of this?
> > > > > >
> > > > > > On Wed, Mar 9, 2016, 5:50 AM Pradeep A. Dalvi <prad@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Dear Folks,
> > > > > > >
> > > > > > > We are creating Projection Operator which will allow apex
users
> > to
> > > > > > project
> > > > > > > (select/drop) certain fields from the incoming tuples.
This
> > > operation
> > > > > > might
> > > > > > > be done unconditionally or based on certain condition.
> > > > > > >
> > > > > > > Use case:
> > > > > > > -------------
> > > > > > > Not all fields of tuples are of interest for the downstream
> > > > operators.
> > > > > In
> > > > > > > such cases, one may want project selective fields to
> downstream.
> > > Also
> > > > > one
> > > > > > > may want to drop few fields, instead of selecting many.
> > > > > > > In certain scenarios, one may want to project certain fields
> > based
> > > on
> > > > > > given
> > > > > > > condition or expression.
> > > > > > >
> > > > > > > Functionality:
> > > > > > > -----------------
> > > > > > > 1. Projection operator shall receive POJO as input tuple
and
> > emit 2
> > > > > POJOs
> > > > > > > on separate output ports i.e. selected and dropped. Selected
> > output
> > > > > port
> > > > > > > shall emit POJO with selected fields and dropped output
shall
> > emit
> > > > POJO
> > > > > > of
> > > > > > > dropped fields.
> > > > > > > 2. Operator needs select or drop fields as input params.
This
> > shall
> > > > be
> > > > > > > specified using comma separated list of fields.
> > > > > > > 3. Operator shall emit POJO only on connected output ports.
In
> > > > another
> > > > > > > words, if dropped output port is not connected, it shall
not
> even
> > > try
> > > > > to
> > > > > > > generate POJOs with dropped fields.
> > > > > > >
> > > > > > > Please feel free to let me know your thoughts on this.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Pradeep A. Dalvi
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Pradeep A. Dalvi

Software Engineer
DataTorrent (India)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message