apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devendra Tagare <devend...@datatorrent.com>
Subject Re: Adding Transform Operator to Malhar
Date Tue, 08 Mar 2016 12:18:29 GMT
+1

Dev


On Tue, Mar 8, 2016 at 5:38 PM, Yogi Devendra <yogidevendra@apache.org>
wrote:

> Forgot to add in earlier email:
>
> +1 for this operator.
>
> ~ Yogi
>
> On 8 March 2016 at 17:03, Yogi Devendra <yogidevendra@apache.org> wrote:
>
> > Can we think of better name than Transform operator?
> > May be:
> > Expression operator
> >
> > Reason:
> > We have many operators which are doing 'transform'. So, this name looks
> > very generic.
> >
> > In fact, from the users perspective, I always consider any operator like
> a
> > black-box doing some transformation.
> >
> > ~ Yogi
> >
> > On 8 March 2016 at 16:46, Mohit Jotwani <mohit@datatorrent.com> wrote:
> >
> >> This can have as many util functions added to it to perform various
> >> transformations to the tuple.
> >>
> >> +1
> >>
> >> Regards,
> >> Mohit
> >> On 8 Mar 2016 16:42, "Sandeep Deshmukh" <sandeep@datatorrent.com>
> wrote:
> >>
> >> > Looks good to me.
> >> >
> >> > Regards,
> >> > Sandeep
> >> >
> >> > On Mon, Mar 7, 2016 at 9:51 PM, Chinmay Kolhatkar <chinmay@apache.org
> >
> >> > wrote:
> >> >
> >> > > Dear Community,
> >> > >
> >> > > We're creating a transform operator which will allow the apex users
> to
> >> > > transform data over a stream using this operator.
> >> > >
> >> > > Use Case:
> >> > > ----
> >> > > When the data has been received from input source, one can transform
> >> data
> >> > > using transform operator and then dump it to destination.
> >> > >
> >> > > Transform means:
> >> > > ----
> >> > > 1. Conversion of fields from one type to another. Eg. int to
> Integer,
> >> > epoc
> >> > > to java.util.Date object, int to String or String to int etc.
> >> > > 2. Deriving new fields from one or more input fields.
> >> > >     For eg. Input contains 2 fields viz. "firstname", "lastname" and
> >> > > derived field is "fullname" which is combination of firstname and
> >> > lastname
> >> > > in some way.
> >> > >     Another example is date of birth present in input tuple,
> generate
> >> age
> >> > > from it.
> >> > > 3. Change in schema from input to output. For eg. Input schema is
> >> subset
> >> > of
> >> > > output schema and output schema contains some extra derived fields
> >> from
> >> > > input fields.
> >> > >
> >> > >
> >> > > Functionality:
> >> > > ----
> >> > > 1. Transform operator will access POJO as input tuple and emit
> >> > another/same
> >> > > POJO output tuple.
> >> > > 2. Operator needs to be configured with input tuple schema and
> output
> >> > tuple
> >> > > schema for this. This can be done via TUPLE_CLASS attribute on
> ports.
> >> > > 3. Generation of output tuple from input tuple can be specified
> using
> >> > > simple java based expressions.
> >> > >     For eg.
> >> > >        outfield1 = <expression using fields of input POJO>
> >> > >        outfield2 = <expression using fields of output POJO>
> >> > > 4. If no expression is mentioned for a certain output field, then
a
> >> > > matching field (name and type) will be picked input POJO and
> contents
> >> > will
> >> > > be copied to output POJO.
> >> > > Otherwise, the output field will be left to default empty value.
> >> > >
> >> > >
> >> > > Expression Support in PojoUtils:
> >> > > ----
> >> > > 1. For achieving above functionality, I thought about extending
> >> PojoUtils
> >> > > which already support much of what is required. Then this utility
> can
> >> be
> >> > > used in Transform operator.
> >> > > 2. Couple of minor enhancements/fixing issues required to this
> >> utility:
> >> > >     a. Multiple POJO support is not required. Hence will stick to
> >> single
> >> > > POJO.
> >> > >     b. PojoUtils does not work well with resolution of getter
> methods
> >> > when
> >> > > more than one fields are mentioned in expression. Will need to fix
> >> that.
> >> > >     c. Following interface can be added to PojoUtils. This will be
> the
> >> > > return method that can be called for executing the expression
> >> > >         interface Expression<O>
> >> > >         {
> >> > >           O execute(Object obj);
> >> > >         }
> >> > >     d. Following methods can be added to PojoUtils for addressing
> >> > > expression support:
> >> > >         // Evaluates given expression
> >> > >         Expression evaluateExpression(String expr, Class
> >> inputObjectType,
> >> > > Class returnType);
> >> > >
> >> > >         // Adds a custom method that can be used in an expression.
> >> > >         // The one who's using this library can utility this method
> to
> >> > > provide predefined functionality to user.
> >> > >        void registerCustomMethod(String qualifiedFunc)
> >> > >
> >> > >
> >> > > Please share your valuable inputs on this.
> >> > >
> >> > > Thanks,
> >> > > Chinmay.
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message