apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogi Devendra <yogideven...@apache.org>
Subject Re: Adding Transform Operator to Malhar
Date Tue, 08 Mar 2016 11:33:20 GMT
Can we think of better name than Transform operator?
May be:
Expression operator

Reason:
We have many operators which are doing 'transform'. So, this name looks
very generic.

In fact, from the users perspective, I always consider any operator like a
black-box doing some transformation.

~ Yogi

On 8 March 2016 at 16:46, Mohit Jotwani <mohit@datatorrent.com> wrote:

> This can have as many util functions added to it to perform various
> transformations to the tuple.
>
> +1
>
> Regards,
> Mohit
> On 8 Mar 2016 16:42, "Sandeep Deshmukh" <sandeep@datatorrent.com> wrote:
>
> > Looks good to me.
> >
> > Regards,
> > Sandeep
> >
> > On Mon, Mar 7, 2016 at 9:51 PM, Chinmay Kolhatkar <chinmay@apache.org>
> > wrote:
> >
> > > Dear Community,
> > >
> > > We're creating a transform operator which will allow the apex users to
> > > transform data over a stream using this operator.
> > >
> > > Use Case:
> > > ----
> > > When the data has been received from input source, one can transform
> data
> > > using transform operator and then dump it to destination.
> > >
> > > Transform means:
> > > ----
> > > 1. Conversion of fields from one type to another. Eg. int to Integer,
> > epoc
> > > to java.util.Date object, int to String or String to int etc.
> > > 2. Deriving new fields from one or more input fields.
> > >     For eg. Input contains 2 fields viz. "firstname", "lastname" and
> > > derived field is "fullname" which is combination of firstname and
> > lastname
> > > in some way.
> > >     Another example is date of birth present in input tuple, generate
> age
> > > from it.
> > > 3. Change in schema from input to output. For eg. Input schema is
> subset
> > of
> > > output schema and output schema contains some extra derived fields from
> > > input fields.
> > >
> > >
> > > Functionality:
> > > ----
> > > 1. Transform operator will access POJO as input tuple and emit
> > another/same
> > > POJO output tuple.
> > > 2. Operator needs to be configured with input tuple schema and output
> > tuple
> > > schema for this. This can be done via TUPLE_CLASS attribute on ports.
> > > 3. Generation of output tuple from input tuple can be specified using
> > > simple java based expressions.
> > >     For eg.
> > >        outfield1 = <expression using fields of input POJO>
> > >        outfield2 = <expression using fields of output POJO>
> > > 4. If no expression is mentioned for a certain output field, then a
> > > matching field (name and type) will be picked input POJO and contents
> > will
> > > be copied to output POJO.
> > > Otherwise, the output field will be left to default empty value.
> > >
> > >
> > > Expression Support in PojoUtils:
> > > ----
> > > 1. For achieving above functionality, I thought about extending
> PojoUtils
> > > which already support much of what is required. Then this utility can
> be
> > > used in Transform operator.
> > > 2. Couple of minor enhancements/fixing issues required to this utility:
> > >     a. Multiple POJO support is not required. Hence will stick to
> single
> > > POJO.
> > >     b. PojoUtils does not work well with resolution of getter methods
> > when
> > > more than one fields are mentioned in expression. Will need to fix
> that.
> > >     c. Following interface can be added to PojoUtils. This will be the
> > > return method that can be called for executing the expression
> > >         interface Expression<O>
> > >         {
> > >           O execute(Object obj);
> > >         }
> > >     d. Following methods can be added to PojoUtils for addressing
> > > expression support:
> > >         // Evaluates given expression
> > >         Expression evaluateExpression(String expr, Class
> inputObjectType,
> > > Class returnType);
> > >
> > >         // Adds a custom method that can be used in an expression.
> > >         // The one who's using this library can utility this method to
> > > provide predefined functionality to user.
> > >        void registerCustomMethod(String qualifiedFunc)
> > >
> > >
> > > Please share your valuable inputs on this.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message