apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Jotwani <mo...@datatorrent.com>
Subject Re: Adding Transform Operator to Malhar
Date Tue, 08 Mar 2016 11:16:24 GMT
This can have as many util functions added to it to perform various
transformations to the tuple.

+1

Regards,
Mohit
On 8 Mar 2016 16:42, "Sandeep Deshmukh" <sandeep@datatorrent.com> wrote:

> Looks good to me.
>
> Regards,
> Sandeep
>
> On Mon, Mar 7, 2016 at 9:51 PM, Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
>
> > Dear Community,
> >
> > We're creating a transform operator which will allow the apex users to
> > transform data over a stream using this operator.
> >
> > Use Case:
> > ----
> > When the data has been received from input source, one can transform data
> > using transform operator and then dump it to destination.
> >
> > Transform means:
> > ----
> > 1. Conversion of fields from one type to another. Eg. int to Integer,
> epoc
> > to java.util.Date object, int to String or String to int etc.
> > 2. Deriving new fields from one or more input fields.
> >     For eg. Input contains 2 fields viz. "firstname", "lastname" and
> > derived field is "fullname" which is combination of firstname and
> lastname
> > in some way.
> >     Another example is date of birth present in input tuple, generate age
> > from it.
> > 3. Change in schema from input to output. For eg. Input schema is subset
> of
> > output schema and output schema contains some extra derived fields from
> > input fields.
> >
> >
> > Functionality:
> > ----
> > 1. Transform operator will access POJO as input tuple and emit
> another/same
> > POJO output tuple.
> > 2. Operator needs to be configured with input tuple schema and output
> tuple
> > schema for this. This can be done via TUPLE_CLASS attribute on ports.
> > 3. Generation of output tuple from input tuple can be specified using
> > simple java based expressions.
> >     For eg.
> >        outfield1 = <expression using fields of input POJO>
> >        outfield2 = <expression using fields of output POJO>
> > 4. If no expression is mentioned for a certain output field, then a
> > matching field (name and type) will be picked input POJO and contents
> will
> > be copied to output POJO.
> > Otherwise, the output field will be left to default empty value.
> >
> >
> > Expression Support in PojoUtils:
> > ----
> > 1. For achieving above functionality, I thought about extending PojoUtils
> > which already support much of what is required. Then this utility can be
> > used in Transform operator.
> > 2. Couple of minor enhancements/fixing issues required to this utility:
> >     a. Multiple POJO support is not required. Hence will stick to single
> > POJO.
> >     b. PojoUtils does not work well with resolution of getter methods
> when
> > more than one fields are mentioned in expression. Will need to fix that.
> >     c. Following interface can be added to PojoUtils. This will be the
> > return method that can be called for executing the expression
> >         interface Expression<O>
> >         {
> >           O execute(Object obj);
> >         }
> >     d. Following methods can be added to PojoUtils for addressing
> > expression support:
> >         // Evaluates given expression
> >         Expression evaluateExpression(String expr, Class inputObjectType,
> > Class returnType);
> >
> >         // Adds a custom method that can be used in an expression.
> >         // The one who's using this library can utility this method to
> > provide predefined functionality to user.
> >        void registerCustomMethod(String qualifiedFunc)
> >
> >
> > Please share your valuable inputs on this.
> >
> > Thanks,
> > Chinmay.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message