apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandni Singh <chan...@datatorrent.com>
Subject Re: Changes to JdbcOutputOperator
Date Mon, 14 Dec 2015 05:34:45 GMT
No I don't think we are restricting Malhar to just abstract classes.
Whenever they are couple of use cases that we see quite often, we add
concrete implementations.

For eg. FileLineInputOperator which is a concrete implementation of
AbstractFileInputOperator. FSSliceReader is an example as well.

In AbstractJdbcOutputOperator case there hasn't been such common
insert/update query.

Also if you look at the example I provided, it is very simple to provide a
concrete implementation.

If you would like to change JdbcPOJOOutputOperator to work for
"UPDATE/MERGE" then please go ahead.

Chandni

On Sun, Dec 13, 2015 at 8:47 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> I see. So, just to understand more, do we plan to keep Malhar restricted to
> the base functionality (as in abstract classes)? And put the configuration
> aspect / concrete implementations in apps that use these operators?
>
> Thanks.
> Bhupesh
>
> On Sat, Dec 12, 2015 at 5:43 AM, Chandni Singh <chandni@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > Here is an example of doing Upsert with JDBC:
> >
> >
> https://github.com/chandnisingh/Malhar/blob/examples/apps/jdbc/src/main/java/com/datatorrent/jdbc/JdbcWriter.java
> >
> > Thanks,
> > Chandni
> >
> > On Fri, Dec 11, 2015 at 11:19 AM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > The operators are under Malhar/lib/db/jdbc.
> > >
> > > Here is one of them:
> > >
> > >
> >
> https://github.com/apache/incubator-apex-malhar/blob/devel-3/library/src/main/java/com/datatorrent/lib/db/jdbc/AbstractJdbcTransactionableOutputOperator.java
> > >
> > > They work with any kind PreparedStatement - insert or update
> > >
> > > Chandni
> > >
> > >
> > >
> > > On Fri, Dec 11, 2015 at 10:59 AM, Bhupesh Chawda <
> > bhupesh@datatorrent.com>
> > > wrote:
> > >
> > >> Hi Chandni,
> > >>
> > >> I don't see an update query being handled in the operator. Could you
> > >> please
> > >> point me to the appropriate class?
> > >> Or did you mean that handling a update query is just a matter of
> > extending
> > >> the class and providing a concrete implementation?
> > >>
> > >> Thanks.
> > >> -Bhupesh
> > >>
> > >> On Fri, Dec 11, 2015 at 10:42 PM, Chandni Singh <
> > chandni@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Hi Bhupesh,
> > >> >
> > >> > The current abstract JDBC Output Operators  in library are generic
> and
> > >> have
> > >> > already been used in multiple POCs and applications. In fact this
> > >> operator
> > >> > has matured through customer use cases. It is not just an insert
> > >> operator.
> > >> > We have used it to perform update and inserts.
> > >> >
> > >> > That said, I don't think it is a good idea to introduce input format
> > in
> > >> >  these abstract implementations. It is written to handle any type
of
> > >> query,
> > >> > be it a procedure call (that was an actual customer use case).
> > >> >
> > >> > Chandni
> > >> >
> > >> > On Fri, Dec 11, 2015 at 2:50 AM, Bhupesh Chawda <
> > >> bhupesh@datatorrent.com>
> > >> > wrote:
> > >> >
> > >> > > Hi All,
> > >> > >
> > >> > > We are planning to proceed with the following approach for JDBC
> > >> *update*
> > >> > > operator:
> > >> > >
> > >> > >    - *Update Query Configuration*
> > >> > >
> > >> > >
> > >> > >    - Example Update Query: *update tableName set a = ?** where
b
> = ?
> > >> and
> > >> > c
> > >> > >    > ?;*
> > >> > >    - Example JSON input array for parameter instantiations: *[{a,
> > >> > >    expression, INTEGER}, {b, expression, VARCHAR}, {c, expression,
> > >> > DATE}]*
> > >> > >
> > >> > > We are also planning to change the JDBC Output Operator in Malhar
> > >> Library
> > >> > > which currently does just insert. We plan to make the input format
> > >> > > consistent for both insert and update and hence the change to
the
> > >> current
> > >> > > way of configuration using JSON. Following would be the config
for
> > >> > inserts:
> > >> > >
> > >> > >    - *Insert Query Configuration*
> > >> > >
> > >> > >
> > >> > >    - Example Insert Query: *insert into tableName values (?,
?,
> .. ,
> > >> ?);*
> > >> > >    - Example JSON input array for parameter instantiations: *[{a,
> > >> > >    expression, INTEGER}, {b, expression, VARCHAR}, .. , {c,
> > >> expression,
> > >> > > DATE}]*
> > >> > >
> > >> > > Please let us know your thoughts.
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > -Bhupesh
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Dec 9, 2015 at 6:38 PM, Bhupesh Chawda <
> > >> bhupesh@datatorrent.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi All,
> > >> > > >
> > >> > > > Would it be a good idea to introduce the update functionality
to
> > the
> > >> > JDBC
> > >> > > > output operator in Apache Apex Malhar library.
> > >> > > >
> > >> > > > The following are possible approaches:
> > >> > > >
> > >> > > >    1. Accept a update query from the user with place holders
for
> > >> > values.
> > >> > > >    Example: *update tableName set a = ?, b = ? where c =
? and
> d >
> > >> ?*.
> > >> > > >    Here "?" will be provided by the user as java expressions
> which
> > >> will
> > >> > > be
> > >> > > >    evaluated from the incoming tuple.
> > >> > > >    2. Another option is to accept in some configuration
format
> > >> (json /
> > >> > > >    xml) the following and formulate the query in the operator.
> > This
> > >> can
> > >> > > become
> > >> > > >    arbitrarily complex.
> > >> > > >    1. update clause columns
> > >> > > >       2. update clause expressions
> > >> > > >       3. where clause columns
> > >> > > >       4. where clause expressions
> > >> > > >
> > >> > > > I am thinking about going ahead with 1. Please let me know
if
> any
> > >> other
> > >> > > > option is possible and whether such a functionality already
> exists
> > >> in
> > >> > > some
> > >> > > > other class.
> > >> > > >
> > >> > > > Thanks.
> > >> > > >
> > >> > > > -Bhupesh
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message