apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AJAY GUPTA <ajaygit...@gmail.com>
Subject Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator
Date Sat, 14 Jan 2017 07:35:37 GMT
Since the query can be different for different databases, the user will
have to provide query to the operator. Rather than this, I believe it's
easier for user to directly execute create table query on DB.

Also, the create table script won't be that heavy that we create script for
it. Probably adding a generic type of query in the docs itself should
suffice.


Ajay

On Sat, 14 Jan 2017 at 10:27 AM, Yogi Devendra <yogidevendra@apache.org>
wrote:

> As Aniruddha pointed out, table creation should be done by dbadmin.
>
> In that case, utility script will be helpful.
>
>
>
> If we embed this code inside operator or application; then it will be
>
> difficult for dbadmin to use it.
>
>
>
> ~ Yogi
>
>
>
> On 14 January 2017 at 03:43, Thomas Weise <thw@apache.org> wrote:
>
>
>
> > -1 for automatic schema modification, unless the user asked for it. See
>
> > comment on JIRA.
>
> >
>
> >
>
> > On Fri, Jan 13, 2017 at 5:11 AM, Aniruddha Thombare <
>
> > aniruddha@datatorrent.com> wrote:
>
> >
>
> > > The tables should be created / altered by dbadmin.
>
> > > We shouldn't worry about table creations as its one-time activity.
>
> > >
>
> > >
>
> > >
>
> > > Thanks,
>
> > >
>
> > > A
>
> > >
>
> > >
>
> > > _____________________________________
>
> > > Sent with difficulty, I mean handheld ;)
>
> > >
>
> > > On 13 Jan 2017 6:37 pm, "Yogi Devendra" <yogidevendra@apache.org>
> wrote:
>
> > >
>
> > > I am not very keen on having utility script.
>
> > > But, "no side-effects without explicit ask by the end-user" is
> important.
>
> > >
>
> > > ~ Yogi
>
> > >
>
> > > On 13 January 2017 at 16:44, Priyanka Gugale <priyag@apache.org>
> wrote:
>
> > >
>
> > > > IMO it's okay to create table in java code. We should document it in
>
> > > > operator guide as well as put a log message when we create table.
>
> > > > And in case you don't have privileges, the operator should throw
>
> > > meaningful
>
> > > > message.
>
> > > >
>
> > > > -Priyanka
>
> > > >
>
> > > > On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra <
>
> > yogidevendra@apache.org>
>
> > > > wrote:
>
> > > >
>
> > > > > My suggestions:
>
> > > > >
>
> > > > >    1. Have a separate utility script for creating this table.
>
> > > > >    2. Have README for the utility script
>
> > > > >    3. Mention about the utility script in the operator javadocs.
>
> > > > >    4. Mention  about the utility script in the application README.
>
> > > > >    5. If at all, you wish to ease out the process; you can
> introduce
>
> > > flag
>
> > > > >    like autoPopulateMetaTable. But. default value of this flag
> should
>
> > > to
>
> > > > be
>
> > > > >    off.
>
> > > > >    6. I would prefer to avoid side-effects unless explicitly asked
> by
>
> > > the
>
> > > > >    end user.
>
> > > > >    7. Relevant exceptions should be caught and should have a
> message
>
> > > > which
>
> > > > >    can be understood by the end user.
>
> > > > >
>
> > > > > ~ Yogi
>
> > > > >
>
> > > > > On 13 January 2017 at 15:57, Hitesh Kapoor <hitesh@datatorrent.com
> >
>
> > > > wrote:
>
> > > > >
>
> > > > > > Hi All,
>
> > > > > >
>
> > > > > > Currently to use JdbcPOJOInsertOutputOperator, user needs to
> create
>
> > > > > > "dt_meta" table to enforce
>
> > > > > > exactly-once processing semantic. If the user fails to create
> this
>
> > > > table
>
> > > > > > before launching the application an exception is thrown.
>
> > > > > > To handle this scenario we can automate the process of creating
>
> > this
>
> > > > > table,
>
> > > > > > assuming the user has the appropriate privileges. The problem
> with
>
> > > this
>
> > > > > > approach is that it may not be a very good idea to modify user's
>
> > > > database
>
> > > > > > automatically , also if the user doesn't has the appropriate
>
> > > privileges
>
> > > > > it
>
> > > > > > will eventually throw an exception (however a different
> exception).
>
> > > > > > So I need your opinion if we should automate the creation of
this
>
> > > > > internal
>
> > > > > > table (if it doesn't exists) or continue with the existing
>
> > behaviour
>
> > > or
>
> > > > > > anything else.
>
> > > > > >
>
> > > > > > Regards,
>
> > > > > > Hitesh
>
> > > > > >
>
> > > > >
>
> > > >
>
> > >
>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message