Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CABAipVYVoV2F-jCa_UagQLoAqD7y7iOjKOZEKpZK-XM3Qde-QA@mail.gmail.com>
References: 
 <CABAipVbvnOpVvs_eR=vn5JWKEGeV=PnJWX+umULBk3R6cXs4OA@mail.gmail.com>
 <CAKt=akFN7xPeV4rWjrcXmK0RgHe_eDqCjUZ5iJf6tspqhCnM8A@mail.gmail.com>
 <CADS+zCsh9pQbbdzFXHc-Tr2XKjOpfEYyKiqW7EbN2AOoxer=_A@mail.gmail.com>
 <CABAipVZ5e4RKWub31m-Rn41Ja2iCCNR8ym8u2ypHkTt+QgDUsg@mail.gmail.com>
 <CABAipVYrKYkQbb94QEL3q_kh8HTikvdO2PWzOe+VnSWuLyfvDA@mail.gmail.com>
 <CAKt=akFLgs2mHetEX2A9U-pFhhhNQPpJmZ+HmAdO-0W+DJbF8Q@mail.gmail.com>
 <CABAipVancCr2Ac58kaHxp+bhz1MjMJsqNKgxcDxMnM49-1w3Jg@mail.gmail.com>
 <CAKt=akHAJdk8mswi5cO2SBhWP_A1p6P0_jmrXM_GuHVW4EZs0A@mail.gmail.com>
 <CABAipVbq-nW7g5r4-mKZmy6UhFwEbY-QPbK8P=3G-N_+ecDXTA@mail.gmail.com>
 <CABAipVYVoV2F-jCa_UagQLoAqD7y7iOjKOZEKpZK-XM3Qde-QA@mail.gmail.com>
From: Bhupesh Chawda <bhupesh@datatorrent.com>
Date: Fri, 22 Jan 2016 19:04:52 +0530
Message-ID: 
 <CAHvM9d2kYudGA4NMYwhRmXdrqW--sn0Sn+fPGTVv3=F+aYKszQ@mail.gmail.com>
Subject: Re: Ability to generate POJO class on the fly
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c2d2e6e1c8b10529ec4976

--001a11c2d2e6e1c8b10529ec4976
Content-Type: text/plain; charset=UTF-8

Yes. So in the JDBC output case, a json object could be readily constructed
by the operator to get a POJO class rather than playing around with JSON
syntax.

On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar <chinmay@datatorrent.com>
wrote:

> @bhupesh, would it help if JSONObject is taken as input to this utility
> rather than json string?
>
> Thanks,
> Chinmay
> On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <chinmay@datatorrent.com> wrote:
>
> > Got it. I think approach 2 is still possible with json based schema.
> > So basically, the operator can create json schema as mention in first
> mail
> > based on column name and type. Then pass that json schema to this utility
> > to get class to be used directly.
> >
> > Thanks,
> > Chinmay.
> > On 22 Jan 2016 18:38, "Priyanka Gugale" <priyanka@datatorrent.com>
> wrote:
> >
> >> Say I need to emit pojo from JDBCInputOperator. I will use on fly POJO
> >> generator for this purpose instead of providing predefined class.
> >> Now to generate POJO on fly there are two options:
> >> 1. Get json as config from user as you specified.
> >> 2. Based on column name and type generate pojo fields and hence pojo
> class
> >> without much of user configuration. (I am only considering one level
> here,
> >> not the nested pojo)
> >>
> >> Option 1 is must have to provide user a way to control how we generate
> >> pojo. But option 2 is useful when user simply want to put column values
> in
> >> some object.
> >>
> >> -Priyanka
> >>
> >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
> >> chinmay@datatorrent.com>
> >> wrote:
> >>
> >> > Hi Priyanka,
> >> >
> >> > Can you give an example?
> >> >
> >> > Thanks,
> >> > Chinmay.
> >> >
> >> > ~ Chinmay
> >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <priyanka@datatorrent.com>
> >> wrote:
> >> >
> >> > > We do need configuration in some form like json, but I was thinking
> we
> >> > can
> >> > > have default wherever possible. This is an option and not
> replacement
> >> to
> >> > > json config.
> >> > >
> >> > > -Priyanka
> >> > >
> >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> >> > > chinmay@datatorrent.com>
> >> > > wrote:
> >> > >
> >> > > > @priyanka
> >> > > >
> >> > > > Advantage with JSON schema is that one can represent nested POJO
> >> > > definition
> >> > > > in that. If you're referring to FieldInfo, I'm not sure if that is
> >> > > possible
> >> > > > there.
> >> > > >
> >> > > > Even if we chose not to have nested POJO now, I think JSON schema
> >> input
> >> > > to
> >> > > > this utility gives more chance to expand easily later.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> >> > > > chinmay@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Sure. We can make this utility to provide byte[] for compiled
> >> class.
> >> > > > > Though the utility will give a OutputStream object.. One can get
> >> > byte[]
> >> > > > > out of that.
> >> > > > >
> >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
> >> tim@datatorrent.com
> >> > >
> >> > > > > wrote:
> >> > > > >
> >> > > > >> Suggestion: +1 for automatic generation of schema. The user
> >> should
> >> > be
> >> > > > able
> >> > > > >> to override the a manually specified schema though.
> >> > > > >>
> >> > > > >> Question: If the mechanism used to generate the class can
> >> produce a
> >> > > > byte[]
> >> > > > >> array, then you can send the byte[] array for the class to
> >> > downstream
> >> > > > >> operators and load the class from the byte[] array in each
> >> operator.
> >> > > An
> >> > > > >> example of how to do this is here:
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> >> > > > >>
> >> > > > >> Thanks,
> >> > > > >> Tim
> >> > > > >>
> >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> >> > > > >> priyanka@datatorrent.com>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> > Hi,
> >> > > > >> >
> >> > > > >> > Suggestion:
> >> > > > >> > This came up with one of the discussion with Pramod, will it
> >> be a
> >> > > good
> >> > > > >> > idea, for database input operators to generate pojo based on
> >> the
> >> > > > >> selected
> >> > > > >> > column field names and types? No need to accept json input
> from
> >> > > user.
> >> > > > >> >
> >> > > > >> > Question:
> >> > > > >> > How can we share this POJO among multiple operators when
> >> > application
> >> > > > is
> >> > > > >> > already launched and class is generated on the fly?
> >> > > > >> >
> >> > > > >> > -Priyanka
> >> > > > >> >
> >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> >> > > > >> > chinmay@datatorrent.com
> >> > > > >> > > wrote:
> >> > > > >> >
> >> > > > >> > > Hi All,
> >> > > > >> > >
> >> > > > >> > > We're planning to add a utility in malhar-library for
> >> > generating a
> >> > > > >> POJO
> >> > > > >> > > class on the fly from given JSON schema.
> >> > > > >> > >
> >> > > > >> > > Use case is where the application is provided with schema
> and
> >> > that
> >> > > > >> needs
> >> > > > >> > to
> >> > > > >> > > be used in one or more operators either as a tuple over the
> >> > stream
> >> > > > OR
> >> > > > >> for
> >> > > > >> > > processing.
> >> > > > >> > >
> >> > > > >> > > General Design:
> >> > > > >> > > 1. Utility will be provided with fqcn of the class and
> schema
> >> > > > >> definition
> >> > > > >> > > provided as json.
> >> > > > >> > > 2. The schema definition will look like following:
> >> > > > >> > >     {
> >> > > > >> > >        "fqcn":"<qualified class name>",
> >> > > > >> > >        "fields": [
> >> > > > >> > >          {
> >> > > > >> > >            "name":"field1",
> >> > > > >> > >            "type":"long"
> >> > > > >> > >          },
> >> > > > >> > >          {
> >> > > > >> > >            "name":"field2",
> >> > > > >> > >            "type": "string"
> >> > > > >> > >          }
> >> > > > >> > >        ]
> >> > > > >> > >     }
> >> > > > >> > > 3. Supported types identified in "type" JSON field are:
> >> > > > >> > >       boolean, char, byte, short, int, float, long, double
> >> > > > >> > > 4. The output of this utility will be a generated .class
> >> file in
> >> > > the
> >> > > > >> form
> >> > > > >> > > of FSDataOutputStream.
> >> > > > >> > > 5. Xbean asm5 library will be used for this.
> >> > > > >> > > 6. Following methods will be added to the generated class:
> >> > > > >> > >     a. Getter/Setter methods for given fields.
> >> > > > >> > >     b. simple toString - Generate string equivalent for all
> >> the
> >> > > > fields
> >> > > > >> > >     c. hashCode method - calculate the overall hashCode
> using
> >> > > > >> individual
> >> > > > >> > > field hashcodes, similar to how String.hashCode generates
> >> > hashCode
> >> > > > of
> >> > > > >> > > string using chars in it.
> >> > > > >> > >     d. equals method - Similar to how String.equals method
> >> has
> >> > > done
> >> > > > it
> >> > > > >> > > using individual char.
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > > Questions I have about the functionality:
> >> > > > >> > > 1. Should the utility also support nested pojo definition
> via
> >> > > schema
> >> > > > >> > json?
> >> > > > >> > >     In such case the field definition can look like
> >> following:
> >> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested
> class>",
> >> > > > >> > > "fields":[....] }
> >> > > > >> > >     The nested class will be provided
> >> > > > >> > >
> >> > > > >> > > 2. Should the utility also support giving out the
> >> > JarOutputStream
> >> > > > >> which
> >> > > > >> > > will have a generated jar?
> >> > > > >> > >
> >> > > > >> > > Please let me know your thoughts.
> >> > > > >> > >
> >> > > > >> > > Thanks,
> >> > > > >> > > Chinmay.
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

--001a11c2d2e6e1c8b10529ec4976--