apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Adding class to CLASSPATH in populateDAG
Date Fri, 15 Jan 2016 18:26:37 GMT
Hi All,

I've create pull request for this:
https://github.com/apache/incubator-apex-core/pull/208

Please review the pull request.

Thanks,
Chinmay.


On Fri, Jan 15, 2016 at 11:40 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Hi Everyone,
>
> If there are no further comments, I'll create the pull request for this
> change.
>
> Tushar, About the 2 parts you've mentioned, 2nd part is taken care of by
> this Jira. I'll log another Jira for 1st and address it there.
>
> I'll come back with pull request link soon.
>
> Thanks,
> Chinmay.
>
>
>
> On Wed, Jan 13, 2016 at 12:18 AM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
>> One use case could be that if we want same application to work on
>> different
>> data (schema) without
>>  compiling the application again. In this case schema can be provided as a
>> json description,
>> with field name and their type. When application starts it will generate
>> the classes using ASM
>> and write it to the temp jar, and configure operators to use the
>> classname.
>> operators can do a Class.forName to load the class based on name and use
>> those. This could enable to users to write generic applications and by
>> providing schema definition and configuring operators based on field
>> names.
>>
>> There are two parts
>> 1) Generating classes based on schema definition.
>> 2) Making generated class available to operators.
>>
>> - Tushar.
>>
>>
>>
>> On Tue, Jan 12, 2016 at 10:11 AM, Chinmay Kolhatkar <
>> chinmay@datatorrent.com
>> > wrote:
>>
>> > Hi Thomas,
>> >
>> > The module that I'm working on will have fields added to the POJO by
>> > current operator as per requirement of further processing stage.
>> >
>> > The fields added may or may not directly go into the final output POJO.
>> > Hence the intermediate stream is mean to carry the required data for
>> > processing of next stage.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>> >
>> > On Tue, Jan 12, 2016 at 9:47 AM, Thomas Weise <thomas@datatorrent.com>
>> > wrote:
>> >
>> > > Since the output POJO is defined by the user, why is there a need to
>> > > generate additional classes for intermediate streams within the
>> module?
>> > >
>> > > Can you give an example?
>> > >
>> > >
>> > > On Mon, Jan 11, 2016 at 8:11 PM, Chinmay Kolhatkar <
>> > > chinmay@datatorrent.com>
>> > > wrote:
>> > >
>> > > > Hi Thomas/David,
>> > > >
>> > > > Here is the details of the usecase.
>> > > > I'm writing a module which has following definition:
>> > > >
>> > > > <Input POJO> -> O1 -> O2 -> O3 -> <Output POJO>
>> > > >
>> > > > O1, O2 & O3 are the operators inside the module and module take
>> certain
>> > > > POJO as input and given certain type if POJO as output.
>> > > >
>> > > > The input and output POJO will be provide by user.
>> > > > But the streams between O1 & O2 and O2 & O3 are supposed to
carry
>> POJO
>> > > > whose definition is unknown to/does not concern user.
>> > > >
>> > > > These streams within the module is supposed to contain some
>> > > intermediately
>> > > > processed POJOs, the definition of which  will be generated by
>> > parameters
>> > > > provided to the module.
>> > > >
>> > > > Here the way I'm trying to achieve this is to generate a class and
>> jar
>> > > for
>> > > > the intermediate streams and want to add those class in classpath.
>> > > >
>> > > > I am able to generate the class and jar for the intermediate
>> streams in
>> > > > populateDAG but I could not find any way to dynamically add those
>> > classes
>> > > > to classpath.
>> > > > Also, as these are temporary classes, they need to be deleted once
>> its
>> > > > utilized.
>> > > >
>> > > > Hence for this request.
>> > > >
>> > > > Please suggest if there is a better/cleaner approach to achieve
>> this.
>> > > >
>> > > > Thanks,
>> > > > Chinmay.
>> > > >
>> > > >
>> > > > On Tue, Jan 12, 2016 at 5:21 AM, David Yan <david@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > > If I remember correctly, all the necessary jar files, including
>> the
>> > > ones
>> > > > > specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to
>> HDFS
>> > > when
>> > > > > the launch command returns with an appid, so it should be safe
to
>> > > delete
>> > > > > temporary jar files after the launch command returns.
>> > > > >
>> > > > > David
>> > > > >
>> > > > > On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi <
>> > tushar@datatorrent.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > +1
>> > > > > > In past we achieved this by writing generated class to a
jar
>> file
>> > in
>> > > > /tmp
>> > > > > > directory and adding that jar in LIBJARS_CONF_KEY_NAME
>> > > > > > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME,
>> > > > > conf.get(StramAppLauncher.
>> > > > > > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar);
>> > > > > > This is a kind of hack and generated jar file does not get
>> cleaned
>> > up
>> > > > as
>> > > > > we
>> > > > > > don't get any callback after application is submitted in
>> > > > > > StreamingApplication. It may be
>> > > > > > good idea to provide a api to register temp jar which will
get
>> > copied
>> > > > to
>> > > > > > application directory, and deleted (optionally) from local
>> machine
>> > > > after
>> > > > > > application is submitted.
>> > > > > >
>> > > > > > Regards,
>> > > > > > -Tushar.
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar <
>> > > > > > chinmay@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Everyone,
>> > > > > > >
>> > > > > > > I have a requirement where I need to add certain classes
to
>> > > classpath
>> > > > > > > during application started (during initialization).
>> > > > > > >
>> > > > > > > I get to know the classes to be added to classpath
only via
>> > > property
>> > > > of
>> > > > > > > application i.e. populateDAG method is the first point
where I
>> > get
>> > > to
>> > > > > > know
>> > > > > > > what need to be added to classpath.
>> > > > > > > The class I want to add primary defines what a tuple
would
>> look
>> > > like.
>> > > > > > > Basically properties will have the field information,
>> populateDAG
>> > > > needs
>> > > > > > to
>> > > > > > > generate the class and add it to classpath for all
the
>> operators.
>> > > > > > >
>> > > > > > > Creating a class is not a problem, but adding to classpath
is.
>> > > > > > > Is there a hook present in Apex platform where I can
add given
>> > > .class
>> > > > > > file
>> > > > > > > to the classpath in populateDAG?
>> > > > > > >
>> > > > > > > If not, I was wondering if this will be a positive
addition to
>> > the
>> > > > > > > platform.
>> > > > > > > I have some code already created for this. If everyone
agrees
>> > this
>> > > to
>> > > > > be
>> > > > > > a
>> > > > > > > good addition to platform, I'll be happy to work on
this and
>> > > > contribute
>> > > > > > to
>> > > > > > > the platform.
>> > > > > > >
>> > > > > > > Please share your thoughts.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Chinmay.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message