Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CABAipVYcb-gK_3mQcm_c7LmrabXP4swzSSM5QL+HyjuNaCMPBw@mail.gmail.com>
References: 
 <CABAipVZ=NoJ_TdPjobDjO48Nym98ccyT02sXk7onkfxj4fdqLw@mail.gmail.com>
	<CAD0KA=MzxTFdOH6_moOONtrfneq6_VkZtaud1Rha8PH9SHRO9w@mail.gmail.com>
	<CAMqituMcB=4GS26=8+LH6t=6dGDShUcS4t3+MyqtFJcMDz_4dQ@mail.gmail.com>
	<CABAipVa2_OuMm9SPG-YF8RtE4aQ0pz9_SeuQSF0GB28GsA0etA@mail.gmail.com>
	<CAKJfLDNikh2-KLAxevQP9zziD5z1peiLA5QO3Nn8LGKgshYTvw@mail.gmail.com>
	<CABAipVbKxDQRCTAiv7Xrvy+f-Gc9K9ptT-3OhxURtDrNNcjZHA@mail.gmail.com>
	<CAD0KA=NzHQe5-0dCXuKfLqaFB2qKHc4Gvj5pcr0t-3ar=ddxcQ@mail.gmail.com>
	<CABAipVYcb-gK_3mQcm_c7LmrabXP4swzSSM5QL+HyjuNaCMPBw@mail.gmail.com>
Date: Fri, 15 Jan 2016 23:56:37 +0530
Message-ID: 
 <CABAipVZnyDHBL5i+j3J5T93EJHHyKxJHBmytbu5pYbE1YDkBBw@mail.gmail.com>
Subject: Re: Adding class to CLASSPATH in populateDAG
From: Chinmay Kolhatkar <chinmay@datatorrent.com>
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1141ed18363e2d0529638b7e

--001a1141ed18363e2d0529638b7e
Content-Type: text/plain; charset=UTF-8

Hi All,

I've create pull request for this:
https://github.com/apache/incubator-apex-core/pull/208

Please review the pull request.

Thanks,
Chinmay.


On Fri, Jan 15, 2016 at 11:40 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Hi Everyone,
>
> If there are no further comments, I'll create the pull request for this
> change.
>
> Tushar, About the 2 parts you've mentioned, 2nd part is taken care of by
> this Jira. I'll log another Jira for 1st and address it there.
>
> I'll come back with pull request link soon.
>
> Thanks,
> Chinmay.
>
>
>
> On Wed, Jan 13, 2016 at 12:18 AM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
>> One use case could be that if we want same application to work on
>> different
>> data (schema) without
>>  compiling the application again. In this case schema can be provided as a
>> json description,
>> with field name and their type. When application starts it will generate
>> the classes using ASM
>> and write it to the temp jar, and configure operators to use the
>> classname.
>> operators can do a Class.forName to load the class based on name and use
>> those. This could enable to users to write generic applications and by
>> providing schema definition and configuring operators based on field
>> names.
>>
>> There are two parts
>> 1) Generating classes based on schema definition.
>> 2) Making generated class available to operators.
>>
>> - Tushar.
>>
>>
>>
>> On Tue, Jan 12, 2016 at 10:11 AM, Chinmay Kolhatkar <
>> chinmay@datatorrent.com
>> > wrote:
>>
>> > Hi Thomas,
>> >
>> > The module that I'm working on will have fields added to the POJO by
>> > current operator as per requirement of further processing stage.
>> >
>> > The fields added may or may not directly go into the final output POJO.
>> > Hence the intermediate stream is mean to carry the required data for
>> > processing of next stage.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>> >
>> > On Tue, Jan 12, 2016 at 9:47 AM, Thomas Weise <thomas@datatorrent.com>
>> > wrote:
>> >
>> > > Since the output POJO is defined by the user, why is there a need to
>> > > generate additional classes for intermediate streams within the
>> module?
>> > >
>> > > Can you give an example?
>> > >
>> > >
>> > > On Mon, Jan 11, 2016 at 8:11 PM, Chinmay Kolhatkar <
>> > > chinmay@datatorrent.com>
>> > > wrote:
>> > >
>> > > > Hi Thomas/David,
>> > > >
>> > > > Here is the details of the usecase.
>> > > > I'm writing a module which has following definition:
>> > > >
>> > > > <Input POJO> -> O1 -> O2 -> O3 -> <Output POJO>
>> > > >
>> > > > O1, O2 & O3 are the operators inside the module and module take
>> certain
>> > > > POJO as input and given certain type if POJO as output.
>> > > >
>> > > > The input and output POJO will be provide by user.
>> > > > But the streams between O1 & O2 and O2 & O3 are supposed to carry
>> POJO
>> > > > whose definition is unknown to/does not concern user.
>> > > >
>> > > > These streams within the module is supposed to contain some
>> > > intermediately
>> > > > processed POJOs, the definition of which  will be generated by
>> > parameters
>> > > > provided to the module.
>> > > >
>> > > > Here the way I'm trying to achieve this is to generate a class and
>> jar
>> > > for
>> > > > the intermediate streams and want to add those class in classpath.
>> > > >
>> > > > I am able to generate the class and jar for the intermediate
>> streams in
>> > > > populateDAG but I could not find any way to dynamically add those
>> > classes
>> > > > to classpath.
>> > > > Also, as these are temporary classes, they need to be deleted once
>> its
>> > > > utilized.
>> > > >
>> > > > Hence for this request.
>> > > >
>> > > > Please suggest if there is a better/cleaner approach to achieve
>> this.
>> > > >
>> > > > Thanks,
>> > > > Chinmay.
>> > > >
>> > > >
>> > > > On Tue, Jan 12, 2016 at 5:21 AM, David Yan <david@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > > If I remember correctly, all the necessary jar files, including
>> the
>> > > ones
>> > > > > specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to
>> HDFS
>> > > when
>> > > > > the launch command returns with an appid, so it should be safe to
>> > > delete
>> > > > > temporary jar files after the launch command returns.
>> > > > >
>> > > > > David
>> > > > >
>> > > > > On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi <
>> > tushar@datatorrent.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > +1
>> > > > > > In past we achieved this by writing generated class to a jar
>> file
>> > in
>> > > > /tmp
>> > > > > > directory and adding that jar in LIBJARS_CONF_KEY_NAME
>> > > > > > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME,
>> > > > > conf.get(StramAppLauncher.
>> > > > > > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar);
>> > > > > > This is a kind of hack and generated jar file does not get
>> cleaned
>> > up
>> > > > as
>> > > > > we
>> > > > > > don't get any callback after application is submitted in
>> > > > > > StreamingApplication. It may be
>> > > > > > good idea to provide a api to register temp jar which will get
>> > copied
>> > > > to
>> > > > > > application directory, and deleted (optionally) from local
>> machine
>> > > > after
>> > > > > > application is submitted.
>> > > > > >
>> > > > > > Regards,
>> > > > > > -Tushar.
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar <
>> > > > > > chinmay@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Everyone,
>> > > > > > >
>> > > > > > > I have a requirement where I need to add certain classes to
>> > > classpath
>> > > > > > > during application started (during initialization).
>> > > > > > >
>> > > > > > > I get to know the classes to be added to classpath only via
>> > > property
>> > > > of
>> > > > > > > application i.e. populateDAG method is the first point where I
>> > get
>> > > to
>> > > > > > know
>> > > > > > > what need to be added to classpath.
>> > > > > > > The class I want to add primary defines what a tuple would
>> look
>> > > like.
>> > > > > > > Basically properties will have the field information,
>> populateDAG
>> > > > needs
>> > > > > > to
>> > > > > > > generate the class and add it to classpath for all the
>> operators.
>> > > > > > >
>> > > > > > > Creating a class is not a problem, but adding to classpath is.
>> > > > > > > Is there a hook present in Apex platform where I can add given
>> > > .class
>> > > > > > file
>> > > > > > > to the classpath in populateDAG?
>> > > > > > >
>> > > > > > > If not, I was wondering if this will be a positive addition to
>> > the
>> > > > > > > platform.
>> > > > > > > I have some code already created for this. If everyone agrees
>> > this
>> > > to
>> > > > > be
>> > > > > > a
>> > > > > > > good addition to platform, I'll be happy to work on this and
>> > > > contribute
>> > > > > > to
>> > > > > > > the platform.
>> > > > > > >
>> > > > > > > Please share your thoughts.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Chinmay.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

--001a1141ed18363e2d0529638b7e--