apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Adding class to CLASSPATH in populateDAG
Date Tue, 12 Jan 2016 04:11:58 GMT
Hi Thomas/David,

Here is the details of the usecase.
I'm writing a module which has following definition:

<Input POJO> -> O1 -> O2 -> O3 -> <Output POJO>

O1, O2 & O3 are the operators inside the module and module take certain
POJO as input and given certain type if POJO as output.

The input and output POJO will be provide by user.
But the streams between O1 & O2 and O2 & O3 are supposed to carry POJO
whose definition is unknown to/does not concern user.

These streams within the module is supposed to contain some intermediately
processed POJOs, the definition of which  will be generated by parameters
provided to the module.

Here the way I'm trying to achieve this is to generate a class and jar for
the intermediate streams and want to add those class in classpath.

I am able to generate the class and jar for the intermediate streams in
populateDAG but I could not find any way to dynamically add those classes
to classpath.
Also, as these are temporary classes, they need to be deleted once its
utilized.

Hence for this request.

Please suggest if there is a better/cleaner approach to achieve this.

Thanks,
Chinmay.


On Tue, Jan 12, 2016 at 5:21 AM, David Yan <david@datatorrent.com> wrote:

> If I remember correctly, all the necessary jar files, including the ones
> specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to HDFS when
> the launch command returns with an appid, so it should be safe to delete
> temporary jar files after the launch command returns.
>
> David
>
> On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
> > +1
> > In past we achieved this by writing generated class to a jar file in /tmp
> > directory and adding that jar in LIBJARS_CONF_KEY_NAME
> > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME,
> conf.get(StramAppLauncher.
> > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar);
> > This is a kind of hack and generated jar file does not get cleaned up as
> we
> > don't get any callback after application is submitted in
> > StreamingApplication. It may be
> > good idea to provide a api to register temp jar which will get copied to
> > application directory, and deleted (optionally) from local machine after
> > application is submitted.
> >
> > Regards,
> > -Tushar.
> >
> >
> > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > I have a requirement where I need to add certain classes to classpath
> > > during application started (during initialization).
> > >
> > > I get to know the classes to be added to classpath only via property of
> > > application i.e. populateDAG method is the first point where I get to
> > know
> > > what need to be added to classpath.
> > > The class I want to add primary defines what a tuple would look like.
> > > Basically properties will have the field information, populateDAG needs
> > to
> > > generate the class and add it to classpath for all the operators.
> > >
> > > Creating a class is not a problem, but adding to classpath is.
> > > Is there a hook present in Apex platform where I can add given .class
> > file
> > > to the classpath in populateDAG?
> > >
> > > If not, I was wondering if this will be a positive addition to the
> > > platform.
> > > I have some code already created for this. If everyone agrees this to
> be
> > a
> > > good addition to platform, I'll be happy to work on this and contribute
> > to
> > > the platform.
> > >
> > > Please share your thoughts.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message