apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Adding class to CLASSPATH in populateDAG
Date Mon, 11 Jan 2016 15:30:10 GMT
Hi All,

I've added APEXCORE Jira for this:
https://issues.apache.org/jira/browse/APEXCORE-304.

Here is how I'm thinking about achieving it:
1) LogicalPlan class will have a new method as follows:

*void addResource(String localPath, boolean deleteOnceCopied);*

This method will just keep the resource information i.e. jar file name in a
data structure in logical plan.

For application containing modules, if this method is used by module, then
during flattening of the DAG, this resource will be copied to the parent
DAG.

2) In StramClient, this added resource will also be copied to lib folder of
the application in HDFS.
After copying to HDFS, if the delete flag is set for this resource, the
local resource will be deleted.


Please share your thoughts on this approach.

Thanks,
Chinmay.




On Fri, Jan 8, 2016 at 1:11 PM, Tushar Gosavi <tushar@datatorrent.com>
wrote:

> +1
> In past we achieved this by writing generated class to a jar file in /tmp
> directory and adding that jar in LIBJARS_CONF_KEY_NAME
> conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME, conf.get(StramAppLauncher.
> LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar);
> This is a kind of hack and generated jar file does not get cleaned up as we
> don't get any callback after application is submitted in
> StreamingApplication. It may be
> good idea to provide a api to register temp jar which will get copied to
> application directory, and deleted (optionally) from local machine after
> application is submitted.
>
> Regards,
> -Tushar.
>
>
> On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Hi Everyone,
> >
> > I have a requirement where I need to add certain classes to classpath
> > during application started (during initialization).
> >
> > I get to know the classes to be added to classpath only via property of
> > application i.e. populateDAG method is the first point where I get to
> know
> > what need to be added to classpath.
> > The class I want to add primary defines what a tuple would look like.
> > Basically properties will have the field information, populateDAG needs
> to
> > generate the class and add it to classpath for all the operators.
> >
> > Creating a class is not a problem, but adding to classpath is.
> > Is there a hook present in Apex platform where I can add given .class
> file
> > to the classpath in populateDAG?
> >
> > If not, I was wondering if this will be a positive addition to the
> > platform.
> > I have some code already created for this. If everyone agrees this to be
> a
> > good addition to platform, I'll be happy to work on this and contribute
> to
> > the platform.
> >
> > Please share your thoughts.
> >
> > Thanks,
> > Chinmay.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message