apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Adding class to CLASSPATH in populateDAG
Date Tue, 12 Jan 2016 04:17:45 GMT
Since the output POJO is defined by the user, why is there a need to
generate additional classes for intermediate streams within the module?

Can you give an example?


On Mon, Jan 11, 2016 at 8:11 PM, Chinmay Kolhatkar <chinmay@datatorrent.com>
wrote:

> Hi Thomas/David,
>
> Here is the details of the usecase.
> I'm writing a module which has following definition:
>
> <Input POJO> -> O1 -> O2 -> O3 -> <Output POJO>
>
> O1, O2 & O3 are the operators inside the module and module take certain
> POJO as input and given certain type if POJO as output.
>
> The input and output POJO will be provide by user.
> But the streams between O1 & O2 and O2 & O3 are supposed to carry POJO
> whose definition is unknown to/does not concern user.
>
> These streams within the module is supposed to contain some intermediately
> processed POJOs, the definition of which  will be generated by parameters
> provided to the module.
>
> Here the way I'm trying to achieve this is to generate a class and jar for
> the intermediate streams and want to add those class in classpath.
>
> I am able to generate the class and jar for the intermediate streams in
> populateDAG but I could not find any way to dynamically add those classes
> to classpath.
> Also, as these are temporary classes, they need to be deleted once its
> utilized.
>
> Hence for this request.
>
> Please suggest if there is a better/cleaner approach to achieve this.
>
> Thanks,
> Chinmay.
>
>
> On Tue, Jan 12, 2016 at 5:21 AM, David Yan <david@datatorrent.com> wrote:
>
> > If I remember correctly, all the necessary jar files, including the ones
> > specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to HDFS when
> > the launch command returns with an appid, so it should be safe to delete
> > temporary jar files after the launch command returns.
> >
> > David
> >
> > On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi <tushar@datatorrent.com>
> > wrote:
> >
> > > +1
> > > In past we achieved this by writing generated class to a jar file in
> /tmp
> > > directory and adding that jar in LIBJARS_CONF_KEY_NAME
> > > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME,
> > conf.get(StramAppLauncher.
> > > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar);
> > > This is a kind of hack and generated jar file does not get cleaned up
> as
> > we
> > > don't get any callback after application is submitted in
> > > StreamingApplication. It may be
> > > good idea to provide a api to register temp jar which will get copied
> to
> > > application directory, and deleted (optionally) from local machine
> after
> > > application is submitted.
> > >
> > > Regards,
> > > -Tushar.
> > >
> > >
> > > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I have a requirement where I need to add certain classes to classpath
> > > > during application started (during initialization).
> > > >
> > > > I get to know the classes to be added to classpath only via property
> of
> > > > application i.e. populateDAG method is the first point where I get to
> > > know
> > > > what need to be added to classpath.
> > > > The class I want to add primary defines what a tuple would look like.
> > > > Basically properties will have the field information, populateDAG
> needs
> > > to
> > > > generate the class and add it to classpath for all the operators.
> > > >
> > > > Creating a class is not a problem, but adding to classpath is.
> > > > Is there a hook present in Apex platform where I can add given .class
> > > file
> > > > to the classpath in populateDAG?
> > > >
> > > > If not, I was wondering if this will be a positive addition to the
> > > > platform.
> > > > I have some code already created for this. If everyone agrees this to
> > be
> > > a
> > > > good addition to platform, I'll be happy to work on this and
> contribute
> > > to
> > > > the platform.
> > > >
> > > > Please share your thoughts.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message